Big Data Lead
Tasks
- Build big data processing pipelines
- Ensure data quality and integrity
- Ensure data security and privacy compliance
- Implement CI/CD pipelines
- Implement ETL pipelines
- Integrate PySpark with ingestion frameworks
- Manage data lineage documentation
- Transform and process data
- Troubleshoot PySpark applications
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Hadoop | Apache Hive | Apache Kafka | Apache Spark | CI/CD | Data Quality | Data Warehousing | DevOps | ETL | PySpark | Python | SQL
Education
N/A
Roles
Big Data Engineer | Data Engineer | Engineer | Hadoop Engineer
Related jobs
-
API Integrations | AWS Glue | AWS Lambda | Amazon Redshift | Amazon S3Mid-level ContractLos Angeles, United States2h ago
-
Lead Data Expert USD 97K-163KData Architecture | Data Automation | Data Modeling | Data Quality | Data VisualizationOnsite 30 to 50 percentSenior-level Full TimeArlington/Rosslyn, Virginia, United States3h ago
-
CRM | Data Mining | Deep learning | Email outreach | Knowledge graphsMid-level Full TimeSan Jose, California, United States3h ago
-
Senior Software Engineer, AI/ML, Google Public Sector USD 174K-252KAlgorithms | C++ | Cloud Object Storage | Data Structures | Distributed ComputingSenior-level Full TimeReston, VA, USA4h ago
-
Staff Software Engineer, Intelligent Database Management USD 207K-300KAI | API Design | AlloyDB | Audit Logging | BigtableSenior-level Full TimeSunnyvale, CA, USA; Kirkland, WA, USA4h ago
-
Forward Deployed Engineer I, GenAI, Google Cloud USD 102K-145KAPI Development | Agent Framework | Agent systems | Cloud Computing | CrewAISenior-level Full TimeSan Francisco, CA, USA; Atlanta, GA, …4h ago
-
C++ | Data Mining | Data Processing | Deep learning | Few-Shot LearningSenior-level Full TimeMountain View, CA, USA4h ago
-
Software Engineer III, AI/ML, Google Ads USD 147K-211KC++ | Data Processing | Debugging | Information Retrieval | Language ProcessingSenior-level Full TimeMountain View, CA, USA4h ago
-
Senior Data Engineer USD 140K-165KAI/ML | Apache Airflow | CRM Integration | Cloud Data | Cloud Data PlatformsSenior-level Full TimeNY, United States7h ago
-
Senior Staff Data Engineer USD 123K-185KAPI | Automated testing | Cloud platform | Data Analysis | Data GovernanceDental insurance | Health care | Paid time off | Retirement plan | Sick leaveSenior-level Full TimeResidence Based, Residence Based, US7h ago
-
ML Engineer USD 190K-320KCost Optimization | Data Versioning | Dataset Operations | Dataset curation | Eval Frameworks401k matching | Dental insurance | Employee assistance program | Health insurance | Stock optionsSenior-level Full TimeSan Francisco10h ago
-
Senior-level Full TimeOklahoma City12h ago
-
Data Engineer USD 105K-115KBig Data | Cloud Computing | Data Modeling | Data Pipelines | Data StorageActive secret clearanceMid-level Full TimeSan Diego, CA, US12h ago
-
Principal Data Engineer USD 200K-240KAWS | Agentic Workflows | Anomaly Detection | Batch pipelines | CCPA401k plan | Commuter benefits | Flexible vacation | Life insurance | Long-term disabilitySenior-level Full TimeBoulder, Colorado or New York City, … R12h ago
-
Associate AI - Data Scientist / ML Engineer (Azure) USD 46K-110KAzure Container | Azure Container Instances | Azure Data | Azure Data Factory | Azure Data LakeDental insurance | Disability insurance | Employee assistance program | Life insurance | Medical insuranceMid-level Full TimeNashville, TN, US14h ago
-
Staff Machine Learning Engineer USD 159K-309KAWS | Airflow | Apache Spark | BigQuery | Cloud platform401k plan with company match | Commuter benefits | Disability coverage | Electric Car Charging Station | Employee assistance programSenior-level Full TimeMountain View, USA15h ago
-
Staff Machine Learning Engineer USD 152K-261KAWS | Airflow | Apache Spark | BigQuery | Cloud platform401k plan with company match | Commuter benefits | Disability insurance | Electric Car Charging Station | Employee assistance programSenior-level Full TimeMountain View, USA15h ago
-
Staff Software Engineer - AI Research Infrastructure USD 190K-270KBackend Services | C plus plus | CI/CD | Cloud infrastructure | Cluster managementSenior-level Full TimeNew York City, New York; San …15h ago
-
Senior AI Solutions Engineer, Enterprise Knowledge Work USD 260K-325KAgentic Systems | Dspy | Evaluation | LLM orchestration | LanggraphCollaborative culture | Flexible working hours | Supportive work environmentSenior-level Full TimeNew York, New York, United States; …15h ago
-
Senior AI Solutions Engineer, Software Engineering USD 260K-325KAgentic Software | Agentic Software Engineering | Agentic Workflows | Benchmarking | Code generationCollaborative culture | Five days per week | Flexible working hours | Supportive work environmentSenior-level Full TimeNew York, New York, United States; …15h ago
-
Quantum Software Engineer II USD 100K-215KAI Tooling | CUDA | Compilers | Complex linear algebra | DebuggingEntry-level Full TimeRedmond, WA, US15h ago
-
Principal, AI Platform Engineer USD 125K-187KAWS | Azure | CI/CD | Data leakage | Deterministic executionSenior-level Full TimeAtlanta, Georgia, US United States, 3034016h ago
-
Lead Data Engineer, Marketing Operations and Engineering USD 139K-257KAWS | Apache Airflow | Apache Spark | Cloud Computing | Cloud platformSenior-level Full TimeSan Jose, United States R16h ago
-
Mid-level Full TimeCharlotte, United States16h ago
-
Generative AI Consultant USD 105K-105KAWS | Azure | CI/CD | Deep learning | Docker401k matching | Health insurance | Paid time off | Parental leave | Professional developmentMid-level Full TimeSan Francisco, CA, United States17h ago