Big Data Lead
Tasks
- Build data lakehouse architecture on Databricks
- Define ingestion and integration strategy
- Deliver data products to business requirements
- Design scalable data pipelines
- Develop PySpark notebooks
- Document workflows pipelines and architecture
- Ensure data security and compliance
- Gather requirements and build technical solutions
- Implement ETL processes
- Ingest and transform data
- Migrate data pipelines
- Optimize data processing performance
- Write Spark SQL transformations
Perks/Benefits
- N/A
Skills/Tech-stack
BTEQ | Data Lakehouse | Data Pipelines | Data integration | Databricks | ETL | IBM Sterling | Microsoft Azure | PySpark | Python | Spark SQL | TPT | Talend | Teradata
Education
N/A
Roles
Big Data Engineer | Data Engineer | Data Engineering | Data Engineering Lead | Engineer | Engineering Lead | Lead
Related jobs
-
Entry-level Full TimeCary, NC, United States2h ago
-
Apache Spark | Clustering | Data Ingestion | Data Ingestion Pipelines | Data ModelingSenior-level ContractLos Angeles, United States2h ago
-
Cloud Computing | Data Pipelines | Experimental Design | Java | Machine LearningCommunity volunteering | Internal mobility | Learning opportunities | Mentorship programs | Recognition programsMid-level Full TimeAustin, TX, United States2h ago
-
AI Solutions Expert USD 137K-236KArtificial Intelligence | Cybersecurity | Data Pipelines | Decision support | EWSenior-level Full TimeUSA-VA-Stafford4h ago
-
Data Analysis | Distributed Systems | Experimentation | Information Retrieval | Language ModelsSenior-level Full TimeSan Jose, California, United States4h ago
-
Apache Spark | Computer Vision | Data Curation | Data Pipelines | GroundingSenior-level Full TimeSunnyvale, CA | Bellevue, WA | …5h ago
-
Staff Software Engineer, NotebookLM, Generative AI, Labs USD 207K-300KAlerting | Computer Vision | Data Processing | Data Structures | Data structures algorithmsSenior-level Full TimeMountain View, CA, USA5h ago
-
Systems Development Engineer, Quantum AI USD 138K-198KAutomation | CI/CD | Cloud APIs | Cloud infrastructure | Configuration ManagementMid-level Full TimeGoleta, CA, USA5h ago
-
Software Engineer III, AI/ML, Display Ads USD 147K-211KAlgorithms | C++ | Data Analysis | Data Processing | Data StructuresSenior-level Full TimeMountain View, CA, USA5h ago
-
AI/ML Engineer USD 117K-171KA/B | A/B Testing | API Development | AWS | B testingDental insurance | Health insurance | Paid time off | Retirement plan | Sick leaveSenior-level Full TimeResidence Based, Residence Based, US8h ago
-
Cloud Data Engineer USD 92K-113KAgile | Apache Airflow | Beam | BigQuery | Cloud ComposerOnsite workMid-level Full TimeAda, MI, United States, 493559h ago
-
Analytics Engineer Manager, CNBC & MS NOW USD 110K-130KData Documentation | Data Modeling | Data Pipelines | Data Quality | Data Testing401k | Dental insurance | Discounts | Free shuttle transportation | Gourmet cafeteriaMid-level Full TimeEnglewood Cliffs, NEW JERSEY, United States13h ago
-
AWS | Azure | Cloud platform | Data Pipelines | Demand forecastingOnsite collaborationSenior-level Full TimeLewisville, TX, US13h ago
-
Forward Deployed Machine Learning Engineer USD 180K-300KAPI Design | Cloud Computing | Deep learning | Diffusion Models | Fine TuningIn-person collaboration days | Remote work flexibility | Travel cost coverageSenior-level Full TimeSan Francisco (USA) R15h ago
-
Director of Data Engineering USD 170K-230KCause analysis | Data Accuracy | Data Architecture | Data Engineering | Data Integration Architectures401k match | Catered lunch Fridays | Commuter benefits | Dental insurance | Paid health insuranceExecutive-level Full TimeNew York, New York, United States16h ago
-
Forward Deployment Engineer - Gen AI USD 162K-224KAWS Bedrock | AWS SageMaker | Autogen | Azure OpenAI | ChromaCareer development opportunities | Individual responsibility | Travel to client sitesMid-level Full TimeNew York, New York, United States16h ago
-
Computer Vision | Experiment design | Experiment tracking | Generative AI | Image ProcessingSenior-level Full TimeAEP Headquarters, United States16h ago
-
Senior Software Engineer - Platform & MLOps USD 152K-230KAWS | Azure | CI/CD | Datadog | DockerDiscretionary incentive plan | Flexible work policy | Learning and development access | Medical benefitsSenior-level Full TimeSeattle, Washington, United States - Remote R16h ago
-
Data Engineer USD 84K-101KAgile | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageCharity donation matching | Digital course library | Employee assistance program | Mindfulness program | Volunteer hours matching programMid-level Full TimeIndia Pune (Mississippi Building)16h ago
-
Lead Software Engineer (Python, LLM) - AskGartner USD 116K-170KAWS | AWS EMR | AWS Glue | AWS Kinesis | AWS QuickSight20 plus PTO days | 401k match | Collaborative culture | Employee assistance program | Employee stock purchase planSenior-level Full TimeIrving - 6011 Connection, United States16h ago
-
Specialist Solutions Architect - AI/ML USD 180K-247KAI guardrails | Amazon Web Services | Apache Spark | Artificial Intelligence | Cloud ComputingMentorship | Remote work | Technical training | Travel up to 30 percentSenior-level Full TimeUnited States17h ago
-
Senior Data Engineer IS - Remote USD 122K-208KAI/ML | APIs | AWS Glue | Apache Airflow | Azure Data401k matching | Dental insurance | Disability insurance | Health insurance | Life insuranceSenior-level Full TimePortland, OR, United States R17h ago
-
Lead Data Engineer USD 115K-130KAgile | Apache Airflow | Azure Data | Azure Data Factory | Cloud Orchestration401k | Dental insurance | Medical insurance | Paid leave | Tuition reimbursementSenior-level Full TimeUniversal City, CALIFORNIA, United States17h ago
-
Corporate Treasury, Liquidity Meterics Strats, Quantitative Engineering, Dallas, Vice President USD 150K-250KC# | C++ | Financial markets | Java | Liquidity RiskExecutive-level Full TimeDallas, TX, United States18h ago
-
Principal AI Platform Engineer USD 167K-220KAgent Orchestration | Backend Development | Braintrust | Cost Optimization | Data PipelinesEquity | Flexible Token Limits | Health, dental, vision coverage | Unlimited paid time offSenior-level Full TimeSan Francisco, California R18h ago