Lead Software Engineer - Databricks/Spark/AWS
Tasks
- Build reusable Python and Java libraries and APIs
- Conduct incident response and root cause analysis
- Deploy infrastructure with Terraform
- Design secure data ingestion and transformation
- Drive Spark performance engineering
- Enforce data quality, lineage, and governance
- Establish lakehouse patterns with Delta Lake
- Implement CI/CD for data projects
- Integrate AWS event orchestration
- Lead data pipeline architecture and delivery
- Lead security enforcement with least privilege
- Mentor engineers and conduct technical design reviews
- Orchestrate Databricks workflows
- Own Databricks cluster strategy and setup
Perks/Benefits
- N/A
Skills/Tech-stack
AWS Glue | AWS IAM | AWS Lambda | AWS S3 | AWS Secrets | AWS Secrets Manager | Amazon CloudWatch | Amazon Kinesis | Amazon MSK | Apache Spark | Broadcast joins | CI/CD | Data Validation | Databricks | Databricks Workflows | Delta Lake | Delta Lake Time Travel | Delta Live Tables | Delta Live) | Encryption | Git | JUnit | Java | Kafka | Least Privilege | Networking | Pytest | Python | SQL | Secrets Manager | Security groups | Shuffle Tuning | Spark AQE | Spark Partitioning | Structured Streaming | Terraform | Time Travel | Unity Catalog | VPC
Education
N/A
Related jobs
-
Data Engineer (remote) USD 87K-110KAgile | BigQuery | Data Governance | Data Modeling | Data Pipelines401k match | Employee assistance program | Employee stock purchase plan | Flexible schedule | Health insuranceMid-level Full TimeWork From Home, United States R1h ago
-
Data Engineering Manager (remote) USD 87K-110KAgile | BigQuery | Data Governance | Data Modeling | Data Quality401k match | Employee assistance program | Employee stock purchase plan | Flexible schedules | Health insuranceMid-level Full TimeWork From Home, United States R1h ago
-
AWS | Agentic AI | Angular | CI/CD | DatabricksHybrid work | Technical mentorshipSenior-level Full TimeNormal, United States5h ago
-
Sr. Data Engineer USD 108K-158KAWS | Apache Spark | Automated testing | Azure Event | Azure Event Hubs401k matching | Dental insurance | Disability insurance | Educational growth | Employee discount programSenior-level Full TimeNew York-TONAWANDA5h ago
-
Artificial Intelligence | Data Modeling | Data Pipelines | Data Quality | Data Visualization401k match | Dental insurance | Life insurance | Medical insurance | Paid time offSenior-level Full TimeNew York6h ago
-
AWS | AWS SDK | Amazon MSK | Apache Kafka | Generative AISenior-level Full TimeNew York, New York, United States7h ago
-
Production Engineer USD 178K-200KApache | Apache Spark | Application Programming | Application Programming Interfaces | C++Entry-level Full TimeMenlo Park, CA7h ago
-
Research Engineer - MSL FAIR Foundations USD 117K-173KBenchmarking | Code review | Data Pipelines | Distributed Systems | Language ModelEntry-level Full TimeMenlo Park, CA7h ago
-
Senior Software Engineer, AI/ML, Search Growth USD 174K-252KA/B | A/B Testing | B testing | Deep learning | Information RetrievalSenior-level Full TimeMountain View, CA, USA7h ago
-
Staff Software Engineer, Agentic Data and Evals USD 207K-300KC++ | CSS | Cloud | Data Storage | Data StructuresSenior-level Full TimeSunnyvale, CA, USA7h ago
-
.NET | App Service | Azure | Azure App | Azure App ServiceDental insurance | Employee assistance plan | Group medical insurance | Gym reimbursement | Health savings accountSenior-level Full TimeBoston, United States8h ago
-
Senior Data Engineer USD 155K-180KAWS | Agile | Alerting | Azure | Bash401k savings plan | Dental insurance | Employee assistance plan | Group medical insurance | Gym reimbursementSenior-level Full TimeBoston, United States9h ago
-
Data Engineer USD 118K-125KAWS | Agile | Apache Spark | Azure | Bash401k savings plan with profit sharing | Commuter parking tax savings | Employee assistance program | Group medical insurance | Gym reimbursementSenior-level Full TimeBoston, United States9h ago
-
Software Engineer, Applied AI USD 130K-500KData Pipelines | Data Quality | Evaluation Frameworks | Experimental Design | GoDental insurance | Equity grant | Free Equinox Membership | Health insurance | Housing bonusMid-level Full TimeSan Francisco12h ago
-
Staff Machine Learning Engineer USD 188K-221KAWS | Amazon Redshift | Amazon SageMaker | Apache Airflow | Apache FlinkGenerous time off | Healthcare | Paid parental leave | Paid personal time off | Paid sick timeSenior-level Full TimeRemote - US R15h ago
-
Staff Software Engineer, Data & AI USD 183K-214KAWS | Airflow | Analytics | Artificial Intelligence | BI AnalyticsSenior-level Full TimeCA - San Francisco; WA - …16h ago
-
ASR | Automatic Speech Recognition | CTC | Data Augmentation | Knowledge Distillation401k matching | Annual offsites | Dental insurance | Free snacks and drinks | Health insuranceSenior-level Full TimeSan Francisco, CA17h ago
-
Application Software Engineer, Data USD 150K-225KAngular | C# | CI/CD | Continuous integration | DeploymentSenior-level Full TimeStarbase, TX18h ago
-
Deployed Engineer (Denver) USD 150K-250KAI Agent | AI agent deployment | AWS | Agent deployment | Agent-based401-k plan | Dental coverage | Flexible vacation | Meals on in office days | Medical coverageSenior-level Full TimeDenver, CO18h ago
-
Deployed Engineer (Las Vegas) USD 150K-250KAWS | Azure | Docker | Failure handling | GCP401k plan | Dental insurance | Flexible vacation | Meals on in office days | Medical insuranceSenior-level Full TimeLas Vegas, NV18h ago
-
Deployed Engineer (Phoenix) USD 150K-250KAWS | Agent Frameworks | Azure | Cloud Computing | Containers401k plan | Dental insurance | Flexible vacation | Meals on in office days | Medical insuranceSenior-level Full TimePhoenix, AZ18h ago
-
Full Stack AI Developer USD 146K-222KAgile | Angular | Auto-tagging | CI/CD | Chunking401k | Education reimbursement program | Flexible schedule | Hybrid schedule | MentorshipSenior-level Full TimeLivermore, CA, United States R18h ago
-
Senior Data Engineer USD 239K-271KAirflow | Alerting | Amazon Redshift | Automated testing | Cost OptimizationFamily planning support | Flexible time off | Lifestyle stipend | Mental health support | Paid parental leaveSenior-level Full TimeSan Francisco, CA R18h ago
-
AI Security | API Security | Adversarial Machine Learning | Data exfiltration | Evasion TechniquesLife insurance | Mental health support | Private medical coverageMid-level Full TimePortland, Oregon, United States18h ago
-
Mid-level Full TimeUnited States - Remote R18h ago