Lead Software Engineer - Databricks, Spark, AWS
Plano, TX, United States
USD 179K-215K (estimate) Senior-level Full Time
Tasks
- Build data ingestion and transformation frameworks
- Build reusable Python and Java libraries and APIs
- Deploy infrastructure with Terraform
- Design secure data ingestion and transformation
- Drive Spark performance engineering
- Enforce data quality, lineage, and governance
- Enforce encryption and least privilege access
- Establish lakehouse patterns with Delta Lake
- Establish observability and runbooks
- Implement CI/CD for data projects
- Integrate AWS eventing and orchestration
- Lead data pipeline architecture and delivery
- Lead incident response and root cause analysis
- Orchestrate jobs with Databricks Workflows
- Oversee unit and integration testing
- Own Databricks cluster strategy and setup
Perks/Benefits
- Backup childcare
- Financial coaching
- Health care coverage
- Mental health support
- On-site health and wellness centers
- Retirement savings plan
- Tuition reimbursement
Skills/Tech-stack
AWS CloudWatch | AWS Glue | AWS IAM | AWS Lambda | AWS S3 | AWS Secrets | AWS Secrets Manager | Amazon Kinesis | Amazon MSK | Apache Spark | CI/CD | Data Governance | Data Lineage | Data Quality | Databricks | Delta Lake | Git | JUnit | Java | Kafka | Pytest | Python | SQL | Secrets Manager | Spark SQL | Structured Streaming | Terraform | Unity Catalog
Education
N/A
Roles
Engineer | Lead | Lead Software Engineer | Software Engineer
Related jobs
-
AWS | Alerting | CI/CD | Compliance | Data LakeCareer growth | Dental insurance | Medical insurance | Paid leave | Retirement plansExecutive-level Full TimeHouston, TX, United States1h ago
-
Senior Data Engineer Principal & Lead –on Site USD 165K-210KAWS Lambda | Amazon Kinesis | Amazon Redshift | Amazon S3 | Data ModelingDay 1 impact | On-site role | Own product and infrastructure | Shape technical vision | Small senior teamSenior-level Full TimeSan Francisco, United States4h ago
-
Algorithms | C++ | Computational lithography | Data Structures | Image ProcessingFinancial benefits | Health benefits | Wellness programsSenior-level Full TimeCalifornia, Sunnyvale4h ago
-
ANSYS Mechanical | Ansys Twin Builder | C++ | Controls | GazeboFinancial benefits | Health benefits | Remote work | Wellness benefitsEntry-level InternshipCalifornia, US (California) Off-Site4h ago
-
Software Engineer, Backend and Infrastructure USD 156K-316KAlerting | CI/CD | Database Design | Distributed Systems | HDFSMid-level Full TimeSan Jose, California, United States4h ago
-
Senior Software Engineer, Backend and Infrastructure USD 156K-387KAlerting | CI/CD | Cloud IDE | Database | HDFSSenior-level Full TimeSan Jose, California, United States4h ago
-
Benchmarking | Data Pipelines | Evaluation metrics | Interpretability | LLM EvaluationSenior-level Full TimeSan Jose, California, United States4h ago
-
Robotics Engineer - Autonomous Drone System USD 167K-230KAnalytics | Anomaly Detection | Autonomous Navigation | BVLOS | C++Senior-level Full TimeFremont, CA5h ago
-
Robotics Control Engineer - Manipulation USD 162K-240KABB Rapid | C++ | Calibration | Cause analysis | CommissioningSenior-level Full TimeMenlo Park, CA | Fremont, CA5h ago
-
Robotics Firmware & Electrical Engineer USD 170K-240KC# | C++ | CAN | Electrical layout | Embedded SystemsSenior-level Full TimeFremont, CA5h ago
-
Production Engineer USD 147K-208KAI workflow | AI workflow automation | Agent Orchestration | C++ | CDNSenior-level Full TimeMenlo Park, CA5h ago
-
Abuse prevention | Adversarial Machine Learning | Agentic Models | Benchmarks | Code reviewMid-level Full TimeMenlo Park, CA5h ago
-
Senior Research Software Engineer, Labs, AI/ML USD 174K-252KC++ | Data Structures | Data Structures and Algorithms | Deep learning | Distributed ComputingSenior-level Full TimeMountain View, CA, USA; Zürich, Switzerland5h ago
-
Software Engineer III, AI/ML, Google Cloud AI USD 147K-211KC++ | Data Processing | Debugging | GenAI | Google CloudSenior-level Full TimeSunnyvale, CA, USA5h ago
-
AI Agents | AppScript | Artificial Intelligence | Business Intelligence | Change ManagementSenior-level Full TimeSan Bruno, CA, USA5h ago
-
Senior Engineering Analyst, AI, Safety USD 159K-231KAbuse detection | C++ | Data Analysis | Fine Tuning | Fraud DetectionEquity compensation | Health insurance | Mentorship | Paid time off | Retirement benefitsSenior-level Full TimeKirkland, WA, USA; Seattle, WA, USA5h ago
-
Staff Software Engineer, Biglakes, BigQuery USD 207K-300KApache Arrow | Apache Flink | Apache Spark | Big Data | Big data analyticsSenior-level Full TimeSunnyvale, CA, USA; Kirkland, WA, USA5h ago
-
Senior Software Engineer, ML Performance Infrastructure USD 174K-252KC# | C++ | Compute technology | Data Storage | Data StructuresSenior-level Full TimeSunnyvale, CA, USA6h ago
-
Forward Deployed Engineer I, GenAI, Google Cloud USD 102K-145KAPI Integration | Accuracy evaluation | Agent systems | Cloud platform | Cost Per RequestSenior-level Full TimeSan Francisco, CA, USA; Atlanta, GA, …6h ago
-
Senior Software Engineer, Jules AI, Labs, AI/ML USD 174K-252KAI infrastructure | Agent systems | Context engineering | Distributed Computing | Generative AISenior-level Full TimeMountain View, CA, USA; New York, …6h ago
-
Senior Software Engineer, Connect Sales, CRM, GenAI, Ads USD 174K-252KComputer Vision | Data Processing | Debugging | Distributed Computing | Generative AISenior-level Full TimeMountain View, CA, USA6h ago
-
Customer Success Physicist- Maryland USD 122K-193KControl Systems | Experimental quantum information | Experimental quantum information science | Information science | PythonTravel for customer engagementMid-level Full TimeMaryland, US8h ago
-
AI Engineer USD 120K-173KArtificial Intelligence | CI/CD | Data Preprocessing | Feature Engineering | MLOpsEmployee resource groups | Flexible family care | Hybrid work | Medical dental prescription coverage | Paid HolidaysSenior-level Full TimeDearborn, MI, United States14h ago
-
Senior Software Engineer, AI Training & Infrastructure USD 200K-300KAWS | Alerting | Algorithms | Automated testing | AzureSenior-level Full TimeSan Mateo15h ago
-
Sr. Embedded Detection Analyst USD 170K-205KAI tools | Alert Correlation | Cause analysis | Data Analysis | Detection engineeringSenior-level Full TimeRemote - USA R15h ago