Lead Software Engineer - Databricks, Spark, AWS
Plano, TX, United States
USD 179K-215K (estimate) Senior-level Full Time
Tasks
- Build data ingestion and transformation frameworks
- Build reusable Python and Java libraries and APIs
- Deploy infrastructure with Terraform
- Design secure data ingestion and transformation
- Drive Spark performance engineering
- Enforce data quality, lineage, and governance
- Enforce encryption and least privilege access
- Establish lakehouse patterns with Delta Lake
- Establish observability and runbooks
- Implement CI/CD for data projects
- Integrate AWS eventing and orchestration
- Lead data pipeline architecture and delivery
- Lead incident response and root cause analysis
- Orchestrate jobs with Databricks Workflows
- Oversee unit and integration testing
- Own Databricks cluster strategy and setup
Perks/Benefits
- Backup childcare
- Financial coaching
- Health care coverage
- Mental health support
- On-site health and wellness centers
- Retirement savings plan
- Tuition reimbursement
Skills/Tech-stack
AWS CloudWatch | AWS Glue | AWS IAM | AWS Lambda | AWS S3 | AWS Secrets | AWS Secrets Manager | Amazon Kinesis | Amazon MSK | Apache Spark | CI/CD | Data Governance | Data Lineage | Data Quality | Databricks | Delta Lake | Git | JUnit | Java | Kafka | Pytest | Python | SQL | Secrets Manager | Spark SQL | Structured Streaming | Terraform | Unity Catalog
Education
N/A
Roles
Engineer | Lead | Lead Software Engineer | Software Engineer
Related jobs
-
ANSYS Mechanical | Ansys Twin Builder | C++ | Controls | GazeboFinancial benefits | Health benefits | Remote work | Wellness benefitsEntry-level InternshipCalifornia, US (California) Off-Site2h ago
-
Robotics Engineer - Autonomous Drone System USD 167K-230KAnalytics | Anomaly Detection | Autonomous Navigation | BVLOS | C++Senior-level Full TimeFremont, CA4h ago
-
Robotics Control Engineer - Manipulation USD 162K-240KABB Rapid | C++ | Calibration | Cause analysis | CommissioningSenior-level Full TimeMenlo Park, CA | Fremont, CA4h ago
-
Robotics Firmware & Electrical Engineer USD 170K-240KC# | C++ | CAN | Electrical layout | Embedded SystemsSenior-level Full TimeFremont, CA4h ago
-
Production Engineer USD 147K-208KAI workflow | AI workflow automation | Agent Orchestration | C++ | CDNSenior-level Full TimeMenlo Park, CA4h ago
-
Abuse prevention | Adversarial Machine Learning | Agentic Models | Benchmarks | Code reviewMid-level Full TimeMenlo Park, CA4h ago
-
Senior Research Software Engineer, Labs, AI/ML USD 174K-252KC++ | Data Structures | Data Structures and Algorithms | Deep learning | Distributed ComputingSenior-level Full TimeMountain View, CA, USA; Zürich, Switzerland4h ago
-
Software Engineer III, AI/ML, Google Cloud AI USD 147K-211KC++ | Data Processing | Debugging | GenAI | Google CloudSenior-level Full TimeSunnyvale, CA, USA4h ago
-
AI Agents | AppScript | Artificial Intelligence | Business Intelligence | Change ManagementSenior-level Full TimeSan Bruno, CA, USA4h ago
-
Senior Engineering Analyst, AI, Safety USD 159K-231KAbuse detection | C++ | Data Analysis | Fine Tuning | Fraud DetectionEquity compensation | Health insurance | Mentorship | Paid time off | Retirement benefitsSenior-level Full TimeKirkland, WA, USA; Seattle, WA, USA4h ago
-
Staff Software Engineer, Biglakes, BigQuery USD 207K-300KApache Arrow | Apache Flink | Apache Spark | Big Data | Big data analyticsSenior-level Full TimeSunnyvale, CA, USA; Kirkland, WA, USA4h ago
-
Senior Software Engineer, ML Performance Infrastructure USD 174K-252KC# | C++ | Compute technology | Data Storage | Data StructuresSenior-level Full TimeSunnyvale, CA, USA4h ago
-
Forward Deployed Engineer I, GenAI, Google Cloud USD 102K-145KAPI Integration | Accuracy evaluation | Agent systems | Cloud platform | Cost Per RequestSenior-level Full TimeSan Francisco, CA, USA; Atlanta, GA, …4h ago
-
Senior Software Engineer, Jules AI, Labs, AI/ML USD 174K-252KAI infrastructure | Agent systems | Context engineering | Distributed Computing | Generative AISenior-level Full TimeMountain View, CA, USA; New York, …4h ago
-
Senior Software Engineer, Connect Sales, CRM, GenAI, Ads USD 174K-252KComputer Vision | Data Processing | Debugging | Distributed Computing | Generative AISenior-level Full TimeMountain View, CA, USA4h ago
-
AI Engineer USD 120K-173KArtificial Intelligence | CI/CD | Data Preprocessing | Feature Engineering | MLOpsEmployee resource groups | Flexible family care | Hybrid work | Medical dental prescription coverage | Paid HolidaysSenior-level Full TimeDearborn, MI, United States12h ago
-
Senior Software Engineer, AI Training & Infrastructure USD 200K-300KAWS | Alerting | Algorithms | Automated testing | AzureSenior-level Full TimeSan Mateo13h ago
-
Sr. Embedded Detection Analyst USD 170K-205KAI tools | Alert Correlation | Cause analysis | Data Analysis | Detection engineeringSenior-level Full TimeRemote - USA R13h ago
-
ML Engineer USD 190K-320KCost Optimization | Data Curation | Dataset versioning | Evaluation Frameworks | Fine Tuning401k employer match | Employee assistance program | Employer paid dental insurance | Employer paid vision insurance | Employer-paid health insuranceSenior-level Full TimeSan Francisco14h ago
-
Software Engineer - Tools & Automation USD 160K-226KArtificial Intelligence | Backend Development | Benchmarking | Context Management | Data PipelinesMid-level Full TimeFoster City, CA15h ago
-
Senior-level Full TimeSan Mateo16h ago
-
Senior-level Full TimeAlameda, CA16h ago
-
Senior Data Engineer USD 160K-180KAWS | Amazon Redshift | Amazon S3 | Apache Airflow | CI/CD401k match | Community service events | Employee assistance program | Employee discounts | Health, dental, vision benefitsSenior-level Full TimeEnglewood Cliffs, NJ, US R16h ago
-
Senior-level Full TimeUnited States17h ago
-
Senior-level Full TimeUnited States17h ago