Lead Software Engineer - Databricks, Spark, AWS
Plano, TX, United States
USD 179K-215K (estimate) Senior-level Full Time
Tasks
- Build data ingestion and transformation frameworks
- Build reusable Python and Java libraries and APIs
- Deploy infrastructure with Terraform
- Design secure data ingestion and transformation
- Drive Spark performance engineering
- Enforce data quality, lineage, and governance
- Enforce encryption and least privilege access
- Establish lakehouse patterns with Delta Lake
- Establish observability and runbooks
- Implement CI/CD for data projects
- Integrate AWS eventing and orchestration
- Lead data pipeline architecture and delivery
- Lead incident response and root cause analysis
- Orchestrate jobs with Databricks Workflows
- Oversee unit and integration testing
- Own Databricks cluster strategy and setup
Perks/Benefits
- Backup childcare
- Financial coaching
- Health care coverage
- Mental health support
- On-site health and wellness centers
- Retirement savings plan
- Tuition reimbursement
Skills/Tech-stack
AWS CloudWatch | AWS Glue | AWS IAM | AWS Lambda | AWS S3 | AWS Secrets | AWS Secrets Manager | Amazon Kinesis | Amazon MSK | Apache Spark | CI/CD | Data Governance | Data Lineage | Data Quality | Databricks | Delta Lake | Git | JUnit | Java | Kafka | Pytest | Python | SQL | Secrets Manager | Spark SQL | Structured Streaming | Terraform | Unity Catalog
Education
N/A
Roles
Engineer | Lead | Lead Software Engineer | Software Engineer
Related jobs
-
Generative AI Consultant USD 105K-105KAWS | Azure | CI/CD | Deep learning | Docker401k matching | Health insurance | Paid time off | Parental leave | Professional developmentMid-level Full TimeSan Francisco, CA, United States6h ago
-
Manufacturing Master Data Engineer USD 90K-112KBills of Material | Cause analysis | Change Management | Compliance | Data Cleansing401k matching | Cross training | Dental insurance | Medical insurance | On-the-job trainingMid-level Full TimeWichita, KS, United States12h ago
-
AI | AWS | Agile | Architecture Design | CI/CDHybrid work | Onsite 3 days per week | W2 employmentSenior-level Contract Full TimeIrvine, CA, United States R13h ago
-
Senior Data Engineer USD 155K-180KAWS | Agile | Azure | Bash | Data Lakes401k savings plan | Commuter and parking tax savings benefit | Dental insurance | Employee assistance plan | Group medical insuranceSenior-level Full TimeBoston, United States14h ago
-
Data Architect USD 135K-165KAPI Integration | API Keys | API Management | Amazon Kinesis | Amazon Web ServicesHighly cleared team support | Security-focused environmentSenior-level Full TimeOther, VA15h ago
-
Lead Data Engineer – Azure Databricks USD 150K-180KARIMA | Azure Databricks | Data Processing | Data analytics | Machine LearningOnsite workSenior-level Full TimeTampa, United States16h ago
-
Mid-level Full TimeAuburn Hills, MI, United States16h ago
-
HR Analytics & Technology Engineer USD 110K-204KAlteryx | Artificial Intelligence | Data Governance | Data Modeling | Data Quality401k matching | Generous time off | Onsite fitness center | Tuition reimbursementSenior-level Full TimeUS-Rhode Island-Providence16h ago
-
Cause analysis | Cloud Computing | Data Enrichment | Data Monitoring | Data Processing401k match | Dental insurance | Health insurance | Life insurance | Paid HolidaysSenior-level Full TimeNew York16h ago
-
Deloitte Technology - US Associate GenAI Engineer USD 80K-100K.Net Core | AWS | Agile | Angular | AzureMid-level Full TimeHermitage, Tennessee, United States17h ago
-
Senior-level Full TimeDallas, Texas, United States17h ago
-
As-a-Service | C++ | Cloud Functions | Cloud platform | Compute TechnologiesSenior-level Full TimeSunnyvale, CA, USA18h ago
-
Research Engineer, Gemini Latent Thinking, DeepMind USD 207K-300KAblation Study | Algorithm Development | Deep learning | Experiment design | Language ModelsSenior-level Full TimeCambridge, MA, USA; Mountain View, CA, …18h ago
-
Software Engineer III, Infrastructure, Cloud Storage USD 147K-211KC++ | Cloud Storage | Compute Technologies | Continuous integration | Data StorageSenior-level Full TimeSeattle, WA, USA18h ago
-
Manufacturing Test Engineer, Data Center USD 120K-172KBSCAN | Bash | Bring-up | Calibration | Correlation studyMid-level Full TimeAtlanta, GA, USA18h ago
-
Software Engineer, Compiling, Quantum AI USD 147K-211KC++ | Classical coding theory | Coding theory | Compiler architecture | CompilersMid-level Full TimeMountain View, CA, USA; Los Angeles, …18h ago
-
Research Scientist, Robotics, Embodied AI, DeepMind USD 147K-211KDeep learning | Language Models | Machine Learning | Python | Reinforcement LearningSenior-level Full TimeMountain View, CA, USA18h ago
-
Software Engineer, Decoding, Quantum AI USD 147K-211KC++ | Concurrency | Device Driver Development | Device driver | Driver DevelopmentMid-level Full TimeLos Angeles, CA, USA; Goleta, CA, …18h ago
-
C# | C++ | Co-design | Compiler technology | Computer ArchitectureSenior-level Full TimeSunnyvale, CA, USA18h ago
-
Kinetic Machine Learning Engineer USD 90K-123KClustering | Data Quality | Data quality profiling | Drift monitoring | Ensemble learning401k | Dental insurance | Educational assistance | Flexible savings account | Health savings accountMid-level Full TimeUnited States19h ago
-
AWS S3 | Almabase | Amazon Athena | Azure Data | Azure Data LakeMid-level Full TimeRemote within Texas, TX, US R21h ago
-
AI Data Platform Lead USD 164K-229KAWS | Airflow | Audit Logging | DBT | Data GovernanceFloating holidays | Wellness daySenior-level Full TimeUnited States1d ago
-
ALB | ALB/NLB | AWS | AWS CDK | AWS CloudSenior-level Full TimeDallas, Texas, United States1d ago
-
Statistics & Python Expert - Freelance AI Trainer USD 146K-146KCombinatorics | Graph theory | MATLAB | NumPy | Number theoryFlexible hours | Freelance opportunities | Project based workSenior-level FreelanceNew York, New York, United States … R1d ago
-
Statistics & Python Expert - Freelance AI Trainer USD 146K-146KCombinatorics | Graph theory | Mathematics | NumPy | Number theoryFreelance opportunity | Part-time project-based workSenior-level FreelanceFlorida, United States - Remote R1d ago