Lead Software Engineer - Databricks/Spark/AWS
Tasks
- Build reusable libraries and APIs with testing
- Design secure data ingestion and transformation frameworks
- Drive Spark performance engineering
- Enforce data quality, lineage, and governance
- Establish lakehouse patterns with Delta Lake
- Implement CI/CD for data projects
- Lead architecture and delivery of data pipelines
- Orchestrate Databricks workflows and integrate AWS eventing
- Own Databricks cluster strategy and setup
Perks/Benefits
- Backup childcare
- Financial coaching
- Health care coverage
- Mental health support
- On-site health and wellness centers
- Retirement savings plan
- Tuition reimbursement
Skills/Tech-stack
AQE | AWS CloudWatch | AWS Glue | AWS IAM | AWS Lambda | AWS S3 | AWS Secrets | AWS Secrets Manager | Amazon Kinesis | Amazon MSK | Apache Spark | Autoscaling | Broadcast joins | CI/CD | Data Governance | Data Lineage | Databricks | Delta Lake | Delta Lake ACID | Delta Live Tables | Delta Live) | Git | JUnit | Java | Kafka | Pytest | Python | SQL | Secrets Manager | Shuffle Tuning | Spark SQL | Spark Streaming | Streaming SQL | Structured Streaming | Structured Streaming SQL | Terraform | Unity Catalog
Education
N/A
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R9d ago
-
AI | Alerting | Dashboards | Data Architecture | Data Governance401k match | Dental insurance | Life insurance | Long-term disability | Medical insuranceSenior-level Full TimeNew York3h ago
-
Senior Data Engineer USD 133K-158KAKS | Azure Data | Azure Data Factory | Azure Data Lake | Azure DatabricksSenior-level Full TimeCincinnati, Ohio, United States4h ago
-
Mid-level Full TimeAtlanta, Georgia, United States; Dallas, Texas, …4h ago
-
Entry-level Full TimeChicago, Illinois, United States; Dallas, Texas, …4h ago
-
Apache Flink | Apache Hudi | Apache Iceberg | Apache Paimon | C plus plusEntry-level Full TimeSan Jose, California, United States5h ago
-
Research Scientist, Generative AI - San Jose USD 194K-410K3D Pose Estimation | C++ | Computer Vision | Diffusion Models | Facial AnalysisMid-level Full TimeSan Jose, California, United States5h ago
-
Senior-level Full TimeSan Jose, California, United States5h ago
-
Research Scientist, ML H-W/S-W Codesign USD 143K-208KAlgorithm Design | Benchmarks | C plus plus | C# | Computer VisionEntry-level Full TimeSunnyvale, CA6h ago
-
Senior Software Engineer, Map Ads Machine Learning USD 174K-253KAds serving | C++ | Data Processing | Data Structures | Data structures algorithmsSenior-level Full TimeMountain View, CA, USA6h ago
-
Software Engineer, AI/ML, Full Stack, Search Ads USD 147K-211KA/B | A/B Testing | Algorithms | Artificial Intelligence | B testingMid-level Full TimeMountain View, CA, USA6h ago
-
Analytical Lead, YouTube Brand Deals USD 152K-222KAI | Dashboards | Data Analysis | Data Modeling | Data PipelinesSenior-level Full TimeSan Bruno, CA, USA; New York, …6h ago
-
Senior Software Engineer, AI/ML, Creative Intelligence USD 174K-253KAd systems | C++ | Data Processing | Data Structures | Data Structures and AlgorithmsSenior-level Full TimeMountain View, CA, USA6h ago
-
Principal Engineer, Agentic AI Data and Insights USD 307K-428KANSI SQL | APIs | Agentic AI | Apache Iceberg | Apache SparkSenior-level Full TimeIrvine, CA, USA; San Jose, CA, …6h ago
-
Mid-level Full TimeMcCarran, NV11h ago
-
Sr. Solutions Engineer USD 180K-247KAmazon Web Services | Apache Hadoop | Apache Kafka | Apache Spark | AzureFlexible work arrangements | Travel reimbursementSenior-level Full TimeUnited States13h ago
-
Embedded Software Engineer, C++ USD 166K-220KARM | Bootloader | C# | C++ | CANHealth benefits | Recovery Benefits | Time off | U S Secret Security Clearance eligibilityMid-level Full TimeLexington, Massachusetts, United States13h ago
-
Senior Robotics Software Engineer USD 190K-286KAlgorithms | Artificial Intelligence | C++ | Code optimization | ConcurrencyRelocation assistanceSenior-level Full TimeAtlanta, Georgia, United States13h ago
-
Robotics Software Engineer USD 165K-218KAgent systems | Algorithms | Artificial Intelligence | C++ | Computer VisionRelocation assistanceMid-level Full TimeAtlanta, Georgia, United States13h ago
-
Analytics Engineer, Hardware Test USD 146K-194KData Modeling | Data Quality | DataFrame | Ontology Modeling | PandasMid-level Full TimeCosta Mesa, California, United States13h ago
-
Mid-level Full TimeRaleigh, NC, US15h ago
-
AI Engineer (Org Wide) USD 119K-235KAI Builder | AI Foundry | AI Search | Azure AI | Azure AI FoundrySenior-level Full TimeFond du Lac, WI, US15h ago
-
Lead AI Architect - Data Platform USD 180K-240KAWS | AWS CDK | Airbyte | Amazon Bedrock | Apache AirflowSenior-level Full TimeUnited States15h ago
-
Engineering Manager, Enterprise Applied AI USD 300K-520KAPIs | Agent Orchestration | Benchmarking | Cloud Native | Data SourcesDental insurance | Equity grant | Free gym membership | Health insurance | Laundry reimbursementMid-level Full TimeSan Francisco15h ago
-
Lead Software Engineer - Machine Learning USD 170K-215KAI Assisted Development | AWS | Automated testing | Cause analysis | Cloud NativeSenior-level Full TimeJersey City, NJ, United States16h ago