Principal Software Engineer - Data Platform (Iceberg/Trino)
United States - Remote
R
USD 160K-196K (estimate) Senior-level Full Time
Tasks
- Define placement triggers for VM versus Kubernetes
- Design lakehouse engine architecture for on premise
- Develop SQL dialect strategy for Trino and Spark SQL
- Implement change data capture pipelines
- Lead Iceberg maintenance for compaction snapshot expiry orphan cleanup
- Mentor engineers and review designs
- Own lakehouse reference architecture
- Partner on storage sizing resource isolation and capacity planning
- Set standards for table layout partitioning and file sizing
- Validate proof of concept for catalog and query engine
Perks/Benefits
Skills/Tech-stack
Apache Iceberg | Catalog Services | Change Data Capture | Compaction | Data Capture | Distributed SQL | Hive Metastore | Iceberg Catalog | Iceberg Catalog Services | Java | Kubernetes | Nessie | Object storage | Orphan File Cleanup | Partitioning | Performance Engineering | Polaris | Presto | Python | Query Planning | REST Catalog | S3-compatible | S3-compatible object storage | SQL | Snapshot Expiry | Spark SQL | Trino
Education
Bachelor of Engineering | Bachelor of Technology | Master of Science
Related jobs
-
AI Governance | AI guardrails | API Integration | Agent systems | AutoGPTLocal onsite days | Remote work 3 daysMid-level Full TimeAustin, TX R7h ago
-
AI Evaluation | AI Governance | AI guardrails | Agent systems | Agent workflowsSenior-level Full TimeAustin, TX R7h ago
-
Mid-level Full TimeAustin, TX R7h ago
-
Analytics Engineer USD 119K-234KAzure Data | Azure Data Factory | Azure SQL | Azure Synapse | Azure Synapse AnalyticsSenior-level Full TimeHouston, TX, United States R8h ago
-
Mid-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL R8h ago
-
Senior Research Engineer / Scientist -AI for Databases USD 207K-368KAWS | Architecture Search | Artificial Intelligence | Automated Model Tuning | C++401k match | Dental insurance | Disability coverage | Health insurance | Life insuranceSenior-level Full TimeSeattle, Washington, United States R8h ago
-
Research Engineer / Scientist -AI for Databases USD 153K-300KArchitecture Search | Artificial Intelligence | Automated Database Tuning | C++ | Database Internals401k match | Life insurance | Long-Term Disability coverage | Medical/Dental/Vision insurance | Paid HolidaysMid-level Full TimeSeattle, Washington, United States R9h ago
-
A/B | A/B Testing | Apache Airflow | Apache Kafka | Apache Spark401k match | Life insurance | Long-term disability | Medical, dental, and vision insurance | Paid HolidaysMid-level Full TimeSan Jose, California, United States R10h ago
-
Quant Analytics [Multiple Positions Available] USD 158K-201KAWS | Alteryx | Association Rule Mining | Association rule | Behavioral analyticsBackup childcare | Comprehensive health care coverage | Domestic travel | Financial coaching | Mental health supportSenior-level Full TimeColumbus, OH, United States R10h ago
-
Research Engineer / Scientist -AI for Databases USD 162K-387KAWS | Architecture Search | Artificial Intelligence | Automated Database Tuning | Azure401k match | Conference attendance | Dental insurance | Life insurance | Long-term disabilityMid-level Full TimeSan Jose, California, United States R10h ago
-
Head of Data Platform USD 176K-271KAI Governance | API Design | AWS | Access Control | Batch Processing401k employer match | Hybrid work | Life insurance | Medical/Dental/Vision insurance | Paid time offExecutive-level Full TimeHOUSTON, US, 77056 R14h ago
-
C# | C++ | Computer Graphics | Computer Vision | Data PipelinesEquity compensation | Health benefits | Onsite work schedule Tuesday Wednesday ThursdaySenior-level Full TimeSan Mateo, CA, United States R16h ago
-
Staff Machine Learning Engineer, Ad Serving USD 195K-510KA/B | A/B Testing | B testing | Continuous Improvement | Feature EngineeringCommuter benefits | Disability benefits | Financial wellness support | Health insurance | Life insuranceSenior-level Full TimeNew York, New York R18h ago
-
Staff Machine Learning Engineer, Ad Serving USD 195K-435KDistributed Systems | Experimentation | Feature Engineering | Java | Machine LearningDisability benefits | Equity awards | Health insurance | Life insurance | Paid time offSenior-level Full TimeBoston, Massachusetts R18h ago
-
AI Agents | AI monitoring | API Integration | Amazon Web Services | Anthropic401k retirement plan | Medical, dental, and vision benefits | Paid Holidays | Paid time off | Variable pay/incentivesSenior-level Full TimeUnited States R19h ago
-
Lead Data Engineer USD 164K-200KAPIs | AWS | Agentic AI | Azure | Batch dataDisability and life insurance | Employee assistance program | Fertility and family forming benefits | Flexible spending account | Healthcare conciergeSenior-level Full TimeHeadquarters, United States R23h ago
-
Pricipal Data Scientist USD 99K-156KData Engineering | Deep learning | Machine Learning | NumPy | PandasHealth and wellbeing resources | Paid volunteer days | Remote work flexibilityEntry-level Full TimeSan Francisco - United States - … R23h ago
-
Data Engineer USD 114K-126KAPI Integration | AWS | AWS Glue | AWS Lambda | AWS Step Functions401k company match | Free flights | Health insurance for dependents | Profit sharing | Remote workMid-level Full TimeTX-Headquarters-HDQ, United States R23h ago
-
Forward Deployed Engineer (FDE) Data42 USD 160K-297KApplication development | Artificial Intelligence | Auditability | Data Governance | Data IngestionSenior-level Full TimeCambridge (USA), United States R23h ago
-
Forward Deployed Engineer (FDE) Data42 USD 160K-297KArtificial Intelligence | Data Governance | Data Modeling | Data Pipelines | Data QualitySenior-level Full TimeCambridge (USA), United States R23h ago
-
Data Engineer USD 73K-99KAzure Data | Azure Data Factory | Azure Synapse | CI/CD | Data Factory401k with profit sharing | Career growth opportunities | Dress for your day policy | Employee resource groups | Flexible working hoursMid-level Full TimeCleveland, United States R23h ago
-
Associate Data and AI Engineer USD 74K-111KAWS | AWS Lambda | Agile | Airflow | Artificial IntelligenceHybrid work arrangement | Remote work optionMid-level Full TimeHartford CT- Home Office, United States R23h ago
-
Unified Data Platform Technical Development Lead USD 107K-195KAPIs | AWS | Agile | Ansible | Apache KafkaSenior-level Full Time6314 Remote/Teleworker US, United States R23h ago
-
Principal AI Software Engineer, GTM USD 194K-263KAWS | AWS CDK | AWS Lambda | Airflow | Amazon BedrockSenior-level Full TimeUnited States R23h ago
-
Sr. Analytics Engineer USD 160K-185KAirflow | CI/CD | DBT | Dagster | Data ModelingCross-functional collaboration | High autonomy | US Pacific overlapSenior-level Full TimeRemote - US R23h ago