Staff Engineer, Datalake Platform
Tasks
- Architect unified Iceberg platform
- Build developer facing client libraries
- Coordinate cross team migration planning
- Define API contracts for Iceberg
- Define access control policy design
- Design authorization model
- Design automated self service tooling for cost efficiency
- Design metastore service for Iceberg table management
- Design object storage abstraction layer
- Enforce data segregation and lifecycle controls
- Ensure backward compatibility and rollback
- Identify storage inefficiencies at petabyte scale
- Implement audit logging and access review infrastructure
- Implement per table credential vending
- Lead compliance architecture
- Mentor senior engineers through architecture decisions
- Migrate Hive Metastore workloads to new platform
- Own technical design reviews and reliability standards
Perks/Benefits
Skills/Tech-stack
Access Control | Amazon S3 | Apache Flink | Apache Iceberg | Apache Spark | Audit Logging | Authorization | Azure Blob | Azure Blob Storage | Blob Storage | Bucket Provisioning | Compaction | Compliance | Compute Engine | Compute Engine Integration | Data Integrity | Data Migration | Data Segregation | Distributed Storage | Distributed Systems | Engine integration | Hive Metastore | IAM | Lifecycle Management | Object storage | PyIceberg | REST API | Rollback Strategy | Snapshot management | Storage Abstraction | Trino
Education
N/A
Roles
Related jobs
-
Staff ML Engineer EUR 56K-84KAWS | Amazon SageMaker | Apache Airflow | Apache Flink | CI/CDHome office stipend | PTO | Paid parental leave | Professional development | Remote alwaysSenior-level Full TimeIreland R1d ago
-
Business Support Engineer EUR 49K-67KAPI Development | Agent Orchestration | Bias Mitigation | Cloud | Data Analysis24 7 Oncall RotationMid-level Full TimeDublin, Ireland1d ago
-
Automation | C plus plus | Capacity Planning | Distributed Systems | GoBlameless culture | Mentorship | Supportive teamSenior-level Full TimeDublin, Ireland1d ago
-
Mid-level Full TimeDublin, Ireland1d ago
-
Senior AI Engineer-1 USD 140K-254KAPIs | APIs development | Agentic Systems | Artificial Intelligence | Data PipelinesSenior-level Full TimeDublin, Ireland (Mountain View)1d ago
-
Senior Software Engineer, Analytics Platform EUR 56K-88KAWS | Ansible | Apache Druid | Apache Flink | Apache KafkaCareer development | Dental plan | Hybrid work options | Mentoring | Paid Maternity LeaveSenior-level Full TimeVirtual Office (Ireland) R1d ago
-
Distributed Systems | Embeddings | Java | Kubernetes | LLM InferenceExposure to cutting-edge technology | Flexible work schedule | High autonomy | Remote workEntry-level Full TimeIreland R2d ago
-
Automation | Capacity Planning | Cloud Computing | Distributed Systems | Fault ToleranceSenior-level Full TimeDublin, Ireland5d ago
-
Principal Data & AI Engineer EUR 70K-137KAWS | Airflow | Apache Spark | Automated testing | BigQuerySenior-level Full TimeIE007 Castletroy Limerick, Ireland5d ago
-
Senior-level Full TimeDublin, Ireland6d ago
-
Senior Data Engineer, Healthcare Economics EUR 66K-120KAWS | Agile | Apache Airflow | Apache Kafka | Apache SparkEmployee donations matching | Lunch provided | Matching Pension Contribution | Private health insurance | Training opportunitiesSenior-level Full TimePrimary location: Dublin, Leinster6d ago
-
SQL Developer / Core Banking Data Engineer EUR 85K-128KAWS | Airflow | Alerting | Automated testing | AzureMid-level Full TimeLoughlinstown, Ireland6d ago
-
Senior-level Full TimeDublin, County Dublin, Ireland7d ago
-
AI DevOps Engineer EUR 71K-90KAPI Development | AWS CloudFormation | AWS IAM | AWS Lambda | Access ControlCareer development | Educational supportSenior-level Full TimeDublin, Co. Dublin, Ireland8d ago
-
Senior AI Engineer EUR 54K-75KAPIs | Agentic Systems | Artificial Intelligence | Data Engineering | Distributed SystemsSenior-level Full TimeDublin, Ireland8d ago
-
AWS Glue | AWS S3 | AWS Step Functions | Amazon Athena | Amazon EMRAutonomy | International collaborative environment | Remote friendly working within EuropeMid-level Full TimeIreland9d ago
-
AI tools | Batch Processing | DBT | Data Engineering | Data MigrationAutonomy | Career growth opportunities | Inclusive culture | Learning and development | Supportive environmentSenior-level Full TimeIreland9d ago
-
Data Engineer EUR 56K-79KAWS Fargate | AWS Glue | AWS Lambda | Amazon Aurora | Amazon Aurora ServerlessAnnual leave | Career development | Company laptop | Company pension | E-learningSenior-level Full TimeDublin9d ago
-
Senior Data Engineer EUR 70K-81KAWS Glue | Apache Hadoop | Apache NiFi | Apache Spark | AzureEnhanced leave options | Health and wellbeing supports | Hybrid work model | Pension and savings options | Training and coachingSenior-level Full TimeDublin, Ireland12d ago
-
Staff AI Software Engineer EUR 86K-115KAPIs | AWS | Agentic AI | Azure | Cloud NativeContinuous learning | Mentorship | Work-life balanceSenior-level Full TimeIreland, Limerick12d ago
-
MES Data Engineer EUR 46K-69KChange Management | Compliance | MES | Manufacturing Operations | Quality managementCareer development | Family health insurance | Life assurance | Modern facility | Pension schemeMid-level Full TimeIreland - Kilkenny : Kilkenny Business …12d ago
-
Lead Data Engineer – AI & Foundation Models EUR 49K-69KAccess Control | Apache Spark | Audit Logging | CI/CD | Data GovernanceSenior-level Full TimeDublin, Ireland13d ago
-
Lead AI/ML Engineer EUR 85K-100KAPIs | Agent workflow | Artificial Intelligence | Azure | Continuous ImprovementSenior-level Full TimeDublin, Ireland14d ago
-
Staff Engineer EUR 85K-124KAWS | Azure | Backend Development | C# | Distributed SystemsHybrid work optionSenior-level Full TimeDublin15d ago
-
Senior Staff Engineer EUR 96K-124KAI Engineering | Availability | C++ | Code review | Cost OptimizationSenior-level Full TimeDublin15d ago