Staff Engineer, Datalake Platform
Tasks
- Architect unified Iceberg platform
- Build developer facing client libraries
- Coordinate cross team migration planning
- Define API contracts for Iceberg
- Define access control policy design
- Design authorization model
- Design automated self service tooling for cost efficiency
- Design metastore service for Iceberg table management
- Design object storage abstraction layer
- Enforce data segregation and lifecycle controls
- Ensure backward compatibility and rollback
- Identify storage inefficiencies at petabyte scale
- Implement audit logging and access review infrastructure
- Implement per table credential vending
- Lead compliance architecture
- Mentor senior engineers through architecture decisions
- Migrate Hive Metastore workloads to new platform
- Own technical design reviews and reliability standards
Perks/Benefits
Skills/Tech-stack
Access Control | Amazon S3 | Apache Flink | Apache Iceberg | Apache Spark | Audit Logging | Authorization | Azure Blob | Azure Blob Storage | Blob Storage | Bucket Provisioning | Compaction | Compliance | Compute Engine | Compute Engine Integration | Data Integrity | Data Migration | Data Segregation | Distributed Storage | Distributed Systems | Engine integration | Hive Metastore | IAM | Lifecycle Management | Object storage | PyIceberg | REST API | Rollback Strategy | Snapshot management | Storage Abstraction | Trino
Education
N/A
Roles
Related jobs
-
Machine Learning Engineer EUR 69K-100KAPI Integration | Algorithms | Amazon S3 | Apache Spark | Azure DataMid-level Full TimeDublin, Ireland1d ago
-
AI engineer II EUR 49K-67KAPIs | Data Preparation | Deep learning | Distributed Systems | ExperimentationMid-level Full TimeDublin, Ireland1d ago
-
Lead AI engineer EUR 48K-62KAPIs | Agentic Systems | Artificial Intelligence | Data Pipelines | Data PreparationSenior-level Full TimeDublin, Ireland1d ago
-
AWS | Access Control | Alerting | Automated testing | CI/CDSenior-level Full TimeDublin, Ireland1d ago
-
Algorithms | Concurrency | Data Structures | Distributed Systems | ElasticsearchDonation matching for financial donations | Flexible location | Flexible work schedule | Health coverage | Parental leaveSenior-level Full TimeIreland R2d ago
-
Algorithms | Concurrency | Data Structures | Database systems | Distributed SystemsDonations match | Flexible schedule | Health coverage | Parental leave | Vacation daysSenior-level Full TimeIreland R2d ago
-
Senior Data Engineer GBP 91K-124KAccess Control | Airflow | BigQuery | CI/CD | Census401k employer match | Hackathons | Health and wellness benefits | Learning and development budget | Open source support fundSenior-level Full TimeGermany R2d ago
-
Platform Engineer - DevOps & Data Pipelines EUR 65K-77KARM Templates | Access Control | Automated testing | Azure Virtual | Azure Virtual DesktopMid-level Full TimeMullingar, Ireland2d ago
-
AI Engineer EUR 42K-70KAI Observability | Access Control | Anthropic Claude | Azure AI | Azure OpenAIMid-level Full TimeMullingar, Ireland2d ago
-
Principal AI/ML Scientist & Engineer EUR 54K-75KAWS | Agentic Workflows | Apache Kafka | Artificial Intelligence | Batch PipelineCompassionate leave | Flexible hours | Paid leave | Pawternal leave | Performance-based bonusesSenior-level Full TimeIreland, Dublin2d ago
-
Senior Data Engineer EUR 71K-125KAWS | Agile | Apache Airflow | Apache Hadoop | Apache KafkaEmployee donations matching | Lunch provided | Pension matching | Private health insurance | Training opportunitiesSenior-level Full TimePrimary location: Dublin, Ireland2d ago
-
AWS Glue | AWS IAM | Access Control | Alerting | Amazon EC2Senior-level Contract Full Time TemporaryIreland - Dublin College Park5d ago
-
AWS CloudWatch | AWS Glue | AWS IAM | Access Control | AlertingSenior-level Contract Full Time TemporaryLocation: Dublin, Ireland5d ago
-
Automation | Compliance | Cost Optimization | Evaluation | GuardrailsEfficient interview process | Fast decision making | International team environment | Ownership and autonomy | Rapid professional growthMid-level Full TimeIreland6d ago
-
Auto Scaling | Blue-Green Deployment | Blue/green | CI/CD | CUDAFlexible work schedule | Fully remote work | Ownership and influence | Remote-first environment | Startup growth opportunitiesSenior-level Full TimeIreland R6d ago
-
AWS | Asynchronous programming | Data Processing | Databases | Distributed SystemsAsynchronous work environment | Autonomy and work-life balance | Career development and internal growth | Company provided hardware and equipment | Remote-first flexibilityMid-level Full TimeIreland7d ago
-
Lead Data Analytics & AI Engineer EUR 78K-117KAPI | AWS | Amazon Kinesis | Apache Airflow | Apache FlinkHealth insurance | Hybrid work | Long-term illness protection | Pension | Referral bonusSenior-level Full TimeIE Dublin, Ireland R7d ago
-
Staff AI Software Platform Engineer EUR 54K-75KAWS | CI/CD | Cloud platform | CloudFormation | Distributed SystemsSenior-level Full TimeIreland, Limerick7d ago
-
EMEA Data & Digital Program Manager EUR 44K-55KAI Vision | AWS CDK | AWS CloudFormation | AWS Glue | AWS LambdaMid-level Full TimeDublin, IRL8d ago
-
AWS | Analytics | Apache Hive | Apache Kafka | Apache SparkBike to work scheme | Flexible working | Free gym membership | Health insurance | Hybrid workingSenior-level Full TimeDublin 2, IE8d ago
-
AWS Glue | AWS Lambda | AWS Redshift | AWS S3 | AnsibleVisa sponsorshipSenior-level Full TimeDublin 2, IE8d ago
-
AI Infrastructure Engineer - EA Experiences EUR 54K-75KAIOps | AWS | AWS CDK | Airflow | Amazon BedrockComplimentary games | Family leave | Healthcare coverage | Hybrid work | International travelSenior-level Full TimeLocations: Galway, Ireland9d ago
-
Data Platform Engineer EUR 45K-57KAirflow | ArgoCD | Avo | CI/CD | DBTCollaborative team | Hybrid work setupMid-level Full TimeAmsterdam; Dublin; Helsinki HQ R9d ago
-
FS Technology Consulting - AI and Data - AI Architecture & Engineering Lead- Senior Manager - Dublin EUR 63K-80KAI Act | AWS | Agentic AI | Audit Logging | AzureAdditional annual leave purchase | Bike to work scheme | Discounted health insurance | Flexible working | Free Unlimited Online GP ConsultationsSenior-level Full TimeDublin 2, IE9d ago
-
FS Technology Consulting - AI and Data - Forward Deployed AI Engineer - Manager - Dublin EUR 57K-80KAWS | Agent systems | Amazon ECS | Audit Logging | AzureBike to work scheme | Discounted health insurance | Flexible working | Free gym membership | Hybrid workingMid-level Full TimeDublin 2, IE9d ago