AI Data Engineer - Security (Kafka Experience)
Tasks
- Apply data governance and RBAC
- Build AWS data lake zones
- Build ELT models in Snowflake
- Design event streaming platforms
- Develop idempotent DAGs
- Encrypt and lifecycle S3 data
- Implement CDC with Snowpipe
- Implement IAM and Secrets Manager
- Ingest data from GraphQL APIs
- Ingest data from REST APIs
- Land data in S3 as Parquet
- Manage Glue Data Catalog
- Manage Kafka schema and performance
- Normalize and enrich data in Python
- Optimize Kafka consumer groups
- Optimize Snowflake warehouse performance
- Orchestrate pipelines with AirFlow
Perks/Benefits
Skills/Tech-stack
API Key | AWS Glue | AWS IAM | AWS Secrets | AWS Secrets Manager | Airflow | Amazon S3 | Apache Parquet | CDC | CI/CD | Data Governance | DynamicFrames | ELT | Git-based CI/CD | GraphQL | Kafka | OAuth2 | PySpark | Python | RBAC | REST API | Secrets Manager | Snowflake | Snowpipe
Education
N/A
Roles
Related jobs
-
Automation Testing | CI/CD | CSS | Cypress | Feature DevelopmentMedical, dental & vision coverage | Paid time off | Parental leave | Reimbursement programs | Retirement planMid-levelRaleigh, United States R16d ago
-
Senior-level Full TimeTeletrabajo R4h ago
-
Senior Full-Stack Data Engineer (Remote) USD 110K-130KAI code assistance | Automated testing | Azure DevOps | CI/CD | Code assistanceFull-time | Fully remote | Long-term | No timezone shiftingSenior-level Full TimeFlorida, Aventura, United States of America R6h ago
-
Senior Data Engineer - Remote USD 150K-160KAPIs | Data Governance | Data Lineage | Data Management | Data Modeling401k | Dental insurance | Health and wellness programs | Health insurance | Life insuranceSenior-level Full TimeAtlanta, GA, US, 30328 R12h ago
-
Research Engineer - RL Infra USD 247K-425KCI/CD | Data parallelism | DeepSpeed | Distributed Training | Experiment trackingConference opportunities | Hackathons | Learning opportunities | Quarterly team offsites | Relocation assistanceSenior-level Full TimeRemote R15h ago
-
Senior Analytics Engineer, Marketing USD 159K-201KAttribution | DBT | Data Modeling | Dimension tables | Dimensional ModelingAnnual equity refresh grants | New hire equity grant | Remote work flexibilitySenior-level Full TimeUnited States - Remote R15h ago
-
Senior-level Full TimeRemote R19h ago
-
Senior-level Full TimeRemote R19h ago
-
Data Engineer PLN 153K-247KAWS | Apache Airflow | Azure | Cloud Computing | Cloud platformAgile/Scrum environment | Choice of technical equipment | Flexible schedule | Flexible working hours | Hybrid or remote workMid-level Full TimeKraków, Lesser Poland Voivodeship, Poland - … R20h ago
-
Data Governance | Django | Langchain | MongoDB | PineconeFlexible work hours | Flexible work location | Health insurance | Leave benefits | Online learning accessSenior-level Full TimeMetro Manila, Philippines - Remote R20h ago
-
Access Control | Airflow | Amazon Redshift | BigQuery | CI/CDFlexible schedule | Remote workSenior-level Full TimePortugal - Remote R20h ago
-
Mid-level Full TimePakistan - Remote R20h ago
-
Data Scientist Specialist (Lending) BRL 184K-184KA/B | A/B Testing | AUC | AWS | Apache SparkDental insurance | Health insurance | Home-office allowance | Life insurance | Meal allowanceSenior-level Full TimeBrazil - Remote R20h ago
-
AVP, Model Ops Engineer (L10) INR 500K-500KAWS | Agile | Amazon Redshift | Apache Airflow | Apache KafkaEmployee well-being programs | Flexible schedule | Work from homeExecutive-level Full TimeRemote Central Region IN, India R20h ago
-
AI Scientist HUF 8800K-12500KCloud Platforms | Computer Vision | Convolutional Neural Networks | Deep learning | ExperimentationFlexibility | Remote or hybrid working | Security | Support and well-beingMid-level Full TimeHungary Remote R20h ago
-
Staff Full Stack Software Engineer, Platform Engineering USD 180K-213KAutoscaling | Batching | Caching | Containers | Data GovernanceCareer development | Employee resource groups | Flexible work from home | Generous PTO | Paid volunteer timeSenior-level Full TimeUS-Texas-Austin, United States R20h ago
-
Sr. Software Engineer - Cloudera Manager USD 152K-190KAWS | Azure | Cloud platform | Distributed Systems | GoCareer development | Employee resource groups | Flexible WFH | Generous PTO | Paid volunteer timeSenior-level Full TimeUS-California-Remote, United States R20h ago
-
Lead Engineer – Embedded Software USD 105K-130KAgile | Build troubleshooting | C# | C++ | Control Theory401k matching | Adoption Assistance | Disability insurance | Employee assistance program | Health coachingSenior-level Full TimeEvendale, United States R20h ago
-
Solutions Architect/Data Engineer USD 145K-175KAWS | AWS IAM | AWS Lambda | Amazon EC2 | Amazon RDS401k matching | Dental insurance | Employee development program | Employee referral program | Medical insuranceSenior-level Full TimeRemote, MD, US R20h ago
-
APIs | Amazon S3 | Batch scheduling | Cloud Storage | Configuration ManagementEntry-level InternshipPalo Alto, California, United States; Remote R21h ago
-
Sr. Analytics Engineer USD 145K-195KAvo | BigEye | BigQuery | DBT | Data CatalogCommuter benefits | Dog-friendly office | Flexible working hours | Free gympass subscription | Home office stipendSenior-level Full TimeSan Francisco, CA, Brooklyn, NY, Cambridge, … R21h ago
-
Platform Engineer USD 150K-240KAWS | ArgoCD | Bazel | CI/CD | DatadogDental insurance | Flexible savings account | Health savings account | Medical insurance | Paid HolidaysMid-level Full TimeUS Remote - Bay Area Preferred R22h ago
-
Senior Gen AI Developer MXN 1040K-1300KAPI Development | AWS | Anthropic | Automated testing | AzureAnnual time off | Continuous development opportunities | Employee insurance | Remote work | Wellness programsSenior-level Full TimeMexico city, Mexico R22h ago
-
Senior Data/ML Engineer USD 151K-205KDBT | Data Architecture | Data Governance | Data Observability | Data Quality401k match | Dental insurance | Equity | Family planning resources | Flexible vacation policySenior-level Full TimeRemote - USA R22h ago
-
Bayesian Inference | CI/CD | Cloud Computing | Computer Vision | Geospatial analysisCollaborative team environment | Remote work | Travel opportunitiesSenior-level Full TimeRemote R22h ago