Senior Data Engineer
Tasks
- Build and optimize ETL and ELT workflows
- Design scalable batch and streaming data pipelines
- Develop medallion architecture patterns
- Ensure compliance with healthcare regulations
- Implement DataOps processes
- Implement auditability lineage and metadata management
- Implement automated testing and CI/CD
- Integrate structured and unstructured data sources
- Lead technical design reviews
- Manage PHI and PII data securely
- Mentor junior engineers
- Monitor pipeline health and performance
- Optimize data storage and processing performance
- Set up monitoring, alerting, and incident response
- Troubleshoot production data issues
Perks/Benefits
Skills/Tech-stack
AWS | Apache Spark | Auditability | Azure | CI/CD | Collibra | Data Lineage | Data Quality | Data orchestration | DataOps | Databricks | Delta Lake | Delta Lake Governance | Delta Lakehouse | ELT | ETL | Git | Informatica | Ingestion | Lakehouse | Learning operations | MLOps | Machine Learning | Machine Learning Operations | Medallion Architecture | Metadata Management | Monitoring | Observability | PySpark | Python | SQL | Unity Catalog | Workflow Management
Education
Roles
Regions
Countries
States
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R26d ago
-
ActiveMQ | Agile | Amazon Web Services | Apache Beam | Apache CassandraMid-level Full TimeSan Mateo, CA; Irving, TX; Sunnyvale, …1h ago
-
Senior-level Full TimeSan Bruno, CA; San Bruo, CA; …1h ago
-
API | Amazon Web Services | Amazon Web Services (AWS) | Apache HDFS | Apache SparkRemote workMid-level Full TimeRemote, US; Sunnyvale, CA; Dallas, TX R1h ago
-
Adobe Analytics | Airflow | Apache Hive | Apache Spark | Data LineageSenior-level Full TimeSan Bruo, CA; San Bruno, CA; …1h ago
-
Mid-level Full TimeAtlanta, GA; San Bruno, CA; Dallas …1h ago
-
Mid-level Full TimeSunnyvale, CA; San Bruno, CA; Dallas …1h ago
-
Mid-level Full TimeSunnyvale, CA; San Bruno, CA; Dallas …1h ago
-
Mid-level Full TimeSunnyvale, Ca; San Bruno, CA; Dallas …1h ago
-
Apache Drill | Apache HBase | Apache Hadoop | Apache Hive | Apache SparkMid-level Full TimeSunnyvale, Ca; San Bruno, CA; Dallas …1h ago
-
Apache Spark | Avro | Azure Data | Azure Data Factory | Azure Data LakeRemote work optionMid-level Full TimeDallas / Philadelphia, USA; Bangalore, India; …1h ago
-
AWS | Amazon S3 | Angular | Anomaly Detection | Apache AirflowMid-level Full TimeIrving, TX; Sunnyvale, CA; Sunnyvale, CA; …1h ago
-
Apache Airflow | Apache Drill | Apache Hive | Apache Spark | ElasticsearchMid-level Full TimeCarlsbad, CA; San Bruno, CA; Dallas …1h ago
-
Mid-level Full TimeSan Bruno, CA; Sunnyvale, CA; Sunnyvale, …1h ago
-
Mid-level Full TimeSunnyvale, CA; San Bruno, CA; Dallas …1h ago
-
Mid-level Full TimeSunnyvale, CA; San Bruno, CA; Dallas …1h ago
-
Apache Hadoop | Apache Hive | Apache Pig | Apache Spark | Business ObjectsMid-level Full TimeDallas, TX; San Bruno, CA; Dallas …1h ago
-
Mid-level Full TimeSan Bruno, CA; San Bruno, CA; …1h ago
-
Azure | Cloud platform | Confluence | Continuous Deployment | Continuous integrationMid-level Full TimeDallas, TX; San Bruno, CA; Dallas …1h ago
-
Apache Airflow | Apache Hive | Apache Spark | Backend Development | BigQueryMid-level Full TimeSunnyvale, CA; San Mateo, CA; San …1h ago
-
Cloud platform | Data Quality | Google Cloud | Google Cloud Platform | HadoopEntry-level Full TimeSan Bruno, CA; Sunnyvale, CA; San …1h ago
-
Apache Airflow | Apache Drill | Apache Hadoop | Apache Hive | Apache ImpalaMid-level Full TimeSunnyvale, CA; San Bruno, CA; Dallas …1h ago
-
Apache Airflow | Apache Spark | BigQuery | Cloud Dataflow | Data QualityEntry-level Full TimeSunnyvale, CA; Sunnyvale, CA; San Mateo, …1h ago
-
Entry-level Full TimeSunnyvale, CA; San Bruno, CA; Dallas …1h ago
-
Algorithms | Apache Kafka | Apache Spark | Azkaban | CassandraSenior-level Full TimeSunnyvale, CA1h ago