Big Data Lead
Tasks
- Automate ETL processes
- Develop data pipelines
- Export and import data formats
- Generate code using Python configuration and Jinja templates
- Implement CI CD in GitHub
- Implement dbt tests for data validation
- Manage AWS S3 data storage and security
- Monitor pipeline runs and reprocess errors
- Optimize SQL query performance
- Orchestrate pipelines sequential and parallel
- Perform Python unit integration and system testing
- Schedule Airflow DAG jobs
- Troubleshoot data pipeline failures
Perks/Benefits
- N/A
Skills/Tech-stack
AWS IAM | Airflow | Amazon ECS | Amazon S3 | CI/CD | Common table expressions | Complex joins | DBT | Data orchestration | Data pipeline | ETL | GitHub | Integration Testing | Jinja | Python | SQL | Snowflake | System Testing | Unit Testing
Education
N/A
Roles
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R14d ago
-
Senior Software Engineer (Storage) - remote in the US USD 100K-150KArgo CD | ArgoCD | Block Storage | CSI | Cluster APISenior-level Full TimeBoston, MA, United States R6h ago
-
AI Forward Deployed Engineer | $120K-$150K + Hybrid + Equity | AI-Based Outage Intelligence Company A USD 120K-150KAI workflows | APIs | Automation | Backend Development | IntegrationDirect impact on company growth | Equity | Hybrid work | Significant ownership | Unlimited PTOMid-level Full TimeKing of Prussia, PA, United States R7h ago
-
Active Learning | DPO | Data Privacy | Deep learning | Few-Shot Learning401k match | Family leave | Flexible time away | Health plansSenior-level Full TimeMountain View, CALIFORNIA, United States7h ago
-
Machine Learning Engineer, Agentic Application USD 139K-216KAPI Design | Distributed Systems | Generative AI | Language Models | Language Processing401k match | ESPP | Family leave | Flexible spending account | Flexible time offEntry-level Full TimeMountain View, CALIFORNIA, United States8h ago
-
C++ | DistributedTraining | ETL | Go | Huggingface401k matching | ESPP | Family leave | Flexible spending accounts | Flexible time awaySenior-level Full TimeMountain View, CALIFORNIA, United States8h ago
-
C++ | Distributed Training | ETL Pipelines | Go | Hugging Face401k plan | ESPP | Family leave programs | Flexible spending accounts | Flexible time awaySenior-level Full TimeMountain View, CALIFORNIA, United States8h ago
-
Abstractive summarization | Active Learning | DPO | Few-Shot Learning | Few-shotFlexible work persona | Remote work flexibilitySenior-level Full TimeMountain View, CALIFORNIA, United States8h ago
-
AI/ML Engineer USD 67K-114KAmazon Web Services | Apache Airflow | Apache Spark | Azure | Cloud platform401k company match | Company paid disability plans | Company paid life insurance | Company-provided tools | Diversity and inclusionEntry-level Full TimeAnniston, Alabama, United States10h ago
-
Data Visualization Engineer - Advanced Engineering USD 74K-125KAccess Control | Alarm Prioritization | Change Management | Client-Server | CybersecurityMid-level Full TimeHeath, OH, United States12h ago
-
Associate AI Software Engineer USD 65K-100KAgile | Confluence | Data Analysis | Data Modeling | Data Visualization401k company match | Adoption Assistance | DailyPay | Dental insurance | Disability insuranceMid-level Full TimeSt. Louis Park, MN, United States12h ago
-
Mid-Level Data Engineer USD 90K-98KAPI Development | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageContract award contingent | Remote workMid-level Full TimeWork from home, VA, United States R13h ago
-
Senior-level Full TimeWork from home, VA, United States R13h ago
-
Data Engineer USD 145K-160KAWS | AWS Glue | Apache Airflow | Apache Spark | Data QualityContract award contingent | Hybrid work | Public trust clearance support | Technical documentation updates | Travel 15 percent or lessMid-level Full TimeFAIRFAX, VA, United States13h ago
-
Sr. Databricks Consultant USD 175K-250KAWS S3 | Apache Spark | Azure Data | Azure Data Lake | CI/CDRemote work | Secret clearance | Travel as neededSenior-level Full TimeWork from home, VA, United States R13h ago
-
Mid-level Full TimeFAIRFAX, VA, United States13h ago
-
Senior Databricks Migration Engineer USD 130K-158KActive Directory | Apache Spark | Azure Active Directory | Data Quality | Data SkippingOnsite work | Public Trust background investigationSenior-level Full TimeArlington, VA, United States13h ago
-
Senior-level Full TimeWork from home, VA, United States R13h ago
-
Data Engineer USD 110K-165KCause analysis | Data Formats | Data Pipelines | Data Processing | Data TransformationCompetitive benefits | Hybrid remote onsite work | TS SCI clearance with poly eligibilitySenior-level Full TimeFAIRFAX, VA, United States13h ago
-
Jr Data Engineer USD 104K-125KData Mining | Data Parsing | Data Transformation | Geospatial Data | JSONHybrid remote work | Onsite coordinationEntry-level Full TimeFAIRFAX, VA, United States13h ago
-
Senior Finance Data Engineer USD 78K-98KAccess Management | Azure Data | Azure Data Platform | Azure DevOps | CI/CD401k company contribution | Community involvement events | Disability benefits | Flexible paid time off | Flexible spending accountsSenior-level Full TimeAtlanta, Georgia, United States14h ago
-
AI Engineer USD 125K-161KAPI Integration | Agile Development | Artificial Intelligence | Best practices | Cloud PlatformsMid-level Full TimeChesterfield, MO, United States14h ago
-
Data Engineer – Microsoft D365 F&O USD 119K-200KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Platform | Azure DevOpsMid-level Full TimeSaint Louis, MO, United States14h ago
-
Associate Principal - Data Engineering | Senior Data Analyst or Business Analyst | Onsite | 12+ years of experience USD 97K-125KAdobe | Analysis Services | Attribution Modeling | Azure Analysis | Azure Analysis ServicesOnsite workMid-level Contract Full TimeOrlando, FL, United States14h ago
-
Senior DevOps Engineer (Storage) - remote in the US USD 200K-250KAir-gapped | Argo CD | ArgoCD | Bash | Cluster APIAttend conferences and working groups | Company outings | Hackathons | Professional development and training | Tech talksExecutive-level Full TimeBoston, MA, United States R15h ago