Module Lead - AWS Glue Job
Tasks
- Apply data quality frameworks for financial data
- Create documentation and process maps
- Design data pipelines with AWS Glue
- Develop AWS Lambda functions for data processing
- Develop distributed data processing with PySpark
- Develop with Python
- Execute and monitor AWS Glue data quality
- Implement AWS Glue data quality rules
- Integrate APIs for data ingestion
- Integrate Glue crawlers for metadata discovery
- Optimize file formats and data structures
- Orchestrate pipelines with AWS Glue workflows
- Query datasets with Amazon Athena
- Query with SQL
- Schedule and manage pipelines with Apache Airflow
- Use Snowflake pipelines
- Write Snowflake stored procedures
Perks/Benefits
Skills/Tech-stack
AWS Glue | AWS Glue Crawlers | AWS Glue Data Quality | AWS Glue Jobs | AWS Glue Workflows | AWS Lambda | Amazon Athena | Amazon S3 | Apache Airflow | Data Quality | Glue Crawlers | Glue Jobs | Glue Workflows | Glue data quality | JSON | Parquet | PySpark | Python | SQL | Snowflake | Snowpark
Education
N/A
Roles
Related jobs
-
Sr. Software Engineer (AI/ML) (PhD / Postdoc) INR 1500K-2400KCausal Inference | Deep learning | Generative AI | Language Processing | Machine LearningAccess to large clinical dataset | Collaboration with healthcare partners | Conference publication support | Research freedomSenior-level Full TimeHyderabad, India6h ago
-
Business Intelligence Engineer, Last Mile Analytics INR 1800K-2400KAmazon QuickSight | Data Analysis | Data Interpretation | Data Visualization | NoSQLMid-level Full TimeHyderabad, Telangana, IND12h ago
-
IT Data Engineer II INT INR 1800K-2500KApache Spark | Cloud Cost Optimization | Cloud platform | Cost Optimization | Data GovernanceMid-level Full TimeBangalore, India12h ago
-
Data Engineer - VOIS INR 2000K-2040KApache Airflow | Apache Spark | BigQuery | CI/CD | Cloud ComposerCollaborative environment | Flexible inclusive workplace | Learning opportunitiesMid-level Full TimePune, IN12h ago
-
AWS CloudFormation | Amazon Kinesis | Amazon Redshift | Amazon S3 | Amazon Web ServicesEqual opportunity employer | Fast track growth opportunities | Great place to workSenior-level Full TimeHyderabad, Office Level 3 & 4, …12h ago
-
TTT-SAP-Python Gen AI-Tax Senior INR 2000K-4512KAI Core | AI Launchpad | API Management | CI/CD | Cloud ALMSenior-level Full TimeBengaluru, KA, IN, 56001612h ago
-
Entry-level Full TimeBangalore, India12h ago
-
Deputy Manager, Global Platform Technology INR 1500K-2000KAgile | Azure Blob | Azure Blob Storage | Azure Cosmos | Azure Cosmos DBMid-level Full TimeChennai IMPT, India12h ago
-
Sr. Big Data Engineer INR 2400K-3500KAWS | Apache Spark | Azure | Cloud Platforms | Data ArchitectureTravel opportunitiesSenior-level Full TimeRemote - India R20h ago
-
Machine Learning Engineer - 2 INR 2000K-2500KAWS | Algorithms | Computer Vision | Data Pipelines | Data StructuresEquity | Family insurance | Flexible hours | Free health teleconsultations | Hybrid workMid-level Full TimeBangalore, India1d ago
-
Data Infrastructure Engineer INR 2000K-4000KAWS | Apache Flink | Apache Kafka | Apache Spark | Batch ProcessingEquipment | Health insurance | Meals | Meaningful equity | Onsite collaboration cultureSenior-level Full TimeBengaluru1d ago
-
Senior-level Full TimeNoida1d ago
-
Senior Machine Learning Engineer - Research INR 2500K-4500KAWS | Azure | BERT | Cloud platform | ClusteringSenior-level Full TimeNoida1d ago
-
Mid-level Full TimeBangalore, Karnātaka, India1d ago
-
Senior Data Engineer INR 3000K-4000KAPIs | AWS | AWS Glue | AWS Lambda | AgileAccess to Headspace app | Career development | Hybrid work model | Incentive programs | Mental health daysSenior-level Full TimeIndia, Bengaluru, Karnataka R1d ago
-
Software Engineer, Data Platform INR 1500K-3000KAPI | API Gateway | AWS | CI/CD | CloudWatchEmployee incentive programs | Flexible vacation | Headspace app | Hybrid work model | Mental health daysMid-level Full TimeIndia, Bengaluru, Karnataka R1d ago
-
EY - GDS Consulting - AI and DATA - AWS Databricks - Senior INR 3000K-4000KAWS | AWS Glue | AWS Lambda | AWS Step Functions | Amazon S3Coaching and learning programs | Flexible work arrangements | Global knowledge exchange | Inclusive & Supportive Culture | Personalized growth planSenior-level Full TimeBengaluru, KA, IN, 5600161d ago
-
Software Developer / Software Engineer INR 960K-1935KAutomation | DBMS | Database | Excel | File handlingMid-level Full TimeTN, Chennai, India1d ago
-
Senior Staff Engineer, Gen AI, Azure INR 2000K-4000KAI/ML | API Development | Azure AI | Azure AI/ML | Backend DevelopmentSenior-level Full TimeRemote, India R2d ago
-
Associate Principal Engineer, Azure Data Factory Architect INR 1800K-2500KADF Pipelines | Azure Blob | Azure Blob Storage | Azure Data | Azure Data FactoryMid-level Full TimeRemote, India R2d ago
-
Senior Staff Engineer,GenAI, Playwright INR 1800K-2829KAI Agents | API Testing | Agile | BDD | CI/CDSenior-level Full TimeRemote, India R2d ago
-
Sr Technical Lead-App Development INR 2500K-4500KAg Grid | Amazon Web Services | Autogen | Azure | Cloud platformSenior-level Full TimeINDIA - PUNE - BIRLASOFT OFFICE …2d ago
-
AI Vision Engineer INR 360K-420K3D Vision | Active Learning | Annotation | Camera systems | Cloud processingFamily-friendly benefits | Inclusive environmentEntry-level Full TimeIND-Tamil Nadu-Chennai-KLA, India2d ago
-
Algorithm Manager INR 1000K-1500KC# | C++ | Computer Vision | Deep learning | Functional ProgrammingFamily-friendly benefitsMid-level Full TimeIND-Tamil Nadu-Chennai-KLA, India2d ago
-
Site Reliability Engineer INR 1500K-2000KAlerting | Ansible | Bash | Chef | Cloud CLIHealthcare benefits | Hybrid work setting | Mentorship | Online learning platform | Paid time offEntry-level Full TimeIND-Pune-Equifax Analytics-PEC, India2d ago