Senior Data Engineering - PSM-Libra
Tasks
- Analyze large data
- Apply SDLC best practices
- Build core services for machine learning and analytics
- Collect data sets
- Deploy machine learning models
- Develop applications on top of Hive
- Develop applications on top of Spark
- Implement automated testing with TDD
- Implement data pipelines
- Integrate multiple data sources
- Integrate solutions with enterprise architecture
- Maintain data pipelines
- Manage infrastructure automation
- Monitor data pipelines
- Process large data
- Process semi structured data
- Process unstructured data
- Query data using Hive
- Rationalize message data
- Rationalize structured data
- Select optimal data solutions
- Set up continuous deployment
- Set up continuous integration
- Store data sets
- Use Confluence for documentation
- Use JIRA for tracking
- Work in agile scrum teams
Perks/Benefits
- N/A
Skills/Tech-stack
Agile | Anaconda | Ansible | Apache Hive | Apache Spark | CSS3 | Cloud platform | Cloudera | Confluence | Continuous Deployment | Continuous integration | Dataflow | Dataproc | Elasticsearch | Elasticsearch stack | Flask | Git | GitLab | Google Cloud | Google Cloud Platform | HDFS | HTML5 | Hortonworks | JavaScript | Jenkins | Jira | PySpark | Python | SDLC | Scala | Scrum | TDD | Tornado | Twisted
Education
N/A
Roles
Related jobs
-
Technical Solutions Engineer INR 500K-700KAPI Integration | Agentic AI | Automation Anywhere | Excel | Incident ManagementEntry-level Full TimeHyderabad, India4h ago
-
Data Engineer - Databricks & Microsoft FabricData Engineer - Databricks & Microsoft Fabric INR 1500K-2800KAgile | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsSenior-level Full TimeChennai, IN, 6000325h ago
-
Lead Data Engineer INR 1500K-2700KApache Kafka | Apache Spark | Automated testing | CI/CD | Data ModelingSenior-level Full TimeBengaluru, KA, India6h ago
-
ASR | AWS | Audio Processing | Cloud platform | ContainerizationFlexible working hours | Fully remote | Mentorship | Professional growthSenior-level Full TimeIndia R6h ago
-
Software Engineer - Data Engineering INR 1500K-2541KAWS | AWS Glue | AWS Lambda | AWS Step Functions | AgileOnsite 5 days per week | Relocation assistance | Visa sponsorshipMid-level Full TimeChennai, Tamil Nadu, India10h ago
-
Software Engineer - Data Engineering INR 1500K-2500KAWS | AWS Glue | AWS Lambda | AWS Step Functions | AgileOnsite work | Relocation assistanceMid-level Full TimeChennai, Tamil Nadu, India10h ago
-
Lead Software Engineer- Data Engineer INR 2000K-3000KAWS | AWS CloudFormation | Agile | Azure DevOps | CI/CDOnsite work five days per week | Relocation availableSenior-level Full TimeChennai, Tamil Nadu, India10h ago
-
EAI Software Engineer INR 1800K-3500KAPI Gateway | Agile | Argo CD | CI/CD | Cloud FunctionsCollaborative team | Inclusive culture | On-site work 4 days a weekMid-level Full TimeHyderabad, India19h ago
-
Staff Engineer, Data Engineering INR 1500K-2500KApache Spark | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageSenior-level Full TimeBengaluru, KA, India20h ago
-
Backend Engineer (Cloud & AI Integration) INR 1500K-3000KAPI Integration | AWS | Cloud Native | Documentation | GitMid-level Full TimeBengaluru, KA, India20h ago
-
Lead - Data Engineering INR 3000K-4000KData Governance | Data Modeling | Data Quality | Data Visualization | Data WarehousingSenior-level Full TimeBengaluru, Karnataka, India, India21h ago
-
AWS | AWS Lambda | AWS S3 | Amazon EMR | Apache AirflowCareer development opportunities | Employee Assistance Program (EAP) | Financial wellness tools | Flexible working environment | LinkedIn LearningSenior-level Full TimePune, MH, India21h ago
-
Senior Data Engineering - GCP Lead INR 2500K-4000KAPIs | Airflow | Artificial Intelligence | BigQuery | Business IntelligenceSenior-level Full TimeIndia - Bengaluru21h ago
-
Mid-level Full TimeIN-TN-Chennai21h ago
-
Streamlit & Databricks API INR 1500K-2500KAgile | CI/CD | Databricks | Databricks Apps | Databricks NotebooksSenior-level Full TimeIN-TN-Chennai21h ago
-
Data Engineer with GCP INR 1500K-2500KBatch Processing | BigQuery | Cloud Storage | Cloud platform | Data GovernanceSenior-level Full TimeIN-AP-Hyderabad R21h ago
-
Agentic AI | Alerting | Anomaly Detection | Apache Airflow | Artifact RegistryMid-level Full TimePune, Maharashtra, India R22h ago
-
Data Engineer INR 1000K-1700KAWS Athena | Airflow | Amazon Web Services | Big Data | CI/CDEmployee assistance program | Flexible working environment | LinkedIn Learning | Volunteer time offMid-level Full TimePune, MH, India22h ago
-
Agentic AI Data Engineer INR 2000K-5000KAWS Glue | AWS Lambda | AWS Step Functions | Amazon Athena | Amazon BedrockSenior-level Full TimePune, Maharashtra, India22h ago
-
Senior Data Engineer INR 3000K-4000KAWS | Airflow | Amazon Aurora | Apache Kafka | Apache SparkBike to work | Career mentorship | Educational assistance | Employee assistance program | Employee discountSenior-level Full TimeMumbai, MH, India R22h ago
-
AWS | AWS CloudFormation | AWS Glue | AWS Lake | AWS Lake FormationFlexible location | Fully remote work | Ownership culture | Technical autonomySenior-level Contract Full TimeIndia R22h ago
-
API Development | Alerting | Batch Processing | Cloud Computing | ComplianceSenior-level Full TimeBengaluru, Karnataka, India23h ago
-
Data Engineer, YouTube INR 1000K-2000KApache Flume | Cloud Dataflow | Data Analysis | Data Architecture | Data GovernanceMid-level Full TimeBengaluru, Karnataka, India23h ago
-
Senior Software Engineer, AI/ML, Search INR 4000K-6000KCode review | Data Processing | Data Structures | Data Structures and Algorithms | DebuggingSenior-level Full TimeBengaluru, Karnataka, India23h ago
-
Senior-level Full TimeHyderabad23h ago