BIBA Practice - Cloud Data Lead
Tasks
- Build ETL ingestion transformation pipelines
- Collaborate on architecture and design reviews
- Deploy Spark on Kubernetes with YARN
- Design Spark batch and streaming pipelines
- Develop Spark applications in Java
- Ensure data quality schema evolution idempotence
- Implement CI CD for data applications
- Implement Structured Streaming exactly once
- Integrate Kafka ingestion with Spark
- Monitor Spark jobs with metrics logging alerting
- Optimize Spark job performance and resource usage
Perks/Benefits
- N/A
Skills/Tech-stack
Airflow | Apache Kafka | Apache Spark | Avro | Azure Blob | Azure Blob Storage | Blob Storage | CI/CD | Catalyst | Cloud Storage | Concurrency | DataFrame | DataSet | EFK | ELK | Elasticsearch | Garbage Collection | GitHub Actions | GitLab CI | Google Cloud | Google Cloud Storage | Grafana | HDFS | Helm | Hive | JVM Tuning | Java | Jenkins | Kafka Connect | Kibana | Kubernetes | Logstash | ORC | Parquet | Prometheus | PySpark | Python | RDD | S3 | Scala | Schema evolution | Spark SQL | Structured Streaming | YARN
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
Related jobs
-
Senior Data Engineer-JT0224 USD 120K-180K.Net Core | .Net Framework | Apache Airflow | Azure | Azure Data401k match | Career growth | Dental insurance | Employee resource groups | Life insuranceSenior-level Full TimeRemote, United States R8h ago
-
Software Engineer 3-BP-184 USD 78K-250KCI/CD | Confluence | Docker | FluxCD | Git401k match | Employee referral programs | Flexible spending accounts | Flexible work arrangements | Mental health supportMid-level Full TimeHanover, MD10h ago
-
Software Engineer 3-BP-181 USD 78K-250KCI/CD | Docker | Git | GitLab | GitOps401k match | Dental insurance | Employee referral programs | Flexible spending accounts | Flexible work arrangementsSenior-level Full TimeHanover, MD10h ago
-
Staff Backend Engineer (AI Platform team) USD 185K-242KAWS | Agent Frameworks | Compliance | Data Privacy | DatadogEquity | Flexible PTO | Medical coverage | Monthly lifestyle stipendSenior-level Full TimeRemote - United States & Canada R11h ago
-
.NET | AWS S3 | Amazon RDS | DTCC RTTM | DTCC Settlement WebContract through June 2027 | East Coast location flexibility | Remote workSenior-level ContractNew York, United States11h ago
-
Senior-level Full TimeTampa, FL, US12h ago
-
AI and Data Science Engineer (TS/SCI Poly) USD 93K-163KAPIs | Artificial Intelligence | CI/CD | Cloud Platforms | ContainerizationMid-level Full TimeMcLean, Virginia, United States13h ago
-
Managed Services Engineer III USD 119K-198KApache Spark | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageSenior-level Full TimeLouisville, Kentucky, United States13h ago
-
Distributed Systems | Linux | Machine Learning | Network Programming | Performance AnalysisSenior-level Full TimeSan Jose, California, United States13h ago
-
Senior AI/ML Engineer USD 117K-195KAPI Development | AWS | AWS SageMaker | Airflow | AzureCareer growth | Customer-focused work | Onsite Pentagon work | Rapid advancementSenior-level Full TimeUSA-VA-Arlington13h ago
-
Mid-level Full TimeUSA-VA-Arlington13h ago
-
AI Forward Deployment Engineer USD 130K-145KAPIs | AWS | Airflow | Artificial Intelligence | AzureClient site travel | On-site workEntry-level Full TimeNew York, New York, United States13h ago
-
Machine Learning Engineer USD 142K-215KAgentic AI | Bayesian Inference | Code review | Deep learning | Error correctionMid-level Full TimeBoston, MA, US14h ago
-
Staff Software Engineer, Generative AI USD 207K-300KAndroid | C++ | Computer Vision | Data Processing | DebuggingSenior-level Full TimeSunnyvale, CA, USA14h ago
-
RTL Design Engineer, Machine Learning Accelerators USD 138K-198KASIC | Code review | High Performance | High-Performance Design | Learning algorithmsMid-level Full TimeSunnyvale, CA, USA14h ago
-
C++ | Computer Graphics | Data Curation | Data Pipelines | Human-in-the-loopSenior-level Full TimeNew York, NY, USA; Mountain View, …14h ago
-
Senior Software Engineer, Managed Spark, OpenSource USD 174K-252KApache Flink | Apache Hive | Apache Hudi | Apache Iceberg | Apache SparkSenior-level Full TimeSunnyvale, CA, USA14h ago
-
Software Engineer III, AI/ML Machine Learning, Core USD 147K-211KC++ | Compute Technologies | Data Preprocessing | Data Storage | Distributed SystemsSenior-level Full TimeNew York, NY, USA; Mountain View, …14h ago
-
Research Scientist, Google Research, GenAI, Experiences USD 147K-211KComputer Vision | Data Mining | Deep learning | High Performance | High-Performance ComputingMid-level Full TimeMountain View, CA, USA; Kirkland, WA, …14h ago
-
Staff Software Engineer, Storage USD 207K-301KC# | C++ | Data Storage | Data Structures | Data Structures and AlgorithmsSenior-level Full TimeSunnyvale, CA, USA14h ago
-
Software Engineer, AI/ML Computer Vision USD 147K-211KC++ | Computer Vision | Data Processing | Data Structures | Data Structures and AlgorithmsMid-level Full TimeSeattle, WA, USA; Kirkland, WA, USA14h ago
-
Senior Staff Software Engineer, TPU Performance USD 262K-365KAuto sharding | Benchmarking | C++ | Compiler optimization | Computer EngineeringBenefits | Bonus | EquitySenior-level Full TimeSunnyvale, CA, USA14h ago
-
Software Engineer, ML Quality, Google Labs USD 207K-300KA/B | A/B Testing | Algorithms | Artificial Intelligence | B testingBonus and equity | Career development | Health insurance | Paid time off | Relocation assistanceSenior-level Full TimeMountain View, CA, USA; San Francisco, …14h ago
-
Sr. Data Scientist USD 114K-220KAirflow | Anomaly Detection | Cloud Platforms | Convolutional Neural Networks | Data acquisition401k matching | Employee stock purchase plan | Healthcare benefits | Paid time off | Parental leaveSenior-level Full TimeIrvine, CA, US16h ago
-
Senior ML/AI Modeler, Risk Automation Machine Learning USD 160K-283KAPI Integration | AWS | Airflow | Artificial Intelligence | Batch ProcessingFlexible time off | Medical insurance | Modern family planning | Remote work | Retirement savings plansSenior-level Full TimeSeattle, WA, United States of America R20h ago