Big Data/PySpark Engineering Lead - Vice President
Location(s): Tampa, Florida, United States
USD 113K-170K Senior-level Full Time
Tasks
- Benchmark performance on legacy systems
- Build CI/CD pipelines
- Collaborate with product managers on data availability
- Conduct code reviews
- Design scalable data processing pipelines
- Develop data models and schemas
- Implement data parity testing
- Implement data validation and encryption
- Integrate Big Data tools and frameworks
- Manage risk and compliance issues
- Mentor engineers
- Migrate data from legacy systems to data lakehouse
- Monitor system health
- Optimize SQL queries
- Orchestrate phased migration cutovers
- Partner with management teams on system enhancements
- Perform schema evolution
- Re engineer ETL and stored procedures
- Translate business requirements into technical specifications
- Troubleshoot performance bottlenecks
- Write Python code for data processing
Perks/Benefits
- 401k matching
- Dental insurance
- Disability insurance
- Health insurance
- Life insurance
- Paid time off
- Vision insurance
- Wellness programs
Skills/Tech-stack
Apache Spark | Automated testing | Avro | Bitbucket | CI/CD | CSV | Code review | Collibra | Data Lakehouse | Data Lineage | Data Modeling | Data Validation | ELT | ETL | Encryption | Flink | Git | HBase | Hadoop | Hive | Iceberg | Impala | Informatica | JSON | Kafka | Latency optimization | MongoDB | NoSQL | Parquet | Performance Benchmarking | Presto | PySpark | Python | SQL | Schema Design | Shell Scripting | Spark SQL | Starburst | Trino | Unix | YARN
Education
Related jobs
-
AC DC Sizing | AC/DC | Arc Flash | Artificial Intelligence | AutomationMultiple project extensions | Paid on time | Remote work optionSenior-level Full TimeJuno Beach, FL R23h ago
-
AI Agents | API Development | Azure | CI/CD | LLMOpsCollaborative cross-functional team | Fully remoteMid-level Full TimeCanada, Ontario; Toronto, None; Toronto, Ontario; … R23h ago
-
API Integration | AWS Bedrock | Auditing | Authentication | AuthorizationMentoring | MentorshipMid-level Full TimeMenlo Park, CA, United States1d ago
-
Senior Software Engineer (Java Full Stack) - Hybrid USD 138K-183KAgile | C plus plus | Concurrency | Data Structures | Embedded SystemsHybrid work | Remote work possibleSenior-level ContractBoston, United States1d ago
-
Lead Palantir AI and Data Science Engineer II USD 159K-265KAI workflows | Analytics | Cloud Computing | Dashboards | Data GovernanceProfessional development | Travel opportunitiesSenior-level Full TimeRosslyn, Virginia, United States1d ago
-
Analytics and Insights Engineer III USD 131K-218KApache Spark | Data Governance | Data Modeling | Data Pipelines | Data QualitySenior-level Full TimeRichmond, Virginia, United States1d ago
-
3D Rendering | A/B | A/B Testing | Agentic AI | AlgorithmsSenior-level Full TimeSan Jose, California, United States1d ago
-
C++ | Computer Vision | Data Processing | Deep learning | Domain AdaptationEntry-level Full TimeSan Jose, California, United States1d ago
-
Senior Software Engineer, Google Cloud Dataproc USD 174K-252KApache Hive | Apache Spark | C++ | Data Warehouse | Data Warehouse InternalsSenior-level Full TimeSunnyvale, CA, USA1d ago
-
Agentic Orchestration | C++ | Data Onboarding | Data Pipeline Engineering | Data pipelineSenior-level Full TimeSan Jose, CA, USA1d ago
-
Senior Software Engineer, AI/ML, Core USD 174K-252KC++ | Data Processing | Data Structures | Data Structures and Algorithms | DebuggingSenior-level Full TimeMountain View, CA, USA1d ago
-
Staff Software Engineer, Search Local Ads Data Platform USD 207K-300KC plus plus | CSS | Data Storage | Data Structures | Data Structures and AlgorithmsSenior-level Full TimeMountain View, CA, USA1d ago
-
Apache Spark | Azure | Azure Data | Azure Data Lake | Azure Data Lake StorageHands-on project experience | Mentorship from engineers | Onsite work 5 days per week | Team collaborationEntry-level Full TimeIrvine, CA, US1d ago
-
Apache Spark | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsLearning opportunities | Mentorship | On site work 5 days per week | Team collaborationEntry-level Full TimeIrvine, CA, US1d ago
-
Analytics Data Engineer USD 94K-130KAzure Data | Azure Data Factory | CI/CD | Data Analysis | Data FactoryRemote workMid-level Full TimeFlagstaff, AZ, US1d ago
-
Director Data Engineering USD 178K-255KARM | Agile | Amazon Web Services | Azure Data | Azure Data Factory401k match | Company paid life insurance | Employee assistance program | Health, dental, and vision insurance | Paid time offExecutive-level Full TimeLeawood, KS, US1d ago
-
AI Application & Cloud Operations Engineer (Mid level) USD 108K-117KAWS | AWS Bedrock | AWS CloudTrail | AWS CloudWatch | AWS PrivateLinkMid-level Contract Full TimePlano, TX, United States1d ago
-
Benchmarking | C plus plus | CUDA | Computational Engineering | Distributed Computing401k match | Generous paid time off | Medical, dental & vision coverage | Stock options | WFH Hardware ReimbursementSenior-level Full TimeSunnyvale, CA R1d ago
-
Software Engineer, Applied AI USD 173K-250KABAC | Agentic Frameworks | Apache Airflow | Audit trails | BIMDental insurance | Generous PTO | Health insurance | Retirement or Pension Plan | Vision insuranceMid-level Full TimeSan Francisco, CA1d ago
-
Staff Data Engineer USD 147K-184KAmazon Kinesis | Amazon Web Services | Apache Airflow | Apache Iceberg | Apache SparkDental insurance coverage | Health insurance coverage | Paid parental leave | Paid vacation | Relocation assistanceSenior-level Full TimeBerkeley, CA1d ago
-
Senior Systems Manager, Paragon Applications - Onsite at Houston Physicians Hospital in Webster, TX USD 83K-124KAcceptance Testing | Access Control | Cause analysis | Change Management | Database AdministrationSenior-level Full TimeWebster, TX, United States1d ago
-
Associate, Client Intelligence (Data Scientist) USD 98K-132KAWS | Apache Spark | Azure | Big Data | Cloud platformEmployee resource groups | Matching gift program | Professional development | Sustainability and Climate Change Initiatives | Volunteerism programMid-level Full TimeNew York, NY, US, 100011d ago
-
Apple Neural Engine | C++ | CPU | Compilers | Computer ArchitectureMid-level Full TimeSeattle1d ago
-
On-device ML Engineering Manager (Tools & Services) USD 183K-296KBackend Development | Benchmarking | C++ | Compilers | Core MLMid-level Full TimeSeattle1d ago
-
On-Device ML Infrastructure Engineer (CoreML Runtime), Graphics, Games and Machine Learning USD 170K-296KC++ | Core ML | GPU Programming | JAX | LLVMMid-level Full TimeCupertino1d ago