Bigdata Developer
Tasks
- Build and optimize ETL and ELT pipelines
- Contribute to AI assisted data quality pipelines
- Design and develop Spark applications
- Design and maintain Hive tables and partitions
- Develop Ansible playbooks for deployments
- Develop and maintain shell scripts for job orchestration
- Extract data from REST APIs
- Handle Kerberos authentication
- Implement OAuth2 token handling
- Implement error handling and logging in scripts
- Load API data into HDFS and Hive
- Manage HDFS encryption zones and data governance
- Manage HDFS operations and data validation
- Manage deployment pipelines with Vault secrets injection
- Manage pagination and retries for API calls
- Migrate Spark applications
- Monitor job health and implement alerting
- Optimize slow running queries
- Parse JSON API responses
- Schedule and manage jobs with AAP and Control-M
- Tune Spark job performance
- Use AI tools for code review and debugging
- Use GitHub Copilot for code acceleration
- Work with Parquet ORC Avro on HDFS
- Write Hive and Spark SQL queries
Perks/Benefits
- Employee assistance program
- Life and disability insurance
- Medical, dental, and vision coverage
- Paid Company Holidays
- Paid time off
- Retirement savings plan
- Sick leave
Skills/Tech-stack
Ansible | Ansible Automation | Ansible Automation Platform | Apache Spark | Automation platform | Avro | Bash | Big Data | CTE | Cloudera CDP | Control-M | Cron | Curl | Data Governance | Data Pipelines | ELT | ETL | GitHub Copilot | HDFS | HDFS Encryption Zones | Hadoop | Hive | HiveQL | JSON | Kerberos | Keytab | Kinit | LLM | OAuth2 | ORC | Parquet | Python | REST APIs | SQL | Scala | Spark DataFrames | Spark Datasets | Spark SQL | Unix Shell | Vault | Window Functions
Education
N/A
Related jobs
-
Software Developer Customer Data Platforms CAD 101K-118KAPI Integration | Adobe Experience | Adobe Experience Platform | Adobe Journey Optimizer | Agile100% remote work | Generous time off | Health benefits | Home office & equipment stipend | Parental leave top-upMid-level Full TimeToronto Office, Canada R1d ago
-
Data Analysis | Data Management | Data integration | Database Integration | DockerAccess to emerging technologies | Agile workflows | Collaborative international environment | Continuous learning | Flexible scheduleSenior-level Full TimeCanada R1d ago
-
Developer, Data Science CAD 85K-112KAirflow | Apache Beam | Apache Spark | BigQuery | CI/CDAccessibility accommodations during hiring | Hybrid workMid-level Full TimeToronto, ON, CA1d ago
-
AI Developer, Intern/Co-op - REMOTE (Fall 2026) CAD 50K-62KAngular | Anthropic API | Artificial Intelligence | CSS | ChatbotsCertificate of internship | Flexible working hours | Internship certificate | Mentorship | Remote workEntry-level Full Time InternshipCAD Remote – ON, Canada R2d ago
-
Embedded Developer and Test Engineer CAD 70K-95KAltium | C# | C++ | Eagle | Embedded SoftwareOnsite workMid-level Full TimeKitchener, ON, Canada4d ago
-
Azure | Power BI | PySpark | Python | Reporting servicesCareer progression | Extended health and dental benefits | Flexible work schedule | Mentorship | Paid time offSenior-level Full TimeOntario; Montreal; Quebec City; Nova Scotia; …4d ago
-
Apache Spark | Cloud Computing | Data Engineering | Databricks | ETLMid-level Full TimeMontréal, Canada4d ago
-
Data Management Developer CAD 90K-140KAuthentication | Encryption | Enterprise Integration | Enterprise Integration Architecture | InformaticaCafeteria access | Donation-matching program | Employee assistance program | Flexible benefits plan | Home office fitness facilitySenior-level Full TimeSaint John, NB, Canada5d ago
-
Anthropic API | Asynchronous programming | Docker | LLM APIs | LoggingFlexible schedule | Fully remote | Performance-based bonusMid-level FreelanceCanada - Remote R5d ago
-
Gen AI Developer Specialist USD 125K-175KAPI Development | Agentic AI | Alerting | CI/CD | ContainerizationSenior-level Full TimeCanada5d ago
-
Embedded Developer CAD 72K-80KAutomated testing | C++ | CI/CD | FastAPI | GRPCComprehensive benefits | Hybrid work | Mentorship | Paid volunteering days | Parental supportEntry-level Full TimeMontreal, QC, Canada R6d ago
-
Quantitative Developer with MatLab CAD 130K-140KAsset Management | MATLAB | Mathematical Modeling | Optimization | PandasDental insurance | Flexible-hybrid work | Health insurance | Learning resources access | Life insuranceMid-level Full TimeToronto, ON, Canada6d ago
-
Mid-level Full TimeNew York, Canada6d ago
-
API Integration | AWS | Accessibility | Azure | D3.jsCollaborative environment | Contract extension | Flexible international environment | Fully remote work | High autonomySenior-level Full TimeCanada R6d ago
-
Python Developer - NLP, ML, Gen AI CAD 94K-141KApplication development | Data Modeling | Generative AI | Language Processing | Machine LearningSenior-level Full TimeLocation(s): Mississauga, Ontario, Canada7d ago
-
Senior Robotic Software Developer CAD 100K-150KC++ | CI/CD | Code Quality | Concurrency | ContainerizationEmployee assistance program | Employer Paid DC Pension | Employer Savings Plan Matching | Fitness Reimbursement Program | Flexible work scheduleSenior-level Full TimeCanada-Ontario-Waterloo7d ago
-
Senior Algorithms Developer CAD 150K-206KAlgorithms | C++ | Combinatorial Optimization | Computational Complexity | Data StructuresSenior-level Full TimeBurnaby, Canada7d ago
-
Azure Artifacts | Azure Boards | Azure Data | Azure Data Factory | Azure DatabricksMid-level ContractDownsview Central, Canada7d ago
-
.NET | Agile | Angular | Bitbucket | C#Entry-level Full TimeDorval, Quebec, Canada7d ago
-
Software Developer - ETL - Senior CAD 139K-208KAzure Artifacts | Azure Boards | Azure Data | Azure Data Factory | Azure DevOpsSenior-level Full TimeToronto, Canada8d ago
-
Software Developer - ETL - Senior CAD 123K-157KApache Spark | Automl | Azure Data | Azure Data Factory | Azure Data GovernanceSenior-level Full TimeToronto, Canada8d ago
-
Software Developer - ETL - Senior CAD 141K-212KAWS Glue | Amazon Aurora | Azure Data | Azure Data Factory | Azure Data LakeSenior-level Full TimeToronto, Canada8d ago
-
Developer, GenAI and Digital Workplace CAD 111K-145KASP.Net Core | Angular | Azure DevOps | C# | CI/CDCareer development investment | Competitive pension plans | Comprehensive group insurance plans | Family building support | Hybrid work modelEntry-level Full TimeMontreal, Canada8d ago
-
Embedded Systems Software Developer CAD 80K-120KANSI B56.5 | C++ | CAN | CI/CD | DebianDental insurance | Employee assistance program | Employer Paid DC Pension | Employer Savings Plan Matching | Fitness Reimbursement ProgramEntry-level Full TimeCanada-Ontario-Kitchener-Strasburg Rd8d ago
-
Mid-level Full TimeCanada- Ottawa- 385 Terry Fox- Bldg …8d ago