Big Data Engineer

Overview:

We are seeking a highly skilled Big Data Engineer to lead the design and implementation of robust big data solutions. The ideal candidate will have extensive experience with big data technologies and a strong background in data processing and analytics. This role requires leadership skills to mentor a team of engineers and ensure high-quality deliverables.

Overall Responsibilities:

• Architectural Design: Lead the design and architecture of big data solutions using HDFS, Spark with Scala, and other technologies to meet organizational needs.
• Team Leadership: Mentor and manage a team of data engineers, providing guidance and support to ensure successful project delivery.
• Data Processing Development: Develop and optimize data processing workflows utilizing Apache Spark and Scala, ensuring scalability and performance.
• Database Management: Use HBase and SQL for efficient data storage and retrieval, and oversee data ingestion processes using Sqoop.
• Scripting & Automation: Create shell scripts and Java applications to automate data processing tasks and streamline workflows.
• Collaboration: Work closely with cross-functional teams, including data scientists and business analysts, to gather requirements and translate them into technical solutions.
• Documentation: Maintain comprehensive technical documentation, including architecture diagrams, data flow diagrams, and operational procedures.
• Performance Tuning: Analyze existing big data infrastructure and recommend optimizations for performance and cost efficiency.
• Optional Technologies: Integrate optional technologies such as Apache Kafka, Akka HTTP, and TeamCity as needed.

Technical Skills:

Primary Skills:

• HDFS: Expertise in Hadoop Distributed File System for data storage and management.
• Apache Spark with Scala: Strong experience in using Apache Spark with Scala for big data processing, with a focus on functional programming.
• HBase: Proficiency in HBase for NoSQL database solutions.
• SQL: Ability to write complex SQL queries for data manipulation and analysis.
• Sqoop: Familiarity with Sqoop for data import/export between HDFS and relational databases.
• Shell Scripting: Competence in writing shell scripts for process automation.
• Java (Core): Solid understanding of core Java concepts.

Optional Skills:

• Apache Kafka: Experience with Kafka for real-time data streaming and processing.
• Akka HTTP: Knowledge of Akka HTTP for building RESTful services.
• TeamCity: Familiarity with TeamCity for continuous integration and deployment.
• U-deploy: Knowledge of U-deploy for deployment automation and orchestration.
• Autosys: Experience with Autosys for job scheduling and management.
• SBT: Familiarity with Scala Build Tool (SBT) for managing Scala projects.

Good to Have:

• Pivotal Gemfire: Experience with Pivotal Gemfire for in-memory data grid solutions.

Experience:

• At least 6+ years of experience in big data engineering or a related field.
• Proven track record of designing and implementing big data solutions in production environments.
• Experience leading technical teams and mentoring junior engineers.

Day-to-Day Activities:

• Conduct daily stand-up meetings to discuss project progress and challenges.
• Develop, test, and maintain data processing workflows and pipelines.
• Collaborate with product managers and data scientists to define data requirements and objectives.
• Monitor data processing jobs for performance and troubleshoot issues as they arise.
• Review code from team members and provide constructive feedback to ensure adherence to best practices.
• Stay updated on industry trends and emerging technologies in big data.

Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.
• Relevant certifications in big data technologies or cloud platforms are a plus.

Soft Skills:

• Strong leadership and mentoring abilities.
• Excellent communication and interpersonal skills.
• Analytical and problem-solving mindset with attention to detail.
• Adaptability and a willingness to learn new technologies.
• Strong organizational skills with the ability to manage multiple priorities.

S​YNECHRON’S DIVERSITY & INCLUSION STATEMENT
 

Diversity & Inclusion are fundamental to our culture, and Synechron is proud to be an equal opportunity workplace and is an affirmative action employer. Our Diversity, Equity, and Inclusion (DEI) initiative ‘Same Difference’ is committed to fostering an inclusive culture – promoting equality, diversity and an environment that is respectful to all. We strongly believe that a diverse workforce helps build stronger, successful businesses as a global company. We encourage applicants from across diverse backgrounds, race, ethnicities, religion, age, marital status, gender, sexual orientations, or disabilities to apply. We empower our global workforce by offering flexible workplace arrangements, mentoring, internal mobility, learning and development programs, and more.

All employment decisions at Synechron are based on business needs, job requirements and individual qualifications, without regard to the applicant’s gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law.

Candidate Application Notice