Lead Big Data Engineer

The Lead Big Data Engineer, working independently, will develop, test, debug and document software components commensurate with their experience as well as direct development staff in support of a software engineering effort.

We are seeking a Big Data Engineer to lead the design, develop, and maintain scalable data pipelines and infrastructure to support our market regulation technology applications. You will play a key role in modernizing existing data pipelines, enabling downstream users to experiment with data using no-code/low-code features.You will collaborate closely with product managers, data scientists, and engineers to deliver high-performance systems that power market surveillance and regulatory workflows.

Essential Job Functions:

• Design and modernize scalable, modular data pipelines from legacy SQL-based workflows.

• Build efficient and reliable ETL processes using Apache Spark and cloud-native tools on AWS.

• Create configurable pipelines that accelerate experimentation and feature selection for data science teams.

• Optimize pipeline performance, reduce storage costs, and improve compute efficiency.

• Ensure compliance with governance standards while maintaining data integrity and lineage.

• Integrate seamlessly with enterprise data platforms, lakes, and cloud-based storage systems.

• Implement and maintain CI/CD pipelines using Jenkins or similar tools.

• Write robust automated tests, including unit, integration, and end-to-end coverage.

• Collaborate within Agile teams, participating in sprint planning, reviews, and retrospectives.

Nice to have:

• Experience in large-scale data engineering or data infrastructure roles.

• Proficient in Python or Scala for building and maintaining data pipelines.

• Deep expertise with Apache Spark, Hive, and distributed computing frameworks.

• Strong knowledge of SQL and NoSQL databases (e.g., PostgreSQL, MongoDB, DynamoDB).

• Experience with feature stores (e.g., Feast, Tecton, SageMaker Feature Store).

• Hands-on with AWS services for compute, storage, and orchestration.

• Experience implementing CI/CD practices and managing production deployments.

• Familiarity with Apache Airflow and workflow orchestration tools.

• Experience with machine learning feature engineering and model serving frameworks.

• Comfortable with automated testing and working in test-driven environments.

• Proven ability to thrive in Agile/Scrum teams and contribute to iterative development cycles.

Education/Experience Requirements:

• Bachelor’s degree in Computer Science, Information Systems or related discipline with at least seven (7) years of related experience, or equivalent training and/or work experience; Master’s degree and past Financial Services industry experience preferred.
• Demonstrated technical expertise in Object Oriented and database technologies/concepts which resulted in deployment of enterprise quality solutions.
• Past experience with developing enterprise quality solutions in an iterative or Agile environment.
• Extensive knowledge of industry leading software engineering approaches including Test Automation, Build Automation and Configuration Management frameworks.
• Strong written and verbal technical communication skills.
• Demonstrated ability to develop effective working relationships that improved the quality of work products.
• Should be well organized, thorough, and able to handle competing priorities.
• Ability to maintain focus and develop proficiency in new skills rapidly.
• Ability to work in a fast paced environment.
• Experience with object oriented programming languages such as Java, Scala or Python.

Work Conditions:

• Ability to work in a hybrid work environment

For work that is performed in CA, CO, HI, MN, VT, IL, Jersey City, NJ, New York City, NY, MA, WA, MD, Washington, DC, NJ State and NY State, please refer to the chart below for the salary range for the corresponding location. In addition to location, actual compensation is based on various factors, including but not limited to, the candidate’s skill set, level of experience, education, and internal peer compensation comparisons.

CA: Minimum Salary $128,000, Maximum Salary $242,600

CO/HI/MN/VT*: Minimum Salary $111,400, Maximum Salary $202,100

IL*: Minimum Salary $122,800, Maximum Salary $222,400
Jersey City, NJ/NYC, NY: Minimum Salary $133,700, Maximum Salary $242,600

MA/WA: Minimum Salary $111,400, Maximum Salary $232,500

MD/Washington, DC: Minimum Salary $128,000, Maximum Salary $232,500

NJ State: Minimum Salary $122,800, Maximum $242,600

NY State: Minimum Salary $111,400, Maximum Salary $242,600

*Including positions performed outside the state but reporting to an office or manager in that state.

Candidates can expect salary offers that range from the minimum to the mid-point of the salary range. FINRA provides full pay ranges so that the candidate can consider their growth potential while at FINRA.

#LI-Hybrid

To be considered for this position, please submit an application. Applications are accepted on an ongoing basis.

The information provided above has been designed to indicate the general nature and level of work of the position. It is not a comprehensive inventory of all duties, responsibilities and qualifications required.

Please note: If the “Apply Now” button on a job board posting does not take you directly to the FINRA Careers site, enter www.finra.org/careers into your browser to reach our site directly.

FINRA strives to make our career site accessible to all users. If you need a disability-related accommodation for completing the application process, please contact FINRA's Employee Relations team at 240.386.4865 or by email at EmployeeRelations@finra.org. Please note that this process is exclusively for inquiries regarding application accommodations. 

Employees may be eligible for a discretionary bonus in addition to base pay. Non-exempt employees are also eligible for overtime pay in accordance with federal, state, or local law. As part of its dedication to employee wellness, FINRA provides comprehensive health, dental and vision insurance. Additional insurance includes basic life, accidental death and dismemberment, supplemental life, spouse/domestic partner and dependent life, and spouse/domestic partner and dependent accidental death and dismemberment, short- and long-term disability, long-term care, business travel accident, disability and legal. FINRA offers immediate participation and vesting in a 401(k) plan with company match and eligibility for participation in an additional FINRA-funded retirement contribution, tuition reimbursement, commuter benefits, and other benefits that support employee wellness, such as adoption assistance, backup family care, surrogacy benefits, employee assistance, and wellness programs.

Time Off and Paid Leave*

FINRA encourages its employees to focus on their health and wellness in many ways, including through a generous time-off program of 15 days of paid time off, 5 personal days and 9 sick days, unless otherwise required by law (all pro-rated in the first year). Additionally, we are proud to support our communities by providing two volunteer service days (based on full-time schedule). Other paid leave includes military leave, jury duty leave, bereavement leave, voting and election official leave for federal, state or local primary and general elections, care of a family member leave (available after 90 days of employment); and childbirth and parental leave (available after 90 days of employment). Full-time employees receive nine paid holidays.

*Based on full-time schedule

Important Information

FINRA’s Code of Conduct imposes restrictions on employees’ investments and requires financial disclosures that are uniquely related to our role as a securities regulator. FINRA employees are required to disclose to FINRA all brokerage accounts that they maintain, and those in which they control trading or have a financial interest (including any trust account of which they are a trustee or beneficiary and all accounts of a spouse, domestic partner or minor child who lives with the employee) and to authorize their broker-dealers to provide FINRA with duplicate statements for all of those accounts. All of those accounts are subject to the Code’s investment and securities account restrictions, and new employees must comply with those investment restrictions—including disposing of any security issued by a company on FINRA’s Prohibited Company List or obtaining a written waiver from their Executive Vice President—by the date they begin employment with FINRA. Employees may only maintain securities accounts that must be disclosed to FINRA at one or more securities firms that provide an electronic feed (e-feed) of data to FINRA, and must move securities accounts from other securities firms to a firm that provides an e-feed within three months of beginning employment.

You can read more about these restrictions here.

As standard practice, employees must also execute FINRA’s Employee Confidentiality and Invention Assignment Agreement without qualification or modification and comply with the company’s policy on nepotism.

Search Firm Representatives

Please be advised that FINRA is not seeking assistance or accepting unsolicited resumes from search firms for this employment opportunity. Regardless of past practice, a valid written agreement and task order must be in place before any resumes are submitted to FINRA. All resumes submitted by search firms to any employee at FINRA without a valid written agreement and task order in place will be deemed the sole property of FINRA and no fee will be paid in the event that person is hired by FINRA.

FINRA strives to make our career site accessible to all users. If you need a disability-related accommodation for completing the application process, please contact FINRA's Employee Relations team at 240.386.4865 or by email at EmployeeRelations@finra.org. Please note that this process is exclusively for inquiries regarding application accommodations.

All qualified applicants will receive consideration for employment without regard to age, citizenship status, color, disability, marital status, national origin, race, religion, sex, veteran status or any other classification protected by federal, state, or local laws.

FINRA abides by the requirements of 41 CFR 60-741.5(a). This regulation prohibits discrimination against qualified individuals on the basis of disability and requires affirmative action by covered prime contractors and subcontractors to employ and advance in employment qualified individuals with disabilities.

FINRA abides by the requirements of 41 CFR 60-300.5(a). This regulation prohibits discrimination against qualified protected veterans and requires affirmative action by covered prime contractors and subcontractors to employ and advance in employment qualified protected veterans.

©2025 FINRA. All rights reserved. FINRA is a registered trademark of the Financial Industry Regulatory Authority, Inc.