Sr. Splunk Engineer

Leidos is seeking a highly proficient Sr. Splunk Engineer to join our current 10-member Strategy, Architecture & Engineering Enterprise Observability team. The Sr. Splunk Engineer will play an essential role in the lifecycle of our infrastructure and related services, from systems analysis, maintenance activities, architecture expansions, security control implementations, to automation and orchestration.

We have planned significant expansion of the Splunk infrastructure and development of new Splunk services and capabilities over the next few years, and you will be responsible for deploying, managing, and enhancing complex systems within the Splunk Enterprise and Cribl environments to achieve these goals.
 

Our current Leidos Splunk infrastructure is complex and dynamic, including but not limited to the following core components, functionality, and services:

• Index volume:  8 TB/day with 600+ users
• On premises physical/Virtual infrastructure
  • Multisite indexer cluster: 60+ nodes
  • Multiple search head clusters: 20+ nodes
  • Multiple deployment servers, standalone instances: 10+ nodes
• Cribl intermediate layer:  30+ nodes processing/reducing syslog data distributed across 3 continents
• ~50k unique hosts
• More than 100 ODBC connections
• Thousands of: unique sources and sourcetypes, dashboards, saved searches, reports, Lookups files and KVstores
• 200+ apps:  including custom and vendor supported
• Standard and custom data models
• Hundreds of custom/modular inputs
• AWS, Azure Cloud integrations
• Kubernetes fluent bit/Loki integrations
• Disaster recovery capabilities
• Various federal and financial governance requirements

In addition to the maintenance and expansion of the current infrastructure, the Strategy, Architecture & Engineering Enterprise Observability team is aggressively pursuing a two-year plan including development objectives in the following areas:

• ITSI: Service deconstruction and System-of-systems frameworks
• Orchestration and Automation: ServiceNow, other Enterprise management systems
• Splunk Interface Development and Integration: Advanced Interface/Dashboard development beyond the standard and familiar Splunk dashboards
• Machine Learning:  Proficiency in applying statistical algorithms to large and complex data sets to deliver more sophisticated monitoring and alerting services

If you have extensive experience in developing, maintaining, and enhancing complex Splunk systems like those described above, and are skilled in any of the four specified development objectives, we’d love to hear from you!

Required Qualifications:

• Bachelor's degree and 8+ years of IT experience, including 5+ years of recent experience in Splunk Engineering role. Additional years of relevant experience will be considered in lieu of Bachelor's degree.
• 5+ experience developing, maintaining, enhancing, complex Splunk environment and services
• Proficiency in maintaining Splunk on Linux infrastructure; including advanced troubleshooting of performance issues
• Adept at creating, deploying, maintain custom Apps and Add-ons; configuring, deploying, maintaining Splunk Apps/TAs
• Adept at configuring and maintaining all core Splunk configuration files
• Proficiency in onboarding and normalizing data
• Experience integrating data into the Splunk Common Information Model
• Adept at creating, testing and deploying highly optimized regular expressions
• Proficiency at creating, deploying, maintaining, troubleshooting all Splunk Knowledge objects
• Experience in an object-oriented programming language, preferably Python and JavaScript
• Understanding of source control tools like git and bitbucket
• Intermediate understanding and ability to use AWS/Azure technology
• Understanding of iterative development Agile methodology
• Working knowledge of ITIL Change & Configuration Management
• US Citizenship is required and able to obtain security clearance

Desired Qualifications:

• Understanding of data collection/reduction/shaping tools such as Cribl Stream
• Proficiency in Interface/Dashboard development well beyond the standard and familiar Splunk dashboard interfaces and capabilities
• Experience with the deployment of machine learning statistical models applied to large and complex data sets
• Understanding the importance of data and how to apply data in decision making.
• Expertise in service decomposition with Splunk ITSI
• Practical applications of automation to workflows and processes - scripting, low-code / no code
• Familiarity with SQL/ODBC interfaces
• Experience in other systems and network management/monitoring products
• Complete or partial Splunk Admin/Architect and Cribl training

Please note that actual salary will be closer to the middle of the posted pay range.

Original Posting Date:

While subject to change based on business needs, Leidos reasonably anticipates that this job requisition will remain open for at least 3 days with an anticipated close date of no earlier than 3 days after the original posting date as listed above.

Pay Range:

The Leidos pay range for this job level is a general guideline only and not a guarantee of compensation or salary. Additional factors considered in extending an offer include (but are not limited to) responsibilities of the job, education, experience, knowledge, skills, and abilities, as well as internal equity, alignment with market data, applicable bargaining agreement (if any), or other law.