Senior Splunk Infrastructure Engineer

• Splunk Administration: Manage the health, performance, and stability of multiple Splunk clusters (Search Head Clusters, Indexer Clusters, and Heavy Forwarders).
• Splunk Enterprise Security (ES) Support: maintain and support the underlying infrastructure for Splunk ES, ensuring optimal performance for security operations.
• Heavy Forwarder & Pipeline Management: Manage applications, parsing rules, and data pipelines on Heavy Forwarders to ensure efficient data ingestion and routing.
• Reliability & HA/DR: Design and maintain High Availability (HA) and Disaster Recovery (DR) strategies to ensure business continuity and platform resilience across regions.
• Safe Configuration Management: Champion proactive engineering practices by implementing safe deployment strategies for SaltStack configurations, including canary testing, validation, and staged rollouts to minimize production incidents.
• Infrastructure as Code: Maintain and write complex SaltStack states and formulas to manage Splunk configurations and underlying Linux VMs.
• GCP Operations: Provision, monitor, and scale infrastructure within Google Cloud Platform.
• System Optimization: Perform deep-dive troubleshooting on Linux systems (kernel tuning, disk I/O, memory management) to ensure optimal Splunk performance.
• On-Call Support: Participate in the on-call rotation to respond to critical incidents affecting Splunk infrastructure availability and performance, ensuring 24/7 reliability.
• Maintenance & Upgrades: Execute distinct maintenance windows, version upgrades, and patching cycles.
• Documentation: Update runbooks and technical documentation within our repository to ensure knowledge sharing, specifically focusing on configuration changes and pipeline architecture.

Required Qualifications

• Splunk Expertise: 5+ years of experience administering large-scale Splunk Enterprise environments. Proven experience with Indexer Clustering and Search Head Clustering.
• Advanced SPL (Splunk Processing Language): Proficiency in writing, debugging, and optimizing complex SPL queries to support dashboarding, alerting, and data analysis.
• Splunk Enterprise Security (ES): Hands-on experience administering and maintaining Splunk ES environments.
• Configuration Management: Strong proficiency with SaltStack (Salt). You must be comfortable writing custom states and managing configurations for thousands of nodes.
• Unix/Linux Internals: Deep understanding of Linux administration (RHEL/CentOS/Ubuntu). Ability to debug resource contention, file system issues, and network bottlenecks.
• Cloud Infrastructure: Hands-on experience with Google Cloud Platform (GCP), specifically GCE and networking.
• Scripting: Proficiency in Python or Bash for automation tasks.