Databricks Data Engineer

*Perform testing and debugging of data pipelines and notebooks to identify and resolve issues and ensure optimal performance.

*Collaborate with cross-functional teams, including finance, IT, and business intelligence teams, to ensure seamless integration and alignment of Databricks with other systems and processes.

*Document technical specifications, data models, pipeline configurations, and development processes to ensure knowledge transfer and maintain system documentation.

*At least 10 years of experience in data engineering, with a minimum of 3 years hands-on experience building and managing Databricks solutions.

Extensive experience with Databricks platform components: Delta Lake, Delta Live Tables, Unity Catalog and Databricks Workflows.

*Strong proficiency in Python and/or Scala for data engineering; SQL expertise required.

*Strong experience with core data engineering practices including data ingestion, transformation (ETL/ELT), data modeling, and pipeline orchestration.

*Functional and technical experience with Sales Incentive Compensation (SIC) — including incentive plan design, quota allocation, attainment tracking, commission calculations, and payout processing.

*Familiarity with SIC platforms (e.g., Anaplan, Oracle ICM, Varicent) and experience integrating them with data engineering platforms is a strong plus.

*Ability to understand complex incentive compensation rules and model them accurately in data pipelines and transformation logic.

*Experience integrating Databricks with cloud platforms (Azure, AWS, or GCP) and data sources such as Azure Data Factory, Event Hubs, Kafka, or equivalent.

*Passion and ability to create holistic, end-to-end, integrated data solutions.

*Stakeholder management skills; ability to challenge constructively and translate complex business requirements into scalable data architectures — not just gather requirements.

*Strong analytical and problem-solving skills, with the ability to understand complex data requirements and implement reliable, efficient solutions.

*Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams and stakeholders.

*Must work well in a team environment while being a resourceful, independent self-starter able to work effectively with minimal direction.

*Skilled at documenting complex data architectures and pipelines in a professional manner.

*Ability to work independently and manage multiple tasks and priorities in a fast-paced environment.

*Passion for data quality, performance optimization, and delivering excellent end-user data experiences.

*Databricks Certified Data Engineer Associate.