Data Engineer - Azure Databricks

Title: Azure/DataBricks Engineer
Location: Remote
Duration: 6+ Months

Notes:  Only candidates meeting the technical and experience criteria will be considered for submission.

The client is seeking strong data engineers to be part of their data product development team. It consists of data engineers who can assist in translating business requirements into data products — creating efficient, scalable, and reusable data pipelines — and collaborating with product owners and stakeholders to design, develop, and implement end-to-end data supply chains in enterprise data warehouse environments using Microsoft Azure, ADF, Python, PySpark, Spark SQL, and Databricks.

Responsibilities:

• Ensure data quality, reliability, and orchestration of data pipelines; implement CI/CD; optimize performance and efficiency of pipelines.
• Collaborate cross-functionally to integrate diverse data sources and continuously refine pipeline architecture.
• Provide technical leadership and hands-on support for Azure Databricks-related projects — including AI/ML model deployment, MLOps integration (e.g., MLflow on Databricks), feature engineering at scale, and operationalization of predictive analytics workflows.
• Apply domain-specific knowledge — especially in healthcare data ecosystems (e.g., HL7, FHIR, PHI/PII handling, claims, EHR/EMR integrations, value-based care analytics) — to ensure compliance, relevance, and real-world impact.
• Follow Azure and Databricks best practices, enforce coding standards, manage job orchestration & observability, and deliver required technical artifacts.

Requirements:

• Bachelor’s degree in computer science, information technology, or a related field.
• Strong hands-on experience in healthcare data domains — including regulatory awareness (HIPAA, HITRUST), clinical/claims data models, and healthcare analytics use cases.
• Strong hands-on experience with AI/Artificial Intelligence — demonstrated ability to build, train, deploy, and monitor ML models in production, preferably using Azure ML, Databricks ML Runtime, or similar platforms. Experience with LLM integration, vector search, or GenAI-enabled data applications is a strong plus.
• Prior experience with data lakes and data warehouses; working knowledge of dimensional modeling is mandatory.
• Demonstrated advanced proficiency in data engineering practices: ETL/ELT, data integrations, reusable pipeline design, and modern data management (Delta Lake, Unity Catalog).
• Advanced skills in PySpark, Python, Spark SQL, Azure optimization techniques, and the full Azure Databricks ecosystem.
• Experience with Azure data services: ADF, Databricks, Delta Lake, Delta Live Tables, Power BI, Azure SQL Server.
• Proficiency in SQL, Python, PySpark, and Spark SQL.
• Experience with Agile methodologies and CI/CD pipelines using Azure DevOps.
• Strong problem-solving, analytical thinking, communication, and collaboration skills.
• Self-motivated, independent contributor with proven ability to multitask across concurrent initiatives.
• A practical coding test will be conducted as part of the technical interview process — candidates should be prepared to demonstrate live coding in Python/PySpark and/or SQL, with emphasis on data transformation logic, performance-aware patterns, and debugging in a Databricks/Azure context.
• 8+ years of professional experience in data engineering within the Azure + Databricks ecosystem.
  
  Kind regards,
  xxxxxxxxxxxxxxx
  xxxxxxxxxxxxxxx