Azure Data Engineering Consultant

PL/SQL
ServiceNow
Databricks Consultant / Developer – Data Pipelines & Integrations
Role Summary
The Databricks Consultant / Developer will design, build, and support data pipelines and integrations on the Azure data platform using Databricks, Airflow, ADF, and ADLS. The role spans both project development (Build) and production support (Run), with a strong focus on DAG orchestration, data transformation, and end‑to‑end pipeline reliability across retail and supply‑chain use cases.

Role Description: Databricks Python L2 Support Engineer
Providing extensive support for the Integration Services
Analyzing issues and root causes of the issues
Understanding of batch programs and interfaces in Aptos Store Inventory Management System
Handling high priority incidents P1P2 till closure Key Responsibilities
1. Data Pipeline & DAG Development (Build)
Design and develop Spark / PySpark notebooks and jobs in Azure Databricks to ingest, transform, and publish batch and near real‑time data across multiple domains (merchandising, supply chain, digital, finance, store systems).
Build and maintain DAGs (primarily in Airflow / Astronomer and Databricks Workflows) to orchestrate dependencies between upstream systems (e.g., RMS, e‑commerce, WMOS, Kafka topics) and downstream data products, ensuring correct ordering, retries, and alerting.
Use ADF and other integration services where required for file, API, and database ingestion into ADLS Gen2, applying appropriate partitioning and optimization techniques for large‑scale workloads.
Implement reusable frameworks for common ETL / ELT patterns (SFTP/Kafka ingestion, CDC, merge/upsert to Delta tables, slowly changing dimensions, data quality checks). 2. Data Transformation & Modeling
Translate business and source‑to‑target requirements into robust data transformations, aggregations, and business rules implemented in Databrick