Azure Databricks ETL

Azure Databricks ETL
Databricks ETL Data Engineer    Databricks, ETL , Python Scripting, Data Factory, Data Migration
Location : REMOTE

Role Descriptions: Job Title Azure Databricks ETL EngineerExperience5-8 years Job SummaryWe are seeking an experienced Azure Databricks ETL Engineer with strong handson expertise in ETL development using Python and SQL to design| build| and optimize scalable data pipelines on Azure Databricks. The role focuses on delivering robust| highperformance data solutions leveraging Azure data ecosystem services for analytics| reporting| and downstream consumption.Key ResponsibilitiesDesign| develop| and maintain ETL  ELT pipelines using Azure DatabricksBuild data transformation workflows using PySpark (Python) and SQLIngest data from diverse Azure sources such as Azure Data Lake Storage (ADLS Gen2)| Azure SQL| and external systemsImplement and manage Delta Lake tables with appropriate partitioning and optimizationEnsure data quality through validation checks| reconciliation| and error handlingOptimize Spark jobs for performance| scalability| and cost efficiencyIntegrate Databricks pipelines with Azure Data Factory (ADF) or equivalent orchestration toolsCollaborate with data architects| cloud engineers| and business stakeholdersSupport deployment| monitoring| and troubleshooting of production data pipelinesMaintain technical documentation for ETL processes and data flowsFollow Azure cloud security| governance| and compliance best practicesMandatory Skills  QualificationsAzure  DatabricksStrong handson experience with Azure DatabricksProficiency in PySpark and Spark SQLExperience working with Delta Lake (ACID transactions| schema evolution)Handson experience with Azure Data Lake Storage (ADLS Gen2)ETL  Data EngineeringStrong Python scripting skills for ETL developmentAdvanced SQL for data transformation and analyticsSolid understanding of ETL concepts| data warehousing| and data modelingExperience processing largescale| highvolume datasetsCloud PlatformGood understanding of Azure data ecosystemAzure Data Factory (ADF)Azure SQL  Synapse (good to have)Azure Key Vault (access secrets  security best practices)Good to Have  PreferredExperience with Azure Synapse AnalyticsExposure to streaming pipelines (Structured Streaming| Event Hub)CICD exposure using Azure DevOpsExperience with monitoring and logging (Azure Monitor| Log Analytics)Understanding of data security| RBAC| and Azure governanceExperience working in AgileScrum delivery modelsSoft SkillsStrong analytical and problemsolving abilitiesGood communication and stakeholder coordination skillsAbility to work independently and within distributed teamsHigh ownership| attention to detail| and quality focusEducationBachelors or Masters degree in Computer Science| Information Technology| or a related field
Essential Skills: Job Title Azure Databricks ETL EngineerExperience5-8 years Job SummaryWe are seeking an experienced Azure Databricks ETL Engineer with strong handson expertise in ETL development using Python and SQL to design| build| and optimize scalable data pipelines on Azure Databricks. The role focuses on delivering robust| highperformance data solutions leveraging Azure data ecosystem services for analytics| reporting| and downstream consumption.Key ResponsibilitiesDesign| develop| and maintain ETL  ELT pipelines using Azure DatabricksBuild data transformation workflows using PySpark (Python) and SQLIngest data from diverse Azure sources such as Azure Data Lake Storage (ADLS Gen2)| Azure SQL| and external systemsImplement and manage Delta Lake tables with appropriate partitioning and optimizationEnsure data quality through validation checks| reconciliation| and error handlingOptimize Spark jobs for performance| scalability| and cost efficiencyIntegrate Databricks pipelines with Azure Data Factory (ADF) or equivalent orchestration toolsCollaborate with data architects| cloud engineers| and business stakeholdersSupport deployment| monitoring| and troubleshooting of production data pipelinesMaintain technical documentation for ETL processes and data flowsFollow Azure cloud security| gov