Lead Software Engineer

Job Description:
·      Design, build, and maintain scalable data pipelines using Azure Databricks, PySpark, Spark SQL, and Delta Lake. 
·      Develop high-performance ETL/ELT workflows for processing structured and unstructured data. 
·      Optimize Spark clusters, jobs, and queries to improve performance and reduce cloud cost.
·      Develop and maintain data lakehouse architecture on Azure Data Lake Storage (ADLS). 
·      Work with Azure Data Factory (ADF) or Synapse pipelines to orchestrate and automate data workflows. 
·      Implement CI/CD for Databricks notebooks using Git integration and DevOps pipelines.
·      Work with Azure Data Factory (ADF) or Synapse pipelines to orchestrate and automate data workflows.
·      Ensure data quality and lineage through tools like SQL, Delta Live Tables, Unity Catalog, and governance frameworks.
·      Collaborate with product, analytics, and engineering teams to understand data requirements.
·      Implement data security practices including RBAC, tokenization, and workspace permissions.