Lead Azure Data Engineer

Lead Azure Data Engineer|| Hamilton, NY|| Rate 62|| 12+

ROLE_DESCRIPTION -

Cloud Platform Expertise
* Deep understanding of Microsoft Azure services like:
* Azure Data Factory (ADF) – ETL/ELT pipelines
* Azure Synapse Analytics – Data warehousing
* Azure Databricks – Big data & machine learning
* Azure Data Lake Storage (ADLS) – Data storage
* Azure SQL Database – Relational data
* Azure Event Hub / IoT Hub – Streaming data ingestion
Data Modeling & Architecture
* Designing data models for structured/unstructured data
* Implementing star/snowflake schemas for analytics
ETL/ELT Development
* Building data pipelines for ingestion, transformation, and loading
* Orchestrating workflows using ADF or Databricks
Programming & Scripting
* SQL (T-SQL for Azure SQL)
* Python or Scala for Databricks
* PowerShell for automation
Big Data & Analytics
* Working with Spark on Databricks
* Handling large-scale data processing
Data Governance & Security
* Implementing RBAC, data encryption, and compliance
* Managing Azure Key Vault for secrets
Monitoring & Optimization
* Using Azure Monitor, Log Analytics
* Performance tuning for pipelines and queries

Roles & Responsibilities
Data Ingestion & Integration
* Collect data from multiple sources (on-prem, cloud, APIs)
* Build scalable ingestion pipelines using ADF or Event Hub
Data Transformation
* Clean, normalize, and enrich data for analytics
* Implement business logic in ETL/ELT workflows
Data Storage & Management
* Design and maintain Data Lake and Data Warehouse
* Optimize storage for cost and performance
Collaboration
* Work with Data Scientists, BI Developers, and Architects
* Ensure data availability and quality for analytics
Security & Compliance
* Apply data governance policies
* Ensure compliance with GDPR, HIPAA, etc.
Performance & Cost Optimization
* Monitor pipelines and optimize for speed and cost
* Scale resources dynamically using Azure features