Data Engineering Lead

We are looking for a hands-on Data Engineering Lead to support the modernization and integration of data across applications. This role will partner closely with API, architecture, and ACE modernization teams to design reusable data structures, optimize data pipelines, and enable reporting across multiple domains.
The ideal candidate has deep SQL expertise, strong PySpark experience, and a track record of building scalable, production-grade data pipelines on cloud platforms. This is a technical leadership role that requires strong communication skills and the ability to work directly with cross-functional teams.

Location: Bethlehem PA OR Holmdel NJ
Duration: Long term contract
Client: One of the largest Insurance company in North America

Key Responsibilities
Data Integration & Pipeline Development
•    Lead the integration of data from FINEOS (FIN) into the Data Lake, spanning Bronze to Silver layers.
•    Build generalized, reusable, reporting-ready data structures, not one-off datasets.
•    Design and maintain domain-standard tables, similar to warehouse semantic layers.
•    Develop SQL and PySpark pipelines to populate standardized data models.
•    Extend existing schemas by identifying missing attributes required for reports.

Data Exploration & Analysis
•    Perform deep data exploration of raw tables and conduct attribute-level mapping.
•    Analyze business requirements and translate them into technical data models.
•    Document logic, transformations, and lineage to support long-term maintainability.

Engineering & Delivery
•    Understand and enhance existing scripts, frameworks, and ETL patterns.
•    Write optimized SQL for high-performance transformations.
•    Integrate new attributes into existing transformation pipelines.
•    Participate in peer reviews and ensure adherence to engineering best practices.
•    Collaborate closely with data architects, API teams, and modernization stakeholders.
•    Deploy code to production and support post-deployment validation.
•    Potentially extend work into additional reporting domains in later phases.

Required Skills
Technical Expertise
•    Very strong SQL (must be exceptional; this is the most critical skill).
•    PySpark and Python for large-scale data transformations.
•    Databricks experience preferred; candidates with strong SQL can learn it quickly.
•    AWS exposure, ideally in environments where Databricks runs on AWS.
•    Ability to perform deep data exploration, schema analysis, and attribute-level mapping.
•    Strong analytical aptitude and critical thinking.