Lead Big Data Engineer

Job Title: Lead Big Data Engineer
Location: Owings Mills, MD
Pay Range- $55-60/hr 
Job Description

10+ years of experience in big data and distributed computing

Very strong hands-on experience with PySpark, Apache Spark, and Python

Strong hands-on experience with SQL and NoSQL databases (DB2, PostgreSQL, Snowflake, etc.)

Proficiency in data modeling and ETL workflows

Proficiency with workflow schedulers such as Airflow

Hands-on experience with AWS cloud-based data platforms

Experience in DevOps, CI/CD pipelines, and containerization (Docker, Kubernetes) is a plus

Strong problem-solving skills and ability to lead a team
Responsibilities

Lead the design, development, and deployment of PySpark-based big data solutions

Architect and optimize ETL pipelines for structured and unstructured data

Collaborate with clients, data engineers, data scientists, and business teams to understand requirements and deliver scalable solutions

Optimize Apache Spark performance through partitioning, caching, and tuning

Implement best practices in data engineering, including CI/CD, version control, and unit testing

Work with AWS cloud platforms

Ensure data security, governance, and compliance

Mentor junior developers and review code for best practices and efficiency