Senior Databricks Data Architect

Job Title: Senior Databricks Data Architect (AWS) - Hands-On Technical Lead

Location: New Jersey (100% onsite, Monday to Friday)

Engagement: Contract, onsite role — local candidates with lead experience only

Purpose of the Role
Lead the strategic design and architectural roadmap of the Enterprise Data Platform on Databricks/AWS while remaining hands-on in design, prototyping, coding, and performance tuning.
Provide high-level technical consultancy and solution design to internal stakeholders and external partners, translating complex business requirements into technical implementations.
Govern platform technical quality by establishing design standards, conducting code reviews, and guiding engineering teams on Databricks Serverless and AWS integration best practices.
Provide technical assurance for major data initiatives ensuring solutions are robust, cost-effective, and compliant with enterprise security standards.
Work onsite Monday through Friday during core business hours (note: the description references EST and also CST for hours — candidates should be able to operate in the required enterprise timezone).
Key Accountabilities
Lead architecture design and review sessions for strategic Databricks initiatives.
Actively contribute production code for architectural components, building frameworks, reusable libraries, and core ETL/ELT pipelines using Python and PySpark.
Prototype Proofs of Concept (e.g., Databricks Asset Bundles, Delta Live Tables) and hand off for broader implementation.
Perform hands-on performance tuning and debugging of high-latency Spark jobs via Spark UI, execution plans, and spill logs.
Define and enforce coding standards, data modeling best practices (Medallion Architecture), and CI/CD protocols.
Conduct deep technical reviews of designs and code to ensure efficiency, maintainability, and security.
Design Unity Catalog governance strategy for granular data access controls and regulatory compliance.
Act as Technical Design Authority for Databricks platform decisions, including adoption of Serverless features and AWS services.
Mentor and coach senior and junior engineers to foster technical excellence and continuous improvement.
Own technical relationships with business stakeholders and translate strategy into architecture.
Engage with executive stakeholders to present roadmaps and demonstrate business value of data initiatives.
Explore emerging technologies (e.g., Generative AI on Databricks, vector databases) and evaluate business impact.
Advise change management on risk and technical impact of architectural changes.
Architect resilience, high availability, and disaster recovery into the data platform.
Key Liaisons
Head of Data & Analytics / CIO
Business Unit Directors & Product Managers
Lead Data Engineers and Data Scientists
Enterprise Security & Infrastructure Architects
Dimensions & Accountability
Ownership of the technical roadmap and architecture for the data estate.
Responsible for architectural decisions impacting critical data assets.
Primary technical advisor for the CST timezone (note: earlier description also references EST).
Education
Degree in Computer Science, Engineering, or equivalent (Master's preferred).
Essential Skills & Attributes
8–10+ years in Data Engineering/Architecture with at least 4 years specialized in Databricks and AWS.
Expert knowledge of Data Lakehouse architecture, Delta Lake internals, and Spark optimization strategies.
Proven data modeling experience (Dimensional, Vault, Relational) and designing complex ETL/ELT pipelines.
Advanced hands-on coding in Python and SQL (production-ready, testable code); familiarity with Scala is a plus.