Position Title      :            Lead Data Engineer

Experience          :            8+ Years

Location               :           Noida/Gurugram/Pune/Bengaluru(Hybrid)

Shift              :            1PM to 10 PM

Duration               :            6 months Contract

We are seeking a Lead Data Engineer with strong, hands-on expertise in PySpark and Databricks to design, build, and optimize scalable data pipelines and lakehouse solutions supporting enterprise analytics platforms. The candidate will lead a small team of engineers, architect robust ETL/ELT frameworks on Databricks, and partner with Data Architects, Analytics, and Business teams to deliver reliable, high-quality, and well-governed data platforms.

Required: 8–10 years overall Data Engineering experience with deep hands-on expertise in PySpark and Databricks. Preferred: Databricks/Azure certification, Delta Lake/Unity Catalog experience, insurance or financial services domain exposure.

The Lead Data Engineer is a hands-on technical leadership role responsible for designing, building, and optimizing large-scale data pipelines on Databricks using PySpark, establishing Lakehouse (medallion) architecture patterns, and guiding a team of engineers to deliver trusted, performant data products. This role works closely with Data Architecture, Data Science, and Business stakeholders to translate requirements into scalable, production-grade data engineering solutions.

Key Responsibilities

A. Data Engineering & Pipeline Development

• Design, develop, and optimize large-scale ETL/ELT pipelines using PySpark on Databricks, processing structured and unstructured data at scale.

• Build and maintain Lakehouse architecture (Bronze/Silver/Gold medallion layers) using Delta Lake, ensuring reliability, scalability, and schema evolution support.

• Develop reusable, parameterized, metadata-driven pipeline frameworks for ingestion, transformation, and curation of data from diverse source systems.

• Optimize Spark jobs for performance and cost (partitioning, caching, cluster sizing/auto-scaling, Photon engine, Z-ordering, file compaction).

• Implement data quality checks, validation rules, and monitoring/alerting to ensure pipeline reliability and data trust.

B. Databricks Platform & Cloud Engineering

• Configure and manage Databricks workspaces, clusters, jobs, and workflows; tune cluster policies for cost and performance.

• Implement data governance, access control, and lineage using Unity Catalog.

• Integrate Databricks with cloud services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, Event Hub/Kafka, and Key Vault (or AWS S3/Glue/EMR equivalents).

• Build and maintain CI/CD pipelines for Databricks notebooks, jobs, and Delta Live Tables using Azure DevOps/GitHub Actions and Databricks Repos.

C. Data Modeling, Architecture & Governance

• Design and maintain data models (dimensional, medallion, canonical) to support analytics, reporting, and downstream consumption.

• Partner with Data Architects on target-state data platform design, migration strategy, and platform standards.

• Ensure adherence to data governance, security, and privacy requirements (data masking, encryption, access controls, regulatory compliance).

D. Leadership & Collaboration

• Lead and mentor a team of data engineers; conduct code reviews and enforce engineering best practices and coding standards.

• Partner with Business Analysts, Data Scientists, and Product Owners to translate requirements into scalable technical solutions.

• Participate in Agile ceremonies, provide effort estimates, and manage delivery timelines and technical risks.

Tools & Environment (Typical)

• Big Data & Processing: PySpark, Databricks, Delta Lake, Spark SQL, Apache Airflow, Delta Live Tables.

• Cloud: Azure (ADLS Gen2, Data Factory, Synapse, Event Hub, Key Vault) or AWS (S3, Glue, EMR).

• Languages: Python, SQL; Scala (nice to have).

• DevOps: Git, Azure DevOps/GitHub Actions, Databricks CLI/API, Terraform (nice to have).

• Orchestration & Monitoring: Databricks Workflows, Airflow, Prometheus/Grafana, Datadog.

Required Qualifications

• 8–10 years of overall Data Engineering experience, including at least 4–5 years of hands-on work with PySpark and Databricks.

• Strong programming skills in Python and expert-level SQL.

• Proven experience building and optimizing large-scale ETL/ELT pipelines and Delta Lake/Lakehouse architectures.

• Solid understanding of Spark internals (partitioning, shuffling, joins, caching) and hands-on performance tuning experience.

• Hands-on experience with cloud data platforms (Azure preferred; AWS/GCP acceptable).

• Experience with CI/CD, version control (Git), and orchestration tools (Airflow/Databricks Workflows).

• Experience leading or mentoring a small team of data engineers.

• Strong communication skills and ability to work effectively with cross-functional and global teams.

Preferred Qualifications

• Databricks Certified Data Engineer Associate/Professional.

• Azure Data Engineer Associate (DP-203) or equivalent cloud certification.

• Experience in the Insurance or Financial Services domain.

• Exposure to Unity Catalog, Delta Live Tables, and MLflow.

• Familiarity with data quality and governance tooling (e.g., Great Expectations, Collibra).

Core Competencies & Behaviors

• Strong problem-solving and analytical thinking.

• Ownership mindset with close attention to data quality and reliability.

• Ability to balance hands-on delivery with technical leadership.

• Collaborative, with clear communication to both technical and business stakeholders.

• Adaptability and effectiveness in a fast-paced, agile environment.

Key Deliverables

• Scalable PySpark/Databricks pipelines (Bronze/Silver/Gold) meeting agreed SLAs.

• Pipeline documentation, runbooks, and monitoring/alerting dashboards.

• Optimized Spark jobs with defined and measured performance benchmarks.

• Data quality and governance framework implementation (Unity Catalog, validation rules).

• Technical mentorship and guidance provided to junior and mid-level engineers.