All fields marked * are required
Position Title : Lead Data Engineer
Experience : 8+ Years
Location : Noida/Gurugram/Pune/Bengaluru(Hybrid)
Shift          : 1PM to 10 PM
Duration : 6 months Contract
We are seeking a Lead Data Engineer with strong, hands-on expertise in PySpark and Databricks to design, build, and optimize scalable data pipelines and lakehouse solutions supporting enterprise analytics platforms. The candidate will lead a small team of engineers, architect robust ETL/ELT frameworks on Databricks, and partner with Data Architects, Analytics, and Business teams to deliver reliable, high-quality, and well-governed data platforms.
Required: 8–10 years overall Data Engineering experience with deep hands-on expertise in PySpark and Databricks. Preferred: Databricks/Azure certification, Delta Lake/Unity Catalog experience, insurance or financial services domain exposure.
The Lead Data Engineer is a hands-on technical leadership role responsible for designing, building, and optimizing large-scale data pipelines on Databricks using PySpark, establishing Lakehouse (medallion) architecture patterns, and guiding a team of engineers to deliver trusted, performant data products. This role works closely with Data Architecture, Data Science, and Business stakeholders to translate requirements into scalable, production-grade data engineering solutions.
Design, develop, and optimize large-scale ETL/ELT pipelines using PySpark on Databricks, processing structured and unstructured data at scale.
Build and maintain Lakehouse architecture (Bronze/Silver/Gold medallion layers) using Delta Lake, ensuring reliability, scalability, and schema evolution support.
Develop reusable, parameterized, metadata-driven pipeline frameworks for ingestion, transformation, and curation of data from diverse source systems.
Optimize Spark jobs for performance and cost (partitioning, caching, cluster sizing/auto-scaling, Photon engine, Z-ordering, file compaction).
Implement data quality checks, validation rules, and monitoring/alerting to ensure pipeline reliability and data trust.
Configure and manage Databricks workspaces, clusters, jobs, and workflows; tune cluster policies for cost and performance.
Implement data governance, access control, and lineage using Unity Catalog.
Integrate Databricks with cloud services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, Event Hub/Kafka, and Key Vault (or AWS S3/Glue/EMR equivalents).
Build and maintain CI/CD pipelines for Databricks notebooks, jobs, and Delta Live Tables using Azure DevOps/GitHub Actions and Databricks Repos.
Design and maintain data models (dimensional, medallion, canonical) to support analytics, reporting, and downstream consumption.
Partner with Data Architects on target-state data platform design, migration strategy, and platform standards.
Ensure adherence to data governance, security, and privacy requirements (data masking, encryption, access controls, regulatory compliance).
Lead and mentor a team of data engineers; conduct code reviews and enforce engineering best practices and coding standards.
Partner with Business Analysts, Data Scientists, and Product Owners to translate requirements into scalable technical solutions.
Participate in Agile ceremonies, provide effort estimates, and manage delivery timelines and technical risks.
Big Data & Processing: PySpark, Databricks, Delta Lake, Spark SQL, Apache Airflow, Delta Live Tables.
Cloud: Azure (ADLS Gen2, Data Factory, Synapse, Event Hub, Key Vault) or AWS (S3, Glue, EMR).
Languages: Python, SQL; Scala (nice to have).
DevOps: Git, Azure DevOps/GitHub Actions, Databricks CLI/API, Terraform (nice to have).
Orchestration & Monitoring: Databricks Workflows, Airflow, Prometheus/Grafana, Datadog.
8–10 years of overall Data Engineering experience, including at least 4–5 years of hands-on work with PySpark and Databricks.
Strong programming skills in Python and expert-level SQL.
Proven experience building and optimizing large-scale ETL/ELT pipelines and Delta Lake/Lakehouse architectures.
Solid understanding of Spark internals (partitioning, shuffling, joins, caching) and hands-on performance tuning experience.
Hands-on experience with cloud data platforms (Azure preferred; AWS/GCP acceptable).
Experience with CI/CD, version control (Git), and orchestration tools (Airflow/Databricks Workflows).
Experience leading or mentoring a small team of data engineers.
Strong communication skills and ability to work effectively with cross-functional and global teams.
Databricks Certified Data Engineer Associate/Professional.
Azure Data Engineer Associate (DP-203) or equivalent cloud certification.
Experience in the Insurance or Financial Services domain.
Exposure to Unity Catalog, Delta Live Tables, and MLflow.
Familiarity with data quality and governance tooling (e.g., Great Expectations, Collibra).
Strong problem-solving and analytical thinking.
Ownership mindset with close attention to data quality and reliability.
Ability to balance hands-on delivery with technical leadership.
Collaborative, with clear communication to both technical and business stakeholders.
Adaptability and effectiveness in a fast-paced, agile environment.
Scalable PySpark/Databricks pipelines (Bronze/Silver/Gold) meeting agreed SLAs.
Pipeline documentation, runbooks, and monitoring/alerting dashboards.
Optimized Spark jobs with defined and measured performance benchmarks.
Data quality and governance framework implementation (Unity Catalog, validation rules).
Technical mentorship and guidance provided to junior and mid-level engineers.
Required
Preferred
All fields marked * are required
Position Title : Lead Data Engineer
Experience : 8+ Years
Location : Noida/Gurugram/Pune/Bengaluru(Hybrid)
Shift          : 1PM to 10 PM
Duration : 6 months Contract
We are seeking a Lead Data Engineer with strong, hands-on expertise in PySpark and Databricks to design, build, and optimize scalable data pipelines and lakehouse solutions supporting enterprise analytics platforms. The candidate will lead a small team of engineers, architect robust ETL/ELT frameworks on Databricks, and partner with Data Architects, Analytics, and Business teams to deliver reliable, high-quality, and well-governed data platforms.
Required: 8–10 years overall Data Engineering experience with deep hands-on expertise in PySpark and Databricks. Preferred: Databricks/Azure certification, Delta Lake/Unity Catalog experience, insurance or financial services domain exposure.
The Lead Data Engineer is a hands-on technical leadership role responsible for designing, building, and optimizing large-scale data pipelines on Databricks using PySpark, establishing Lakehouse (medallion) architecture patterns, and guiding a team of engineers to deliver trusted, performant data products. This role works closely with Data Architecture, Data Science, and Business stakeholders to translate requirements into scalable, production-grade data engineering solutions.
Design, develop, and optimize large-scale ETL/ELT pipelines using PySpark on Databricks, processing structured and unstructured data at scale.
Build and maintain Lakehouse architecture (Bronze/Silver/Gold medallion layers) using Delta Lake, ensuring reliability, scalability, and schema evolution support.
Develop reusable, parameterized, metadata-driven pipeline frameworks for ingestion, transformation, and curation of data from diverse source systems.
Optimize Spark jobs for performance and cost (partitioning, caching, cluster sizing/auto-scaling, Photon engine, Z-ordering, file compaction).
Implement data quality checks, validation rules, and monitoring/alerting to ensure pipeline reliability and data trust.
Configure and manage Databricks workspaces, clusters, jobs, and workflows; tune cluster policies for cost and performance.
Implement data governance, access control, and lineage using Unity Catalog.
Integrate Databricks with cloud services such as Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, Event Hub/Kafka, and Key Vault (or AWS S3/Glue/EMR equivalents).
Build and maintain CI/CD pipelines for Databricks notebooks, jobs, and Delta Live Tables using Azure DevOps/GitHub Actions and Databricks Repos.
Design and maintain data models (dimensional, medallion, canonical) to support analytics, reporting, and downstream consumption.
Partner with Data Architects on target-state data platform design, migration strategy, and platform standards.
Ensure adherence to data governance, security, and privacy requirements (data masking, encryption, access controls, regulatory compliance).
Lead and mentor a team of data engineers; conduct code reviews and enforce engineering best practices and coding standards.
Partner with Business Analysts, Data Scientists, and Product Owners to translate requirements into scalable technical solutions.
Participate in Agile ceremonies, provide effort estimates, and manage delivery timelines and technical risks.
Big Data & Processing: PySpark, Databricks, Delta Lake, Spark SQL, Apache Airflow, Delta Live Tables.
Cloud: Azure (ADLS Gen2, Data Factory, Synapse, Event Hub, Key Vault) or AWS (S3, Glue, EMR).
Languages: Python, SQL; Scala (nice to have).
DevOps: Git, Azure DevOps/GitHub Actions, Databricks CLI/API, Terraform (nice to have).
Orchestration & Monitoring: Databricks Workflows, Airflow, Prometheus/Grafana, Datadog.
8–10 years of overall Data Engineering experience, including at least 4–5 years of hands-on work with PySpark and Databricks.
Strong programming skills in Python and expert-level SQL.
Proven experience building and optimizing large-scale ETL/ELT pipelines and Delta Lake/Lakehouse architectures.
Solid understanding of Spark internals (partitioning, shuffling, joins, caching) and hands-on performance tuning experience.
Hands-on experience with cloud data platforms (Azure preferred; AWS/GCP acceptable).
Experience with CI/CD, version control (Git), and orchestration tools (Airflow/Databricks Workflows).
Experience leading or mentoring a small team of data engineers.
Strong communication skills and ability to work effectively with cross-functional and global teams.
Databricks Certified Data Engineer Associate/Professional.
Azure Data Engineer Associate (DP-203) or equivalent cloud certification.
Experience in the Insurance or Financial Services domain.
Exposure to Unity Catalog, Delta Live Tables, and MLflow.
Familiarity with data quality and governance tooling (e.g., Great Expectations, Collibra).
Strong problem-solving and analytical thinking.
Ownership mindset with close attention to data quality and reliability.
Ability to balance hands-on delivery with technical leadership.
Collaborative, with clear communication to both technical and business stakeholders.
Adaptability and effectiveness in a fast-paced, agile environment.
Scalable PySpark/Databricks pipelines (Bronze/Silver/Gold) meeting agreed SLAs.
Pipeline documentation, runbooks, and monitoring/alerting dashboards.
Optimized Spark jobs with defined and measured performance benchmarks.
Data quality and governance framework implementation (Unity Catalog, validation rules).
Technical mentorship and guidance provided to junior and mid-level engineers.
Required
Preferred