Lead Data Engineer(C2/D1)

Job Overview:

We are looking for an accomplished Lead Data Engineer to drive the architectural design, development, and

optimization of our enterprise-scale data ecosystem. In this senior role, you will spearhead the build-out of

high‑performance data pipelines, enforce rigorous data quality and validation frameworks, and ensure end‑to‑end

reliability, scalability, and integrity of data flows across the organization. You will play a pivotal role in shaping our

data engineering strategy, enabling advanced analytics and mission‑critical, data‑driven decision-making within our

insurance-focused business domains.

Key Responsibilities:

• Lead the design and implementation of scalable data architectures on Azure, leveraging Databricks, Delta

Lake, and related Azure data services.

• Build and optimize high‑volume ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, and Delta Live

Tables.

• Define and enforce best practices for data engineering across notebooks, jobs, CI/CD, Unity Catalog,

security, and workspace governance.

• Integrate and orchestrate data pipelines using Azure Data Factory, Azure Synapse pipelines, or Azure

Databricks Workflows.

• Drive performance tuning for Spark jobs, cluster configurations, and data storage layers to balance speed

and cost efficiency.

• Implement robust data quality and validation frameworks using tools like Great Expectations, DLT

expectations, or custom PySpark checks.

• Ensure compliance, governance, and lineage tracking through Unity Catalog, Purview integration, and

RBAC/ABAC policies.

• Architect end‑to‑end data solutions supporting analytics, ML, actuarial models, and business-critical

reporting workloads.

EXL Service

10 Exchange Place, Suite 2200, Jersey City, NJ T: +1.201.748.4700 www.exlservice.com

• Evaluate new Databricks features, MLflow enhancements, Photon execution,

serverless compute, and recommend adoption strategies.

• Familiar with working on Agile methodologies - scrum, sprint planning, backlog refinement etc.

Candidate Profile:

• 8-12 years experience on Data Engineering role working with Databricks & Azure Cloud technologies.

• Bachelor’s degree in computer science, Information Technology, or related field.

• Strong proficiency in PySpark, Python, SQL.

• Strong experience in data modeling, ETL/ELT pipeline development, and automation

• Hands-on experience with performance tuning of data pipelines and workflows

• Proficient in working on Azure cloud components Azure Data Factory, Azure DataBricks, Azure Data Lake

etc.

• Experience with data modeling, ETL processes, Delta Lake and data warehousing.

• Experience on Delta Live Tables, Autoloader & Unity Catalog.

• Preferred - Knowledge of the insurance industry and its data requirements.

• Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of

information with attention to detail and accuracy.

• Excellent communication and problem-solving skills to work effectively with diverse teams

• Excellent problem-solving skills and ability to work under tight deadlines