Work Location: Remote (India)

Candidates should be flexible to work from an EXL office whenever business requires. While the role is primarily remote, it is not a permanent work-from-home position.

Key Responsibilities

• Design, build, and productionise batch and incremental ETL/ELT pipelines using Databricks (notebooks, Jobs/Workflows, DLT), PySpark, and Spark SQL across bronze/silver/gold Delta Lake layers.

• Implement the medallion architecture with Delta Lake features — ACID merges/upserts (SCD1/SCD2), schema evolution/enforcement, time travel, OPTIMIZE/Z-ORDER, vacuuming — and manage data with Unity Catalog.

• Orchestrate workloads via Databricks Workflows and/or Azure Data Factory; parameterise pipelines, implement restartability, checkpointing, and idempotent re-runs.

• Tune Spark for performance and cost: partitioning strategy, join/broadcast optimisation, skew handling, caching, cluster sizing/autoscaling, Photon usage, and job-level cost monitoring.

• Build data quality checks (expectations, reconciliation against source, row/measure-level controls) and implement failure alerting, logging, and lineage-friendly design.

• Ingest from diverse sources — RDBMS (SQL Server/Oracle), files (CSV/Parquet/JSON/XML), APIs, and streaming/CDC feeds — into ADLS Gen2 landing zones.

• Contribute to CI/CD for data: Git-based development, pull-request reviews, Azure DevOps/GitHub Actions pipelines, environment promotion, and infrastructure/config as code.

• Collaborate with Data Modelers, BDAs, testers, and onshore leads in Agile ceremonies; convert mapping specifications into robust, reviewed code; mentor junior engineers.

• Provide L3 production support for owned pipelines: triage incidents, perform root-cause analysis, and drive permanent fixes within SLAs.

Must-Have Skills & Experience

• 9–12 years overall; strong recent hands-on Databricks and PySpark delivery experience (3+ years Databricks preferred).

• Expert-level Spark (DataFrame API, Spark SQL, window functions, UDF trade-offs) and advanced Python for data engineering.

• Advanced SQL — complex transformations, performance tuning, analytical/window queries on large volumes.

• Delta Lake internals and medallion/lakehouse architecture in production.

• Azure data stack: ADLS Gen2, Azure Data Factory, Key Vault, and Databricks administration basics (clusters, pools, jobs, secrets).

• Data warehousing concepts: star schemas, fact/dimension loading patterns, SCD handling, reconciliation.

• Git-based SDLC with CI/CD exposure (Azure DevOps or GitHub).

• Insurance or financial-services data experience — ideally P&C (policy/claims/premium/reinsurance structures).

Good-to-Have

• Delta Live Tables, Unity Catalog governance, Databricks Asset Bundles.

• Streaming (Structured Streaming, Auto Loader, Event Hubs/Kafka) and CDC tools.

• dbt on Databricks; Airflow; Terraform.

• Guidewire, Duck Creek, or London-market source systems; Snowflake or Synapse coexistence.

• Databricks Data Engineer Associate/Professional certification.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.

Professional & Communication Skills

• Clear written and spoken English; comfortable presenting design decisions to onshore architects and client stakeholders.

• Proven offshore-delivery discipline: crisp status reporting, proactive risk flagging, dependable overlap-hours availability.

• Ownership mindset — drives issues to closure without follow-up.

Work Location: Remote (India)

Candidates should be flexible to work from an EXL office whenever business requires. While the role is primarily remote, it is not a permanent work-from-home position.

Key Responsibilities