About the role:
We are looking for a highly skilled Senior Data Engineer with strong expertise in Azure Databricks, PySpark, and T-SQL to support a large-scale enterprise data modernization initiative. This role requires strong technical ownership, architectural thinking, and the ability to mentor and guide other data engineers while delivering scalable, high-performance data solutions on Azure.

Key Responsibilities:

Analyze and modernize complex data transformation logic written in T-SQL and implement optimized solutions using Databricks and PySpark.
Design and build end-to-end data ingestion pipelines using Azure Data Factory.
Manage data movement from multiple source systems into RAW and downstream data layers ensuring reliability and scalability.
Define, capture, and analyze data pipeline performance metrics to proactively identify bottlenecks.
Perform advanced Databricks performance tuning, including partitioning, caching, file format optimization (Delta/Parquet), and query optimization.
Optimize large-scale read/write workloads within Databricks environments.
Implement monitoring, alerting, and best practices for pipeline reliability and performance.
Provide technical mentorship and guidance to junior data engineers.
Conduct design and code reviews and drive adoption of engineering best practices.
Collaborate with architects, analysts, and business stakeholders to deliver enterprise-grade data solutions.

Mandatory Skills & Experience:

6+ years of experience in Data Engineering.
Minimum 3+ years of hands-on experience with Azure DatabricksStrong expertise in Core SQL and complex T-SQL.
Hands-on experience with PySparkProven experience with Azure Data Factory.
Strong experience in performance optimization of large-scale data pipelines.

Education:

Bachelor’s degree in Computer Science, Engineering, IT, or a related discipline.

Analyze and modernize complex data transformation logic written in T-SQL and implement optimized solutions using Databricks and PySpark.
Design and build end-to-end data ingestion pipelines using Azure Data Factory.
Manage data movement from multiple source systems into RAW and downstream data layers ensuring reliability and scalability.
Define, capture, and analyze data pipeline performance metrics to proactively identify bottlenecks.
Perform advanced Databricks performance tuning, including partitioning, caching, file format optimization (Delta/Parquet), and query optimization.
Optimize large-scale read/write workloads within Databricks environments.
Implement monitoring, alerting, and best practices for pipeline reliability and performance.
Provide technical mentorship and guidance to junior data engineers.
Conduct design and code reviews and drive adoption of engineering best practices.
Collaborate with architects, analysts, and business stakeholders to deliver enterprise-grade data solutions.

Mandatory Skills & Experience:

6+ years of experience in Data Engineering.
Minimum 3+ years of hands-on experience with Azure DatabricksStrong expertise in Core SQL and complex T-SQL.
Hands-on experience with PySparkProven experience with Azure Data Factory.
Strong experience in performance optimization of large-scale data pipelines.

Education:

Bachelor’s degree in Computer Science, Engineering, IT, or a related discipline.

Analyze and modernize complex data transformation logic written in T-SQL and implement optimized solutions using Databricks and PySpark.
Design and build end-to-end data ingestion pipelines using Azure Data Factory.
Manage data movement from multiple source systems into RAW and downstream data layers ensuring reliability and scalability.
Define, capture, and analyze data pipeline performance metrics to proactively identify bottlenecks.
Perform advanced Databricks performance tuning, including partitioning, caching, file format optimization (Delta/Parquet), and query optimization.
Optimize large-scale read/write workloads within Databricks environments.
Implement monitoring, alerting, and best practices for pipeline reliability and performance.
Provide technical mentorship and guidance to junior data engineers.
Conduct design and code reviews and drive adoption of engineering best practices.
Collaborate with architects, analysts, and business stakeholders to deliver enterprise-grade data solutions.

Mandatory Skills & Experience:

6+ years of experience in Data Engineering.
Minimum 3+ years of hands-on experience with Azure DatabricksStrong expertise in Core SQL and complex T-SQL.
Hands-on experience with PySparkProven experience with Azure Data Factory.
Strong experience in performance optimization of large-scale data pipelines.

Education:

Bachelor’s degree in Computer Science, Engineering, IT, or a related discipline.