About the role:
We are looking for a highly skilled Senior Data Engineer with strong expertise in Azure Databricks, PySpark, and T-SQL to support a large-scale enterprise data modernization initiative. This role requires strong technical ownership, architectural thinking, and the ability to mentor and guide other data engineers while delivering scalable, high-performance data solutions on Azure.

Key Responsibilities:

• Analyze and modernize complex data transformation logic written in T-SQL and implement optimized solutions using Databricks and PySpark.

• Design and build end-to-end data ingestion pipelines using Azure Data Factory.

• Manage data movement from multiple source systems into RAW and downstream data layers ensuring reliability and scalability.

• Define, capture, and analyze data pipeline performance metrics to proactively identify bottlenecks.

• Perform advanced Databricks performance tuning, including partitioning, caching, file format optimization (Delta/Parquet), and query optimization.

• Optimize large-scale read/write workloads within Databricks environments.

• Implement monitoring, alerting, and best practices for pipeline reliability and performance.

• Provide technical mentorship and guidance to junior data engineers.

• Conduct design and code reviews and drive adoption of engineering best practices.

• Collaborate with architects, analysts, and business stakeholders to deliver enterprise-grade data solutions.
  

Mandatory Skills & Experience:

• 6+ years of experience in Data Engineering.

• Minimum 3+ years of hands-on experience with Azure DatabricksStrong expertise in Core SQL and complex T-SQL.

• Hands-on experience with PySparkProven experience with Azure Data Factory.

• Strong experience in performance optimization of large-scale data pipelines.
  

Education:

• Bachelor’s degree in Computer Science, Engineering, IT, or a related discipline.