Lead Data Engineer

Skills : ADB + Lead + Pyspark
Primary Skills : Axure data bricks , Pyspark ,Sql .

Experiece: 9  to 13 years

 Key Responsibilities • Lead a team of Data Engineers in designing, developing, and supporting Databricks-based data solutions.

1. • Build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Spark SQL, and Delta Lake.

2. • Implement and support Medallion Architecture (Bronze, Silver, Gold) for enterprise data platforms

3. . • Develop data ingestion frameworks for batch, CDC, incremental, and streaming data processing.

4. • Collaborate with Architects, Product Owners, Business Analysts, and stakeholders to understand requirements and translate them into technical solutions.

5. • Ensure adherence to coding standards, design patterns, and engineering best practices. • Perform code reviews, technical design reviews, and mentor team members.

6. • Optimize Spark jobs, cluster utilization, and overall platform performance. • Implement data quality checks, reconciliation controls, and monitoring frameworks.

7. • Support CI/CD implementation, source code management, and automated deployment processes. • Troubleshoot production issues and drive root cause analysis and resolution.

8. • Work closely with architecture and governance teams to ensure compliance with security, metadata, lineage, and data governance standards.

9. • Participate in effort estimation, sprint planning, release management, and project delivery activities.