Pyspark Developer (Data Engineer)

Description: Responsibilities:

• Development and Maintain Data Pipelines: Design, implement, and optimize end-to-end ETL/ELT pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.

• Utilize Python and Pyspark: Write efficient, scalable and maintainable code in Python and leverage Pyspark for large-scale data processing in distributed computing environments. Also be able to review existing code and identify areas of improvement.

• Ensure Data Quality and Integrity: Implement data validation, cleansing, transformation and reconciliation processes to ensure data accuracy and consistency throughout the data lifecycle

• Collaborate with Stakeholders: Work closely with IT teams and business stakeholders to gather data requirements and translate them to technical solutions.

• Troubleshoot and Optimize: Monitor job performance, troubleshoot complex data issues and fine-tune for performance and scalability.

• Adhere to Best Practices: Participate in code reviews, establish coding standards, and implement CI/CD pipelines for automated testing and deployment.

Skills:

• Strong hands-on coding proficiency in Python, Pyspark and SQL (Microsoft SQL Server preferred)

• Experience with big data frameworks (Hadoop, Spark).

• Experience with cloud platforms ( AWS, Azure or GCP)

• Experience with Code versioning tools ( Bitbucket, Github )

• Experience with CI/CD and setting up pipelines.

• Solid understanding of database design principles, data modelling, schemas and data warehousing solutions. • Excellent problem-solving and analytical skills to troubleshoot complex data issues independently.