Job Description

Job Location: Pune/ Noida/Bengaluru/ Gurugram

Work mode: 5 days(2 days WFO )

Total Experience: 4-6yrs

QA Automation Engineer – Data Engineering:

Role Summary:

We are looking for a QA Automation Engineer with strong expertise in Data Engineering validation and automated testing for modern cloud-based data platforms. The role involves validating end-to-end data pipelines built on Medallion Architecture (Bronze, Silver, Gold layers) in Databricks, ensuring data quality, transformation accuracy, scalability, and reliability across batch and near real-time workloads.

The ideal candidate should have experience in automation frameworks, ETL/Data testing, cloud data platforms, and CI/CD integration within enterprise-grade analytics ecosystem.

Key Responsibilities

Data Pipeline Testing:

• Validate end-to-end data ingestion, transformation, and consumption pipeline across bronze, silver, and gold layers.

• Perform source-to-target data validation for structured.

• Validate business rules and data transformations.

• Ensure data consistency, completeness, uniqueness and referential integrity.

Automation Testing:

• Design and develop automated test frameworks for data validation and regression testing.

• Automate ETL and data quality testing using python, SQL, PySpark, or testing frameworks.

• Build reusable validation scripts for schema validation, reconciliation, and data drift detection.

• Integrate automated test suites into CI/CD pipelines.

Databricks and Cloud Validation:

• Validate Databricks notebooks, workflows, Delta Tables and job orchestration.

• Test Delta Lake functionalities include ACID compliance, schema evolution and time travel.

• Validate performance and scalability of Spark-based transformations.

Required Skills:

• SQL

• Python

• PySpark

• Databricks

• Batch Pipelines

• Incremental Loads

• CDC pipelines

• Knowledge of Medallion Architecture concepts

Preferred Qualifications:

• 4-6 Years of experience in QA/Data Testing.

• 2+ years of experience in Databricks/Data Lakehouse projects.

• Experience working in Agile/Scrum delivery models.

• Certification in Databricks, Cloud or Testing tool is an advantage.