Key Responsibilities

Data Pipeline Testing:

· Validate end-to-end data ingestion, transformation, and consumption pipeline across bronze, silver, and gold layers.

· Perform source-to-target data validation for structured.

· Validate business rules and data transformations.

· Ensure data consistency, completeness, uniqueness and referential integrity.

Automation Testing:

· Design and develop automated test frameworks for data validation and regression testing.

· Automate ETL and data quality testing using python, SQL, PySpark, or testing frameworks.

· Build reusable validation scripts for schema validation, reconciliation, and data drift detection.

· Integrate automated test suites into CI/CD pipelines.

Databricks and Cloud Validation:

· Validate Databricks notebooks, workflows, Delta Tables and job orchestration.

· Test Delta Lake functionalities include ACID compliance, schema evolution and time travel.

· Validate performance and scalability of Spark-based transformations.