Back to all jobs
Cosette Network

Role: Databricks Architect

Noida, Bengaluru, Gurgaon, Pune, Mumbaihybrid12.0 - 15.0 years2 openings

Apply for this position

All fields marked * are required

Name & contact details are extracted from your resume automatically.

Job Description

Key Role & Responsibilities

  • Define lakehouse architecture: medallion (bronze/silver/gold) patterns, batch/streaming designs, and multi-workspace strategies.

  • Design and implement data pipelines using Spark, Delta Lake, and Databricks workflows (Jobs/Workflows, DLT where applicable).

  • Establish governance and security using Unity Catalog, access controls, lineage, and data quality gates.

  • Optimise performance: cluster policies, autoscaling, partitioning, file sizing, caching, Spark tuning, and job orchestration.

  • Build CI/CD and release governance for notebooks, repos, jobs, and infrastructure-as-code.

  • Integrate Databricks with enterprise ecosystem (cloud storage, event streaming, data warehouse, BI tools).

  • Conduct solution workshops with customers; provide options and trade-offs; create phased implementation roadmaps aligned to business value.

  • Mentor teams, enforce engineering standards, and ensure operational excellence (monitoring, incident response, SRE practices).

 

Must Have

  • 12+ years experience with a strong Data Engineering background (ETL/ELT, distributed compute, production-grade pipelines).

  • 5+ years hands-on Databricks experience in architecture/technical leadership roles.

  • Strong experience in Apache Spark (PySpark/Scala), Delta Lake, pipeline design, and performance tuning.

  • At least 1 Databricks Certification.

  • Experience with data orchestration and DevOps practices (Git, CI/CD, testing frameworks).

  • Experience designing secure data platforms (RBAC, secrets, network/security integration, compliance considerations).

  • Strong customer-facing skills: requirements discovery, solution design, and stakeholder management.

 

 

Good to Have

  • Streaming experience (Kafka/Event Hubs, Structured Streaming, CDC patterns).

  • ML/AI enablement experience (MLflow, feature engineering, model lifecycle) as it relates to platform design.

  • Cloud certifications or platform-specific certifications.

 

Education

  • Bachelor’s/Master’s in Computer Science, Engineering, or related fields.

 

Key Skills: Databricks, Python, Spark, Data Architecture, Data Pipelines

 

Skills

Required

Databricks certification mustexperience working on Databricks architecture for 5 years