Databricks Platform Engineer / Manager (AWS)

Job Summary

Own the design, administration, security, governance, performance, automation, and cost optimization

of the enterprise Databricks platform on AWS. Enable data engineering, analytics, and AI teams through

scalable, secure, and reliable platform operations.

Key Responsibilities

● Manage Databricks workspaces across environments (Dev, Test, UAT, Prod).

● Maintain platform availability, upgrades, performance, and reliability.

● Define cluster policies, instance pools, and standardized configurations.

● Integrate and manage AWS IAM, SSO, authentication, authorization, PATs, service principals, and

secret scopes.

● Administer Unity Catalog, RBAC, governance, compliance, and audit controls.

● Support Git integration, CI/CD pipelines, Terraform-based IaC, and automated deployments.

● Monitor platform health, troubleshoot incidents, and support production operations.

● Optimize Databricks usage, cluster sizing, auto-scaling, and cloud costs.

● Maintain audit logging, security compliance, and operational documentation.

Required Skills

● Databricks: Workspace administration, clusters, cluster policies, instance pools, workflows,

Repos, Unity Catalog, DBR management.

● AWS: IAM, VPC, EC2, S3, CloudWatch, KMS, networking, security groups.

● DevOps: Git, CI/CD, Terraform, IaC, Azure DevOps/GitHub Actions/Jenkins.

● Operations: Monitoring, incident management, troubleshooting, performance tuning, cost

optimization.

● Strong cloud architecture, networking, security, scalability, HA, and production support

experience.

Required Experience

● Enterprise Databricks administration across multiple workspaces.

● Unity Catalog and AWS IAM integration.

● Cluster governance, runtime upgrades, CI/CD, Git integration, monitoring, audit logging, and

secrets management.

● Terraform, DevOps, production support, and cost optimization initiatives.

Preferred Qualifications

● Large-scale Databricks environments (100+ users).

● Terraform-based provisioning.

● Lakehouse architecture, Delta Lake, FinOps, governance, and security frameworks.

Nice to Have

● AWS Organizations, Control Tower, PrivateLink, VPC endpoints.

● CloudWatch, Datadog, Splunk.

● Apache Spark internals and AI/ML workloads on Databricks.

Education & Certifications

● Bachelor's degree in a relevant field.

● Preferred: Databricks certifications, AWS Solutions Architect, AWS SysOps Administrator,

Terraform Associate.

Ideal Candidate

A platform engineering professional with strong AWS and Databricks expertise, capable of owning

platform operations, security, governance, automation, and cost optimization while collaborating with

engineering, DevOps, and security teams.