EY - GDS Consulting - AI And DATA - Databricks Platform Engineer-Gig

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. 

Job Title

EY – GDS Consulting – AI and Data – Azure Databricks Platform Engineer / Administrator – Senior

At EY

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all.

Job Summary

We are looking for an experienced Azure Databricks Platform Engineer / Administrator with 5–10 years of experience in cloud-based big data platforms and 2+ years of hands-on Azure Databricks administration.

The ideal candidate will design, provision, and operate enterprise-scale Databricks workspaces, enforce security, cost, and performance governance, and act as an escalation point for data engineering and ML teams. This role focuses on delivering a reliable, secure, and well-governed Databricks platform that accelerates product teams while meeting enterprise compliance requirements.

Key Responsibilities

• Platform Administration & Governance
  • Design, provision, and operate Azure Databricks workspaces using best practices.
  • Create and manage clusters, cluster policies, and instance pools.
  • Plan and execute Databricks Runtime (DBR) upgrades.
  • Configure VNet-injected workspaces, network security controls, and private endpoints.
  • Implement SCIM-based user and group provisioning with Azure AD.
  • Configure and manage secret scopes using Azure Key Vault.
  • Monitor jobs, audit logs, and Databricks system tables to identify platform health, performance, and cost anomalies.

• Unity Catalog Stewardship
  • Set up and manage Unity Catalog metastores, catalogs, and schemas.
  • Implement fine-grained access control, including object ownership and privilege delegation.
  • Enable row-level and column-level security.
  • Support data lineage, auditing, and governance using Unity Catalog features.

• ETL, Streaming & Delta Live Tables Support
  • Partner with data engineering teams to build, operate, and optimize Spark and Delta Lake pipelines.
  • Support batch, streaming, and Delta Live Tables (DLT) workloads.
  • Assist with Databricks Workflows for orchestration and dependency management.
  • Troubleshoot and resolve performance and reliability issues in pipelines.

• Security & Data Privacy
  • Integrate Azure Active Directory (AAD) single sign-on (SSO).
  • Enforce enterprise security standards across identity, network, and data access layers.
  • Support data privacy and compliance requirements for regulated datasets.

• Automation, DevOps & Infrastructure as Code
  • Implement Infrastructure as Code using Terraform and the Databricks Terraform provider.
  • Automate workspace and Unity Catalog object management using Databricks CLI and REST APIs.
  • Build and maintain CI/CD pipelines for Databricks assets and configurations.

• Cost Optimization & Observability
  • Apply resource tagging and cost-allocation strategies.
  • Configure auto-termination, cluster sizing, and workload optimization practices.
  • Provide cost visibility and recommendations to engineering and business teams.

• Documentation & Knowledge Transfer
  • Create and maintain platform runbooks, operational SLAs, and troubleshooting playbooks.
  • Deliver onboarding sessions and best-practice guidance to data engineering and ML teams.
  • Act as the escalation point for complex platform-related incidents.

Required Qualifications & Skills

Experience

• 5+ years of experience administering cloud-based big data platforms.
• 2+ years of hands-on experience with Azure Databricks.
• Proven experience supporting large-scale Spark ETL and ML workloads.

Technical Skills

• Deep knowledge of Apache Spark 3.x and Delta Lake.
• Hands-on experience with Databricks REST APIs and platform administration.
• Strong knowledge of Azure services including ADLS Gen2, Azure Key Vault, VNets, and networking concepts.
• Proficiency in Python and scripting using PowerShell or Bash.
• Strong experience with Terraform and Infrastructure as Code practices.

Governance & Compliance

• Hands-on experience with Unity Catalog privilege model, audit tables, and lineage.
• Understanding of enterprise data governance and compliance requirements.

Soft Skills

• Strong incident management and troubleshooting skills.
• Excellent documentation and stakeholder communication abilities.
• Ability to work collaboratively with distributed engineering and ML teams.

Preferred Certifications

• Databricks Platform Administrator Accreditation
• Databricks Certified Data Engineer Professional
• Databricks Lakehouse Fundamentals Accreditation
• Microsoft Azure Administrator Associate (AZ-104)
• (Preferred) Microsoft Azure Data Engineer Associate (DP 203) or successor certification

Nice to Have

• Experience with Immuta, BigID, Collibra, or similar data privacy and governance platforms.

EY | Building a better working world 

 
EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.  

 
Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.  

 
Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.