Azure Databricks Platform Engineer

Azure Databricks Platform Engineer / Architect

Location: London - 3 days onsite / 2 days remote per week

We're looking for a hands-on Azure Databricks Platform Engineer / Architect to join a major enterprise Data Platform transformation. This is a delivery-focused role combining architecture, engineering, platform optimisation and production support - ideal for someone who enjoys getting into the detail rather than operating purely at governance level.

You'll play a key role in optimising the Databricks platform, controlling costs through FinOps, and enhancing the Discovery Zone to support the migration of analytics and data science workloads from POSIT/RStudio to Databricks.

What You'll Be Doing

Databricks Platform & Serverless

• Design, configure and optimise Azure Databricks environments across Serverless, Jobs Compute, Interactive Compute and Classic Clusters.

• Assess workloads and define the most appropriate compute model based on performance, SLA, utilisation and cost.

• Implement compute policies, autoscaling, quotas, budgets and operational guardrails.

• Troubleshoot production issues and continuously improve platform performance and reliability.

• Optimise Databricks SQL, Delta Lake and workload performance.

FinOps & Governance

• Establish practical FinOps processes covering ownership, cost centres, projects, applications and teams.

• Implement tagging standards and automated compliance checks through CI/CD.

• Deliver granular cost attribution, reporting, budgets, alerts and spend controls.

• Analyse billing and usage data to identify cost anomalies, inefficient workloads, storage overhead and unnecessary data movement.

Discovery Zone & POSIT/RStudio Migration

• Enhance the Databricks Discovery Zone to support analytics and data science migration.

• Enable secure APIs, external data ingestion, application deployment, BI/reporting, scheduling, orchestration and local IDE development.

• Develop reusable onboarding and migration patterns to improve security, supportability and delivery speed.

• Work with data scientists, engineers, architects and business stakeholders to drive adoption.

Data Engineering & Integration

• Build scalable ingestion and transformation solutions using Python, PySpark, SQL and Delta Lake.

• Deliver batch, incremental and CDC processing, schema evolution, reconciliation, data quality and error handling.

• Integrate REST APIs, SaaS platforms, databases, files/object storage and enterprise systems.

• Implement secure authentication, secrets management and governed end-to-end data flows.

What We're Looking For

• Strong hands-on Azure Databricks implementation, administration and troubleshooting experience.

• Deep knowledge of Databricks Serverless, compute optimisation, Jobs, Workflows and Databricks SQL.

• Strong Azure networking, identity, security, monitoring, secrets management and private connectivity experience.

• Excellent Python, PySpark and SQL development skills.

• Enterprise data engineering and production platform experience.

• Strong understanding of Delta Lake, CDC, orchestration, data quality and operational monitoring.

• Proven experience implementing FinOps, tagging, cost attribution, budgets and cost optimisation.

• Ability to operate across architecture, engineering, implementation and production support.

Highly Desirable

• Experience migrating POSIT/RStudio workloads to Databricks.

• Experience converting R-based analytics/data science workloads and libraries.

• AI/ML, LLM, RAG, vector retrieval, model serving or ML lifecycle experience.

• Large-scale data platform transformation experience.

• Experience with chargeback/showback models and enterprise FinOps.

• Experience working within complex or highly regulated organisations.

This is an excellent opportunity for a Databricks specialist who wants genuine ownership of platform engineering, optimisation and transformation - with the scope to influence both technology and operating models.

If you are interested in this opportunity, please apply now with your updated CV in Microsoft Word/PDF format.

Disclaimer
Notwithstanding any guidelines given to level of experience sought, we will consider candidates from outside this range if they can demonstrate the necessary competencies.
Square One is acting as both an employment agency and an employment business, and is an equal opportunities recruitment business. Square One embraces diversity and will treat everyone equally. Please see our website for our full diversity statement.