Lead Data Engineer

About the Role

Our client is looking for there AI & Data function is central to driving growth through the creation of new data monetisation revenue streams. As part of the Product Engineering team, the Lead Data Engineer will take technical ownership of the migration and modernisation of complex mobile network data processes onto a Databricks-based platform, supporting insight and analytics delivery via Tableau.

This is a senior, hands-on leadership role where you will shape data engineering standards, guide technical direction, and ensure the delivery of high-quality, resilient, and scalable data pipelines across the organisation's core data platforms.

Key Responsibilities

• Technical Leadership & Ownership: Lead the design, delivery, and optimisation of Databricks-based data solutions, acting as the technical authority for complex data engineering initiatives.

• Data Process Migration: Own the planning and execution of migrations from legacy data processes to Databricks, ensuring continuity of data supply and improvements in quality, efficiency, and resilience.

• Pipeline Development & Optimisation: Rapidly understand highly complex data flows and re-engineer them into performant, scalable Spark-based pipelines.

• Standards & Continuous Improvement: Define and drive best practices in data engineering, CI/CD, testing, and operational excellence across the team.

• Customer-Centric Delivery: Ensure new data outputs align with existing products and stakeholder expectations, maintaining consistency and trust in analytics and monetisation outputs.

• Team Leadership & Collaboration: Provide technical leadership to other engineers, including task delegation, code reviews, and mentoring, while influencing ways of working across the wider engineering function.

Skills & Experience Required

• Extensive experience developing and optimising data pipelines using Python, SQL, or equivalent languages.

• Deep, hands-on expertise with Databricks and Apache Spark in large-scale, production environments.

• Strong understanding of databases and advanced analytical query optimisation.

• Proven experience leading complex data migration and modernisation programmes.

• Demonstrable experience designing and operating CI/CD pipelines, workflow automation, and Infrastructure as Code.

• Strong experience with Microsoft Azure cloud services.

• Hands-on experience with dbt and modern analytics engineering practices.

• Experience delivering and maintaining Tableau Cloud solutions.

• Comfortable working with version control, collaborative development, and cross-functional teams.

• Desirable: experience with geospatial data and spatial analysis techniques.

Why Join

You'll play a key role in shaping the future of data engineering within a telecommunications environment, working with large-scale network data and modern cloud platforms. This role offers the opportunity to influence technical strategy, mentor engineers, and deliver data products that directly support commercial growth and data monetisation initiatives.