Software Engineer, PySpark, VP

Role Overview

This vice president–level role offers an opportunity for an experienced Software Engineer to design, develop, test, and implement high-quality software across the full system lifecycle. The role focuses on data engineering and full stack software development, with strong emphasis on PySpark, Python, and AWS-based platforms. You will work within a permanent feature team, collaborating with engineers, architects, and stakeholders to ensure resilient, scalable, and efficient software solutions that support business objectives.

Key Responsibilities

• Design, develop, test, and implement robust and resilient software solutions across the lifecycle
• Apply Agile methods to backlog-driven software development within a permanent feature team
• Develop expertise in the platform across business, applications, data, and infrastructure layers
• Collaborate with principal engineers, architects, and stakeholders to understand platform architecture and business requirements
• Produce long-lived software that is flexible to adapt to future needs
• Contribute to intentional architecture and emergent design through experimentation, prototyping, and innovative solutions
• Automate build, test, and deployment activities using executable patterns
• Lead the adoption of programming best practices with focus on scalability, automation, virtualisation, optimisation, availability, and performance
• Ensure compliance with information security policies and practices within the financial sector
• Communicate complex technical concepts clearly and effectively to stakeholders

Required Skills and Experience

• Strong full stack experience in software design and implementation
• Minimum of twelve years of experience in Python or PySpark, with AWS and Airflow expertise
• Hands-on experience writing advanced Spark SQL or ANSI SQL queries, including query optimisation and performance tuning
• Experience with continuous integration, DevOps practices, GitLab, and Artifactory
• Strong collaboration skills with the ability to deliver pragmatic solutions that meet business needs
• Proven experience leading programming best practices around scalability, automation, virtualisation, optimisation, availability, and performance
• Strong stakeholder management and communication skills