Manager Data Engineer - AWS Databricks

Key Responsibilities:

• Lead the design, development, and optimization of scalable and secure data pipelines using AWS services such as Glue, S3, Lambda, EMR , and Databricks Notebooks , Jobs, and Workflows.
• Oversee the development and maintenance of data lakes on AWS Databricks , ensuring performance and scalability.
• Build and manage robust ETL/ELT workflows using Python and SQL , handling both structured and semi-structured data.
• Implement distributed data processing solutions using Apache Spark / PySpark for large-scale data transformation.
• Collaborate with cross-functional teams including data scientists, analysts, and product managers to ensure data is accurate, accessible, and well-structured.
• Enforce best practices for data quality, governance, security , and compliance across the entire data ecosystem.
• Monitor system performance, troubleshoot issues, and drive continuous improvements in data infrastructure.
• Conduct code reviews, define coding standards, and promote engineering excellence across the team.
• Mentor and guide junior data engineers, fostering a culture of technical growth and innovation.

QualificationsRequirements

• 8+ years of experience in data engineering with proven leadership in managing data projects and teams.
• Expertise in Python, SQL, Spark (PySpark) , and experience with AWS and Databricks in production environments.
• Strong understanding of modern data architecture, distributed systems, and cloud-native solutions.
• Excellent problem-solving, communication, and collaboration skills.
• Prior experience mentoring team members and contributing to strategic technical decisions is highly desirable.