Senior Data Engineer - AWS Databricks

Responsibilities:

• Design, develop, and optimize scalable data pipelines using Databricks in AWS such as Glue, S3, Lambda, EMR, Databricks notebooks, workflows and jobs.
• Building data lake in WS Databricks.
• Build and maintain robust ETL/ELT workflows using Python and SQL to handle structured and semi-structured data.
• Develop distributed data processing solutions using Apache Spark or PySpark .
• Partner with data scientists and analysts to provide high-quality, accessible, and well-structured data.
• Ensure data quality, governance, security, and compliance across pipelines and data stores.
• Monitor, troubleshoot, and improve the performance of data systems and pipelines.
• Participate in code reviews and help establish engineering best practices.
• Mentor junior data engineers and support their technical development.

QualificationsRequirements

• Bachelors or masters degree in computer science, Engineering, or a related field.
• 5+ years of hands-on experience in data engineering , with at least 2 years working with AWS Databricks .
• Strong programming skills in Python for data processing and automation.
• Advanced proficiency in SQL for querying and transforming large datasets.
• Deep experience with Apache Spark/PySpark in a distributed computing environment.
• Solid understanding of data modelling, warehousing, and performance optimization techniques.
• Proficiency with AWS services such as Glue , S3 , Lambda and EMR .
• Experience with version control Git or Code commit