Sr. Data Engineer

• Design, implement, and optimize end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.
• Develop data pipelines to extract and transform data in near real-time using cloud-native technologies.
• Implement data validation and quality checks to ensure accuracy and consistency.
• Monitor system performance, troubleshoot issues, and implement optimizations to enhance reliability and efficiency .
• Collaborate with business users, analysts, and other stakeholders to understand data requirements and deliver tailored solutions.
• Document technical designs, workflows, and best practices to facilitate knowledge sharing and maintain system documentation.
• Provide technical guidance and support to team members and stakeholders as needed.

Desirable Competencies:

• 8+ years of work experience.
• Proficiency in writing complex SQL queries on MPP systems (Snowflake/Redshift).
• Experience in Databricks and Delta tables.
• Data engineering experience with Spark/Scala/Python.
• Experience in Microsoft Azure stack (Azure Storage Accounts, Data Factory, and Databricks).
• Experience in Azure DevOps and CI/CD pipelines.
• Working knowledge of Python.
• Comfortable participating in 2-week sprint development cycles.