Sr Data Engineer, Azure Databricks, Python, PySpark

Job Responsibilities:

• Design, develop, and implement robust and scalable data pipelines using Azure Data Factory (ADF).
• Efficiently extract, transform, and load (ETL) data from various disparate sources into Azure Data Lake Storage (ADLS).
• Utilize Azure Databricks for advanced data processing, transformation, and analytics.
• Develop complex data solutions and scripts primarily using Python and PySpark.
• Collaborate with data scientists, analysts, and other engineering teams to understand data requirements and deliver high-quality data solutions.
• Monitor, optimize, and troubleshoot data pipelines to ensure performance, reliability, and data integrity.
• Implement best practices for data governance, security, and quality within the Azure ecosystem.

Required Skills:

• Proficiency in designing, developing, and implementing data pipelines using Azure Data Factory.
• Strong experience in extracting, transforming, and loading data from various sources into Azure Data Lake Storage (ADLS).

Good to have skills :

• Hands-on experience with Azure Databricks.
• Strong programming skills in Python.
• Expertise in PySpark for big data processing.