Azure Data Engineer/Architect

Key Responsibilities

• Design, develop, and maintain CI/CD pipelines for Azure Databricks and Azure Data Factory (ADF) workloads
• Build and manage ETL pipelines, triggers, and data workflows using Azure Data Factory
• Develop and deploy Databricks jobs using PySpark, Python, and SQL notebooks
• Implement DevOps processes using Jenkins, Bitbucket, Git, and Azure DevOps
• Automate deployment and integration of data pipelines across environments
• Monitor, troubleshoot, and optimize pipeline performance and reliability
• Work with Azure storage services and ensure efficient data movement and processing
• Implement security practices including Azure Key Vault, secrets management, and access control
• Collaborate with data engineers, architects, and DevOps teams for seamless delivery
• Ensure compliance with Azure governance, security, and enterprise standards
• Mentor and guide team members on DevOps and CI/CD best practices