Data Engineer

This role is for a Data Engineer to design and implement end-to-end data engineering solutions on Azure. The ideal candidate will be responsible for developing and optimizing data pipelines, ensuring data quality and security, and collaborating with cross-functional teams to build scalable data architectures.

Responsibilities

• Design and implement end-to-end data engineering solutions on Azure, utilizing services like Azure Data Factory (ADF), Azure Databricks, and Synapse Analytics.
• Develop and maintain data pipelines (ETL) that extract, transform, and load data from various sources into Azure data repositories, including data lakes.
• Optimize and fine-tune data pipelines for performance, scalability, and cost-efficiency.
• Automate tasks and deploy production-standard code with unit testing, continuous integration, and versioning.
• Load transformed data into various destinations, including data warehouses, high-speed indexes, and real-time reporting systems.
• Ensure data quality and accuracy through testing and validation.
• Implement data security measures, including encryption, access controls, and auditing.
• Provide guidance and support for data governance, including metadata management and data lineage.
• Collaborate with cross-functional teams to understand data requirements and translate them into scalable data architectures.
• Troubleshoot and resolve data pipeline issues to ensure minimal downtime.
• Utilize SQL for data transformation and perform data modeling when necessary.
• Create and maintain comprehensive documentation of data processes and architectures.

Skills

Required Skills:

• Expertise in ETL (Extract, Transform, Load) processes.
• Strong skills in Data Analysis and Business Analysis.
• Hands-on experience with Azure Data Factory (ADF) and Azure Databricks.
• Proficiency in SQL for data transformation and modeling.
• Experience with data security measures and data governance.
• Ability to automate tasks and deploy production-standard code.
• Strong communication and collaboration skills.