Job Description
Inviting applications for the role of Lead Consultant - Data Engineer.
In this role, a Data Engineer will leverage cloud technologies to manage and analyze their data. This role demands expertise in Databricks, Azure Data Factory (ADF), Python, PySpark and Unity Catalog to efficiently process and analyze large datasets. Data Engineer is responsible for designing and implementing scalable data pipelines, optimizing data workflows, ensuring data quality, collaborating with cross-functional teams, and leveraging cloud technologies to enhance data processing and analytics capabilities.
Responsibilities
.Data Pipeline Design and Development:
oArchitect, build, and optimize data ingestion and transformation pipelines using Azure ADF and Databricks.
oImplement data integration and transformation solutions using Azure Databricks.
oDevelop and deploy data models and solutions using Azure services.
.Data Management and Optimization:
oPull, ingest, transform, stitch, and wrangle data from various sources for advanced analytics.
oDesign, implement, and deploy data loaders to load data into the engineering sandbox.
oMonitor and optimize data pipelines for performance and reliability.
.Collaboration and Support:
oCollaborate with cross-functional teams to gather requirements and understand data needs.
oProvide input to machine learning (ML) engineers and cloud engineers for designing and implementing data management or architecture solutions.
oAssist ML engineers in pulling, filtering, tagging, joining, parsing, and normalizing datasets.
.Data Quality and Security:
oImplement data quality checks, validation rules, and governance policies to ensure accuracy, reliability, and security of data assets.
oTroubleshoot and resolve data-related issues promptly.
oImplement data security and privacy measures to protect sensitive information.
.Implement Unity Catalog:
oManage data governance and security using Unity Catalog to ensure compliance and protect sensitive information.
oETL development:Develop scalable data ingestion or ETL from Workday and HR systems.
oLeverage data best practices and tools and assist ML engineer in pulling, filtering, tagging, joining, parsing, and normalizing data sets for use.
Qualifications We Seek in You!
Minimum Qualifications / Skills
.A bachelor's degree in computer science, Information Technology, Business, or a related field is required
.Experience in Databricks, Azure ADF, Python, Pyspark and Unity Catalog
.Experience in working on data ingestion or ETL from Workday or any other HR systems
.Experience in RBAC security models in Unity Catalog
.Exposure to CI/CD DevOps practices
.Experience with PySpark and SQL for building scalable data transformation jobs, data querying, analysis, and data modelling
.Expertise in Azure Databricks, including its features for big data processing and collaborative notebooks.
.Strong programming skills in Python for data manipulation and scripting.
.Proficiency in data engineering tools like Databricks, Apache Spark, and Unity Catalogue
.Hands-on experience with Azure Data Factory for pipeline orchestration, scheduling, and monitoring
Preferred Qualifications / Skills
.Certifications in Azure data engineering or related fields.