Azure Databricks

Key Responsibilities:

• Data Pipeline Development: Design, build, and maintain scalable ETL (Extract, Transform, Load) data pipelines using Azure Databricks, Apache Spark, and Python.
• Spark Optimization: Develop and optimize Spark jobs for large-scale data processing on Databricks. Ensure that the jobs run efficiently, leveraging the capabilities of distributed computing for optimal performance.
• Data Integration: Integrate data from various sources, including structured and unstructured data, into the Azure cloud environment using Databricks and related tools.
• Collaboration with Data Scientists & Analysts: Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver robust data solutions that enable advanced analytics, machine learning, and reporting.
• Azure Integration: Work closely with Azure services such as Azure Data Lake, Azure SQL Database, Azure Blob Storage, Azure Synapse Analytics, and Azure Data Factory for comprehensive data processing solutions.
• Data Transformation: Use Spark SQL, PySpark, and Databricks notebooks to perform data transformations and enable the conversion of raw data into actionable insights.
• Automation & Scheduling: Implement automated job scheduling and orchestration for regular data processing tasks, ensuring data is consistently processed and available for downstream consumption.
• Performance Tuning & Troubleshooting: Optimize the performance of data workflows and Spark applications on Databricks. Troubleshoot and resolve data-related issues and bottlenecks.
• Cloud Security: Ensure that data security and compliance standards are followed for cloud-based solutions, including managing data access, encryption, and auditing within the Azure Databricks environment.
• Monitoring & Logging: Implement logging and monitoring practices for the Azure Databricks environment to track job performance, failures, and troubleshooting efforts.
• Documentation & Best Practices: Maintain proper documentation for data pipelines, processes, and technical workflows. Follow best practices for coding, version control, and deployment.
• Stay Updated with Technology Trends: Keep up to date with the latest developments in Azure Databricks, Apache Spark, and related technologies. Apply new techniques to improve performance and scalability.

Required Qualifications & Skills:

• 3-5 years of hands-on experience in data engineering and working with Azure Databricks.
• Strong proficiency in Apache Spark, particularly in Databricks for building large-scale data pipelines and distributed data processing applications.
• Solid experience with Azure cloud services, including Azure Data Lake, Azure SQL Database, Azure Blob Storage, Azure Synapse, and Azure Data Factory.
• Proficiency in Python, Scala, or SQL for data engineering tasks, with a focus on PySpark for data processing.
• Experience working with structured and unstructured data from a variety of sources, including relational databases, APIs, and flat files.
• Familiarity with Databricks notebooks for developing, testing, and sharing data workflows, and using them for collaboration.
• In-depth understanding of ETL processes, data pipelines, and data transformation techniques.
• Hands-on experience with cloud-based data storage solutions (e.g., Azure Data Lake, Blob Storage) and data warehousing concepts.
• Knowledge of data security best practices in a cloud environment (e.g., data encryption, access controls, Azure Active Directory).
• Experience with CI/CD pipelines and version control systems like Git.
• Familiarity with containerization and deployment practices using Docker and Kubernetes is a plus.
• Strong debugging, performance tuning, and problem-solving skills.
• Excellent written and verbal communication skills, with the ability to collaborate effectively across teams.
• Bachelor's degree in Computer Science, Information Technology, or a related field.