Azure Specialist-CDM Smith

Responsibilities:

• Databricks Platform Expertise: Act as a subject matter expert for the Databricks platform within the Digital Capital team, providing technical guidance, best practices, and innovative solutions.
• Databricks Workflows and Orchestration: Design and implement complex data pipelines using Azure Data Factory or Qlik Replicate.
• End-to-End Data Pipeline Development: Design, develop, and implement highly scalable and efficient ETL/ELT processes using Databricks notebooks (Python/Spark or SQL) and other Databricks-native tools.
• Delta Lake Expertise: Utilize Delta Lake for building reliable data lake architecture, implementing ACID transactions, schema enforcement, time travel, and optimizing data storage for performance.
• Spark Optimization: Optimize Spark jobs and queries for performance and cost efficiency within the Databricks environment. Demonstrate a deep understanding of Spark architecture, partitioning, caching, and shuffle operations.
• Data Governance and Security: Implement and enforce data governance policies, access controls, and security measures within the Databricks environment using Unity Catalog and other Databricks security features.
• Collaborative Development: Work closely with data scientists, data analysts, and business stakeholders to understand data requirements and translate them into Databricks-based data solutions.
• Monitoring and Troubleshooting: Establish and maintain monitoring, alerting, and logging for Databricks jobs and clusters, proactively identifying and resolving data pipeline issues.
• Code Quality and Best Practices: Champion best practices for Databricks development, including version control (Git), code reviews, testing frameworks, and documentation.
• Performance Tuning: Continuously identify and implement performance improvements for existing Databricks data pipelines and data models.
• Cloud Integration: Integrate Databricks with other cloud services (e.g., Azure Data Lake Storage Gen2, Azure Synapse Analytics, Azure Key Vault) for a seamless data ecosystem.
• Traditional Data Warehousing & SQL: Design, develop, and maintain schemas and ETL processes for traditional enterprise data warehouses. Demonstrate expert-level proficiency in SQL for complex data manipulation, querying, and optimization within relational database systems.

Required Skills:

• Proficiency in Databricks and Databricks Workflows and Orchestration.
• Hands-on experience in Python for automation and scripting.
• Strong knowledge of Azure Data Lakes, Data Warehouses, and cloud architecture.
• Proficiency in designing web applications and data engineering solutions (Solution Architecture).
• Familiarity with DevOps Basics, including Jenkins and CI/CD pipelines.
• Excellent verbal and written communication skills.
• Ability to quickly grasp new technologies and adapt to changing requirements (Fast Learner).
• Experience integrating Databricks with other cloud services (e.g., Azure Data Lake Storage Gen2, Azure Synapse Analytics, Azure Key Vault) for a seamless data ecosystem.
• Extensive experience with Spark (PySpark, Spark SQL) for large-scale data processing.