Data Engineer - Architect

Job Description: The Senior Data & AI Architect designs, develops, and implements complex data architectures, machine learning (ML) models, and AI-driven solutions using Azure (preferred), AWS, and GCP. This role requires hands-on expertise in data engineering, ML model development, big data frameworks, and cloud infrastructure. The architect will manage large-scale data processing, optimize ML pipelines, and deploy AI solutions while ensuring security, scalability, and performance.

Key Responsibilities:

Data & AI Architecture Design:

• Design scalable, reliable data pipelines using Azure Data Factory, Synapse, and Data Lake.
• Architect ML models with cloud integration for business use cases using Azure ML.
• Design data lakes/warehouses for batch and real-time processing.
• Integrate AI models with real-time data systems (Azure Event Hub, Stream Analytics).

Cloud Architecture:

• Build AI solutions using Azure AI services (Cognitive Services, ML, etc.).
• Implement strong data governance with Azure Data Catalog, Policy, and Security Center.
• Ensure high availability, fault tolerance, and disaster recovery using Azure services.

Performance Optimization:

• Optimize distributed data processing with Apache Spark on Azure Databricks.
• Improve query performance in Synapse with partitioning, indexing, and caching.
• Optimize ML model efficiency and inference times using Azure ML AutoML.

ML Lifecycle Management:

• Implement CI/CD pipelines for ML models using Azure DevOps.
• Apply MLOps for model monitoring, retraining, and lifecycle management.
• Ensure scalable model serving with Azure Kubernetes Service (AKS).

Real-time Data & Integration:

• Deploy real-time data pipelines with Azure Event Hub, Stream Analytics, and Kafka.
• Design data integration strategies using Azure Data Factory and APIs.

AI Solution Delivery:

• Lead AI product development (recommendation engines, predictive analytics).
• Drive end-to-end AI solution delivery, integrating models into business workflows.

Candidate Profile:

• 10+ years of experience in data architecture, data engineering, AI/ML, and cloud computing.
• Expertise in Azure (Data Factory, Synapse, ML, AKS); AWS/GCP experience is a plus.
• Strong knowledge of big data frameworks (Apache Spark, Databricks), Python, SQL, Terraform, and Kubernetes.

Key Attributes:

• Excellent problem-solving skills with a focus on optimizing performance, cost, and scalability.
• Experience designing secure, cloud-native AI systems.
• Ability to manage complex challenges independently in a fast-paced environment.