EMR _ Spark SME

Technical Skills:

• 5+ years of experience in big data technologies with hands-on expertise in AWS EMR and Apache Spark.
• Proficiency in Spark Core, Spark SQL, and Spark Streaming for large-scale data processing.
• Strong experience with data formats (Parquet, Avro, JSON) and data storage solutions (Amazon S3, HDFS).
• Solid understanding of distributed systems architecture and cluster resource management (YARN).
• Familiarity with AWS services (S3, IAM, Lambda, Glue, Redshift, Athena).
• Experience in scripting and programming languages such as Python, Scala, and Java.
• Knowledge of containerization and orchestration (Docker, Kubernetes) is a plus.
• Architect and develop scalable data processing solutions using AWS EMR and Apache Spark.
• Optimize and tune Spark jobs for performance and cost efficiency on EMR clusters.
• Monitor, troubleshoot, and resolve issues related to EMR and Spark workloads.
• Implement best practices for cluster management, data partitioning, and job execution.
• Collaborate with data engineering and analytics teams to integrate Spark solutions with broader data ecosystems (S3, RDS, Redshift, Glue, etc.).
• Automate deployments and cluster management using infrastructure-as-code tools like CloudFormation, Terraform, and CI/CD pipelines.
• Ensure data security and governance in EMR and Spark environments in compliance with company policies.
• Provide technical leadership and mentorship to junior engineers and data analysts.
• Stay current with new AWS EMR features and Spark versions to recommend improvements and upgrades.
• Requirements and Skills
• Performance tuning and optimization of Spark jobs.
• Problem-solving skills with the ability to diagnose and resolve complex technical issues.
• Strong experience with version control systems (Git) and CI/CD pipelines.
• Excellent communication skills to explain technical concepts to both technical and non-technical audiences.
• Qualification:
• Education qualification: B.Tech, BE, BCA, MCA, M. Tech or equivalent technical degree from a reputed college.
• Certifications:
• AWS Certified Solutions Architect Associate/Professional
• AWS Certified Data Analytics Specialty