Pyspark Developer

Key Responsibilities:

• Develop, optimize, and maintain Big Data solutions using PySpark, Spark, Scala, and Hadoop.
• Implement and manage AWS services including EMR, S3, IAM, Lambda, SNS, and SQS.
• Write efficient SQL queries, including subqueries, joins, and CTEs for data processing.
• Perform Spark UI monitoring, optimization, and debugging to improve performance.
• Develop Python scripts for automation, data processing, and integration with Big Data workflows.
• Collaborate with cross-functional teams to design and implement scalable data pipelines.
• Ensure data quality, reliability, and security in large-scale data processing environments.