AWS Glue | Pyspark Developer

Warm welcome from SP Staffing Services! We're excited to tell you about a permanent opportunity for a PySpark Developer to join our team.

• Experience: We're seeking professionals with 4 to 6 years of experience.
• Location: This role is open in Chennai, Hyderabad, Bangalore.

What You'll Do

• Implement data ingestion pipelines from various sources, including databases, S3, and files.
• Build ETL/Data Warehouse transformation processes.
• Develop Big Data and non-Big Data cloud-based enterprise solutions using PySpark and SparkSQL, along with related frameworks and libraries.
• Create scalable, reusable, and self-service frameworks for data ingestion and processing.
• Integrate end-to-end data pipelines to move data from source to target repositories, ensuring data quality and consistency.
• Conduct processing performance analysis and optimization.
• Implement best practices in design & analysis, automation (Pipelining, IaC), testing, monitoring, and documentation.
• Work with both structured and unstructured data.

Good to Have

• Experience with cloud-based solutions.
• Knowledge of data management principles.