Key Responsibilities:

PySpark Development:

Design, implement, and optimize PySpark solutions for large-scale data processing and analysis.
Develop data pipelines using Spark to handle data transformations, aggregations, and other complex operations efficiently.
Write and optimize Spark SQL queries for big data analytics and reporting.
Handle data extraction, transformation, and loading (ETL) processes from various sources into a unified data warehouse or data lake.

Data Pipeline Design & Optimization:

Build and maintain ETL pipelines using PySpark, ensuring high scalability and performance.
Implement batch and streaming processing to handle both real-time and historical data.
Optimize the performance of PySpark applications by applying best practices and techniques such as partitioning, caching, and broadcast joins.

Data Storage & Management:

Work with large datasets and integrate them into storage solutions such as HDFS, S3, Azure Blob Storage, or Google Cloud Storage.
Ensure efficient data storage, access, and retrieval through Spark and other tools (e.g., Parquet, ORC).
Maintain data quality, consistency, and integrity throughout the pipeline lifecycle.

Cloud Platforms & Big Data Frameworks:

Deploy Spark-based applications on cloud platforms such as AWS (Amazon EMR), Azure HDInsight, or Google Dataproc.
Work with cloud-native services such as AWS Lambda, S3, Google Cloud Storage, and Azure Data Lake to handle and process big data.
Leverage cloud data processing tools and frameworks to scale and optimize the PySpark jobs.

Collaboration & Integration:

Collaborate with cross-functional teams (data scientists, analysts, product managers) to understand business requirements and develop appropriate data solutions.
Integrate data from multiple sources and platforms (e.g., databases, external APIs, flat files) into a unified system.
Provide support for downstream applications and data consumers by ensuring timely and accurate delivery of data.

Performance Tuning & Troubleshooting:

Identify bottlenecks and optimize Spark jobs to improve performance.
Conduct performance tuning of both the cluster and individual Spark jobs, leveraging Spark's in-built tools for monitoring.
Troubleshoot and resolve issues related to data processing, application failures, and cluster resource utilization.

Documentation & Reporting:

Maintain clear and comprehensive documentation of data pipelines, architectures, and processes.
Create technical documentation to guide future enhancements and troubleshooting.
Provide regular updates on the status of ongoing projects and data processing tasks.

Continuous Improvement:

Stay up to date with the latest trends, technologies, and best practices in big data processing and PySpark.
Contribute to improving development processes, testing strategies, and code quality.
Share knowledge and provide mentoring to junior team members on PySpark best practices.

Required Qualifications:

2-4 years of professional experience working with PySpark and big data technologies.
Strong expertise in Python programming with a focus on data processing and manipulation.
Hands-on experience with Apache Spark, particularly with PySpark for distributed computing.
Proficiency in Spark SQL for data querying and transformation.
Familiarity with cloud platforms like AWS, Azure, or Google Cloud, and experience with cloud-native big data tools.
Knowledge of ETL processes and tools.
Experience with data storage technologies like HDFS, S3, or Google Cloud Storage.
Knowledge of data formats such as Parquet, ORC, Avro, or JSON.
Experience with distributed computing and cluster management.
Familiarity with Linux/Unix and command-line operations.
Strong problem-solving skills and ability to troubleshoot data processing issues.

PySpark