Develop and maintain high-quality data processing applications using Apache Spark and related technologies.
Design and implement data transformations with a focus on performance and scalability.
Collaborate with data engineers and analysts to integrate data sources and services.
Write and execute unit tests to ensure code quality and reliability.
Participate in code reviews and adhere to best coding practices.
Work within an Agile development framework to deliver features and improvements.
Troubleshoot and resolve issues in existing data processing applications.
Stay up-to-date with the latest industry trends and technologies.
Mentor junior developers and provide technical guidance.
Communicate effectively with team members and stakeholders to ensure project success.
Manage multiple tasks and meet deadlines in a fast-paced environment.
Utilize tools like JIRA, Confluence, and Git/Bitbucket for project management and version control.
Experience with testing frameworks like ScalaTest or JUnit.
Qualifications
Bachelor s degree in Computer Science, Software Engineering, or a closely related field.
Technical Requirement:
4+ years of experience with Apache Spark (preferably Spark 3.0 or above).
Proficiency in Scala or Java programming languages.
Experience with big data technologies like Hadoop, Kafka, or HDFS.
Experience with Orchestration tool like airflow.
Experience working with Managed cloud Spark infrastructure like AWS EMR and in general AWS tools like S3, SQS, VPC and API Gateway.
Familiarity with data processing frameworks and libraries.
Strong understanding of distributed computing and parallel processing.
Knowledge of Agile methodologies and good coding practices.
Ability to work both independently and as part of a team.
Strong communication and interpersonal skills.
Experience with version control systems like Bit Bucket/Git.
Experience with CI/CD tools like Jenkins, Docker, or Kubernetes.

Apache Spark Developer

Job Description

Required Skills