Databricks Pyspark

• Develop and optimize data processing jobs using PySpark to handle complex transformations and aggregations efficiently
• Design and implement robust data pipelines on the AWS platform (Databricks exposure is a plus)
• Utilize AWS services like EC2, S3 for scalable data processing and storage
• Manage SQL database schema design, query optimization, and performance tuning
• Design and maintain data warehouses following best practices in data modeling and ETL
• Use Databricks clusters and Spark jobs to ensure data quality and performance
• Implement security measures, backup, and disaster recovery using AWS best practices
• Use GitHub and CI/CD for source code management and deployment
• Orchestrate data workflows using Apache Airflow
• Collaborate with data scientists and business analysts for advanced analytics projects

Mandatory Skills

• Databricks
• PySpark