Senior/Lead Data engineer

• Expertise in Python Language is MUST.
• SQL (should be able to write complex SQL Queries) is MUST
• Hands on experience in Apache Flink Streaming Or Spark Streaming MUST
• Hands On expertise in Apache Kafka experience is MUST
• Data Lake Development experience.
• Orchestration (Apache Airflow is preferred).
• Spark and Hive: Optimization of Spark/PySpark and Hive apps
• Trino/(AWS Athena) (Good to have)
• Snowflake (good to have).
• Data Quality (good to have).
• File Storage (S3 is good to have)

Responsibilities:

• Bachelor's degree in Computer Science, Engineering, or a related field
• 5 + years experience in data engineering, ELT development, and data modeling.
• Proficiency in using Apache Airflow and Spark for data transformation, data integration, and data management.
• Experience implementing workflow orchestration using tools like Apache Airflow, SSIS or similar platforms.
• Demonstrated experience in developing custom connectors for data ingestion from various sources.
• Strong understanding of SQL and database concepts, with the ability to write efficient queries and optimize performance.
• Experience implementing Data Ops principles and practices, including data CI/CD pipelines.
• Excellent problem-solving and troubleshooting skills, with a strong attention to detail.
• Knowledge of data streaming and real-time data processing technologies (e.g., Apache Kafka).
• Strong understanding of software development principles and practices, including version control (e.g., Git) and code review processes.
• Experience with Agile development methodologies and working in cross-functional Agile teams.
• Ability to adapt quickly to changing priorities and work effectively in a fast-paced environment.
• Excellent analytical and problem-solving skills, with a keen attention to detail.
• Strong written and verbal communication skills, with the ability to effectively communicate complex technical concepts to both technical and non-technical stakeholders.