Mid-Level Data Engineer

• Provide estimates for requirements, analyze and develop as per the requirement.
• Developing and maintaining data pipelines and ETL (Extract, Transform, Load) processes to extract data efficiently and reliably from various sources, transform it into a usable format, and load it into the appropriate data repositories.
• Creating and maintaining logical and physical data models that align with the organizations data architecture and business needs. This includes defining data schemas, tables, relationships, and indexing strategies for optimal data retrieval and analysis.
• Collaborating with cross-functional teams and stakeholders to ensure data security, privacy, and compliance with regulations.
• Collaborate with downstream application to understand their needs and build the data storage and optimize as per their need.
• Working closely with other stakeholders and Business to understand data requirements and translate them into technical solutions.
• Identify risks and provide relevant solutions to complex problems and on-time delivery
• Familiar with Agile methodologies and have prior experience working with Agile teams using Scrum/Kanban
• Proficiency in Software Development best practices - Secure coding standards, Unit testing, Code coverage, Quality gates.
• Ability to lead and deliver change in a very productive way.
• Lead Technical discussions with customers to find the best possible solutions.
• Working closely with the Project Manager, Solution Architect and managing client communication (as and when required)
• Proactively identify and implement opportunities to automate tasks and develop reusable frameworks.
• Collaborate with the other team members to ensure all services are reliable, maintainable, and well-integrated into existing platforms.
• Provide technical write-ups and drawings to promote the proposed solutions.

What should you bring along

Must Have

• Must have knowledge of Data analytics
• Must have knowledge of Building an application using AWS Services.
• Hand on Expertise of 10+ years in AWS services like S3, Lambda, Glue, Athena, RDS, Step functions, Amazon Q, SNS, SQS, API Gateway, Security, Access and Role permissions, Logging and monitoring Services.
• Good hand on knowledge on Python, Spark, Hive and Unix, AWS CLI
• Hand on experience in the Terraform, GIT, GIT Actions. CICD pipeline
• Excellent knowledge in Data modeling and Designing ETL pipeline.
• Must have strong knowledge in using different databases such as MySQL, Oracle and Writing complex queries.
• Strong experience working in a continuous integration and Deployment process
• Ability to learn quickly, be organized and detail oriented.

Nice to Have:

• AI Project implementation and AI methods.

Must have technical skill

• Pyspark , AWS, SQL

Good to have Technical skills

• Terraform, GIT, GIT Actions. CICD pipeline, AI