Key Responsibilities:

Data Modeling and ETL: Design, build, optimize, and maintain data models and ETL processes to support business requirements.
Data Pipeline Development: Develop end-to-end data pipelines and workflows from source to target systems.
Data Infrastructure Management: Build, deploy, and manage scalable data infrastructure to handle large volumes of data efficiently.
Data Access and Security: Coordinate data access and security to ensure analysts and data scientists can easily access required data.
Programming and Data Transformation: Develop Python and PySpark programs for data analysis, custom frameworks for rules generation, and business transformation logic using Spark DataFrames/RDDs.
Integration with Big Data Ecosystem: Work with HBase, Hive, and other big data tools to design and implement efficient data solutions.
DevOps and Deployment: Implement DevOps best practices for data pipelines and related infrastructure deployment.

Required Education:

Bachelor's Degree in Computer Science, Information Technology, or a related field

Preferred Education:

Required Technical and Professional Expertise:

5+ years of experience in Big Data technologies including Hadoop, Spark, Scala, and Python
Experience with HBase and Hive
Developed Python and PySpark programs for data ingestion, analysis, and transformation
Worked on building custom Python frameworks for rules generation
Experience in reading/writing data from HBase using PySpark and applying business transformations using Spark DataFrames/RDDs
Familiarity with AWS tools such as S3, Athena, DynamoDB, Lambda, Jenkins, and Git

Preferred Technical and Professional Expertise:

Understanding of DevOps principles and practices
Experience in building scalable, end-to-end data ingestion and processing solutions
Strong knowledge of object-oriented and functional programming with Python, Java, or Scala

Data Engineer-Data Platforms-AWS