Data Engineer/Data Analyst

Data Pipeline Architecture and Development:

• Design robust, scalable, and efficient data pipelines to gather data from diverse sources (databases, APIs, flat files, etc.)
• Implement ETL/ELT procedures, ensuring data is extracted, transformed, and loaded into appropriate storage systems for analysis
• Choose and use optimal data warehousing/lake solutions (e.g., Snowflake, Redshift, BigQuery) depending on needs

Data Quality and Integrity:

• Thoroughly check data for inconsistencies, errors, and missing values, establishing cleaning processes
• Implement data validation rules and run regular checks to maintain data quality standards for the data science team
• Monitor pipelines for failures, promptly addressing data anomalies or breaks

Data Modeling and Optimization:

• Collaborate with data scientists to design data models according to their analysis requirements
• Transform raw data into usable, analysis-ready formats
• Optimize data structures and databases to improve query performance and efficiency for the data scientists' workloads

Feature Engineering:

• Work closely with data scientists to understand model requirements and engineer new features from the existing data