Design and implement a new backfill model for reporting data capable of handling hundreds of millions of row additions and updates efficiently.
Revamp the end-to-end experience for product teams adding or changing API-backed datasets to improve ergonomics and clarity.
Enhance the Aqueduct Dependency Resolver system, focusing on:
Error management
Observability
Delegating issue resolution to product teams
Lead integration efforts with modern Data Platform tools like Falcon for real-time data and manage deprecation of older systems.
Improve data warehouse management practices, ensuring data freshness and reliability.
Collaborate with product teams to understand their reporting and data requirements.
Design and implement scalable solutions for data ingestion, processing, and storage.
Mentor engineers and define the group's technical direction and strategy.
Who You Are:
8+ years of experience writing high-quality production-level code.
Proven expertise in designing and implementing large-scale data processing systems.
Strong understanding of distributed systems and data pipeline architectures.
Proficiency in at least one modern programming language (e.g., Go, Java, Python, Scala).
Experience with big data technologies such as Hadoop, Flink, Spark, Kafka, Pinot, Trino, Iceberg.
Solid grasp of data modeling and database systems.
Excellent problem-solving skills and the ability to tackle complex challenges.
Strong communicator and team collaborator.
Experience mentoring engineers and leading technical initiatives.
Preferred Qualifications:
Experience with real-time data processing and streaming systems.
Familiarity with data warehouse technologies and best practices.
Background in migrating legacy systems to modern architectures.
Contributions to open-source projects or active involvement in technical communities.

Staff Software Engineer

Job Description

Required Skills