Lead Data Engineer
@A Leading BioTech VC
Pay Range
$240,000 - $260,000
About the Company
Posthaste Labs is a boutique consulting firm that provides development and data services to VCs and startups. Our focus is on building the highest quality products that will enable our customers to quickly gain traction in the market while ensuring that their code and data ecosystems are robust and able to scale.
About the Role
As a Lead Data Engineer with Posthaste Labs, you will be responsible for architecting and leading the development of both the business logic and technical implementation for the data pipelines that are used by the client to run their operations. In this leadership role, you will mentor team members, establish engineering best practices, and drive strategic data architecture decisions. The pipelines will be developed using dbt and will need to merge data across several distinct data sources using information that changes across time. This position will be embedded directly with the client in a leading BioTech firm while having full autonomy to determine the tools, technologies, processes, and team structure. You will work directly with executive stakeholders, shaping strategic requirements, defining roadmaps, and leading the execution of complex deliverables. You will lead a team of 6 engineers within the overall development of solving end-to-end use-cases. This is an hourly contract position to start. The candidate should be able to consistently provide 30+ hours a week but can set their own schedule and may work up to 50 hours if desired. Candidates must be based in the US but can set their hours for when and where they want to work.
Key Responsibilities
- • Technical leadership: Architect and drive the overall data engineering strategy, establishing patterns, best practices, and standards for the team to follow.
- • Team mentorship: Mentor and guide team members in technical decisions, code reviews, and professional development. Foster a culture of engineering excellence.
- • Client engagement: Proactively identify gaps, issues and strategic data needs while sharing expert opinions with client leadership. Lead high-stakes interactions and initiatives with clear, executive-level communication.
- • Entity resolution: Design and oversee the development of sophisticated matching algorithms that will reconcile data about entities from disparate sources, rank matches and intelligently correlate records together using advanced ML techniques.
- • Data quality transparency: Architect and implement comprehensive monitoring and validation frameworks to provide constant transparency into the input and output data quality across all pipelines.
- • Pipeline architecture: Design and oversee the creation of well-defined dbt models that optimize reusability, performance, and maintainability while ensuring alignment with long-term strategic goals.
- • Data preparation for modeling: Lead the strategy for data preparation across multiple ML use-cases including unstructured text classification, ensuring data quality standards that enable model performance at scale.
- • Measurement & optimization: Lead the analytical measurement and efficacy of the entity resolution process as well as creating tooling that surfaces data quality issues early. Drive continuous improvement initiatives.
- • Strategic planning: Convert highly ambiguous, strategic initiatives into structured roadmaps with clear milestones, dependencies, and success metrics.
Experience and Qualifications
- • 8+ years using SQL / building data pipelines in SQL (snowflake experience highly preferred)
- • 5+ years developing entity resolution algorithms with proven success at scale
- • 5+ years with dbt, including architecture of large-scale dbt projects
- • 8+ years working with Python for data engineering
- • 4+ years experience with Spark (preferred)
- • 5+ years in startups with demonstrated leadership experience
- • 3+ years leading or mentoring engineering teams
- • Proven track record of architecting and scaling analytical workloads into production systems
- • Strong strategic thinking with ability to translate business needs into technical solutions
- • Highly opinionated on analytical approaches and able to articulate tradeoffs between different methodologies to technical and non-technical stakeholders
- • Demonstrated ability to lead complex, ambiguous projects from conception to successful delivery
- • Experience establishing engineering best practices and standards within a team