Olli Health Senior AI Engineer Remote · Full time Company website

Apply for Senior AI Engineer

Olli Health is a Series A funded healthtech startup modernizing home healthcare with advanced AI tools, and we are growing. We're looking for a creative, low-ego, collaborative, startup-native engineer energized by challenging, nuanced problems with the potential to have a massive impact. Built by home health experts and seasoned product managers and engineers from Amazon, Google, Cohere Health, and Arrive Health, Olli Health streamlines ICD-10 coding and documentation with industry-leading speed and accuracy, freeing nurses to focus on patients while helping agencies maximize revenue, reduce compliance risk, and scale efficiently.

Key Responsibilities:

• Own an LLM problem area end-to-end: drive the roadmap, design, and shipping of one or more LLM-driven product surfaces, collaborating with product, clinicians, and engineering partners.
• Run high-leverage LLM experiments to advance the quality, response reliability (consistency, refusal, tool-use correctness), latency, and cost of our LLM workflows, and make calibrated model, prompt, retrieval, and agent design decisions.
• Evolve our evaluation and observability stack to better measure model behavior locally, in CI, and in production, and extend instrumentation and analytics so we can better track cost, latency, usage, and quality as our products scale.
• Discover, pilot, and integrate new quality signals, and build the human-in-the-loop workflows that feed our evaluation and training pipelines.
• Raise the team’s LLM bar through code/design review, pattern-setting, mentorship, and written design docs.
• Proactively identify areas of high risk in an ever-changing LLM landscape and integrate countermeasures into our workflows
• Partner with customers and internal clinical users to identify high-leverage opportunities, observe real usage, and ship customer-facing LLM features through to production under our HIPAA and clinical-safety constraints. 

Requirements:

• LLM Systems: Hands-on experience shipping and iterating production LLM features (prompting, retrieval, tools/agents, failure modes), grounded in relevant NLP fundamentals (e.g. embeddings, retrieval, tokenization, extraction/classification) to make informed tradeoffs.
• Evaluation and production alignment: Define and run offline and/or online evaluation; use outcomes to drive releases; operate LLM-backed features after deployment (monitoring, regressions, drift, closing gaps between eval and production).
• Engineering Execution: 5+ years demonstrating strong programming skills; disciplined software practices suitable for shared production codebases (design clarity, review participation, testability where appropriate). You are capable of using LLMs to accelerate development, but require an understanding of the code at a deep level before trusting the output. 
• Startup mentality: An enjoyment of that early startup balance between efficiency, scalability, and how to prioritize competing and complex demands meet customer needs.
• Regulatory and Safety Posture: Comfort working on applications subject to HIPAA and real clinical or operational consequences; eagerness to treat privacy, safe defaults, and proportionate controls as non-negotiable design inputs.
• Authorization: Must be authorized to work in the US and CA (no visa sponsorship available).

Desirable Skills (Bonuses):

• Proven success in the Seed/Series A stage startups.
• Experience training, evaluating, and iterating on supervised learning models in PyTorch or TensorFlow, including transformer-based architectures.
• Advanced degree (PhD or comparable industry/research depth) in ML, NLP, CS, or a related field.
• Experience with model observability, cost/latency optimization, LLM serving infrastructure, and the trade-offs between different frontier model providers.
• Experience with our stack: AWS, PostgreSQL, Redis, Python, React.
• Experience building data pipelines for analytics and ML-ready datasets.
• Experience working with healthcare datasets (e.g. payer claims data, clinical notes, EHR charts, ICD-10 coding, etc.).
• Experience building secure systems with HIPAA compliance as a core principle.

At Olli we value a growth mindset and problem-solving skills as much as specific tool proficiency and years of experience. If you have a solid foundation in engineering principles and are eager to learn, we want to hear from you.

What We Offer:

• A key early engineering role with equity vesting options.
• A chance to be at the forefront of healthtech innovation, building real AI tools that meet compliance standards while solving real problems (we don't build marketing gimmicks).
• A collaborative and inclusive work environment that prioritizes debate, ethical decision-making, and is allergic to inflated egos.
• Flexible remote work environment with travel to team onsite meetings 1-2 times per year.
• Opportunities for professional growth and leadership.
• Benefits: 100% coverage of Health, Vision, Dental, and Life insurance premiums for employees; unlimited PTO; 401K plan; individual budget for your preferred laptop and monitor setup; fully remote; great colleagues working on a challenging problem for meaningful customers (older adults).