Evaluation on private operational data
Task-level suites built from synthetic and de-identified records that mirror a client's real distribution, scored per step, not just end to end.
Applied AI engineering lab · Palo Alto
Engineered by hand.
Engineered by hand.
Run by agents.
We build production agents for healthcare and other regulated enterprises, and the systems that make them safe to run: typed tools, durable execution, gated actions and an audit trail for every step.
t (s)kindspanattributesdur
Systems
In production.
Each system is an agent, the connectors it is allowed to touch, the gate it must pass, and the eval suite that decides whether a change ships.
01Clinical operations
Clinical Manager
Ranks the day's clinical work by risk, drafts the follow-ups, and routes each one to the clinician who owns it for approval.
| Domain | Clinical operations |
|---|---|
| Status | Production |
| Trigger | Schedule and change events from the systems of record |
| Reads | Schedules, orders, documentation and staffing, scoped per role |
| Writes | Drafts only. Every outbound action passes the approval gate |
| Gate | Role-scoped approval, bound to the exact payload |
| Models | Routed per step through the gateway; structured outputs |
| Evals | Ranking and drafting suites, run in CI on every change |
02Regulatory compliance
Compliance Manager
Tracks every credential, training and audit date as state, assembles evidence continuously, and surfaces risk before it becomes a finding.
| Domain | Regulatory compliance |
|---|---|
| Status | Production |
| Trigger | Daily evaluation plus change events on people and documents |
| Reads | Credentials, trainings, audit calendars and policy library |
| Writes | Staged evidence and drafted notices, gated before commit |
| Gate | Role-scoped approval, bound to the exact payload |
| Models | Retrieval over the client's own policies, cited by section |
| Evals | Deadline, retrieval and citation suites, run in CI |
Architecture
The reference design.
Every deployment is built on the same spine. The model is one component. Most of the engineering is everything around it.
// agents/compliance-manager.ts (illustrative)
export const complianceManager = agent({
id: "compliance-manager",
trigger: [cron("0 6 * * *"), on("credential.updated")],
models: route({ reason: "frontier", extract: "fast" }),
tools: [credentials.list, trainings.status, policies.search,
notices.draft, binder.stage],
scope: { phi: "deny", write: ["binder.staging"] },
output: Findings, // zod schema, validated
gates: [approve({ role: "compliance_lead",
on: ["notices.send", "binder.commit"],
bind: "payload_sha256" })],
evals: suite("compliance/*", { blockOn: "regression" }),
audit: "append-only",
});Invariants
Six rules every system holds.
Not guidelines. Each one is enforced in code, in CI, or both, and a change that breaks one does not merge.
- INV-01
Least privilege
Every connector is scoped to the fields one job needs. No agent holds a credential broader than its task.
- INV-02
Typed boundaries
Tool inputs and model outputs are validated against schemas at every boundary. Invalid output is a failed step, never a guess.
- INV-03
Idempotent effects
Every side-effecting call carries an idempotency key. Runs are resumable state machines; a retry cannot double-send.
- INV-04
Gated consequences
Actions are classed by consequence. Anything that leaves the system needs a role-scoped approval bound to the exact payload.
- INV-05
Append-only record
Inputs, tool calls, model calls, outputs and approvals are written to an append-only audit log, reviewable per run.
- INV-06
Evals gate merges
Prompt, model and tool changes run the eval suites in CI. A regression blocks the merge like a failing unit test.
Research
Where the hard problems are.
Open problems we work on because production agents in regulated settings depend on them.
Long-horizon reliability
Durable orchestration for multi-step agents: checkpointing, partial-failure recovery, and behavioral regression testing across model versions.
Deployment in regulated settings
Consequence classing, approval design and audit schemas for environments where an error carries clinical or legal weight.
Model-agnostic systems
Per-step routing across frontier models from every lab behind one gateway, so a model change is configuration with an eval run, not a rewrite.
Security and compliance
Security is the architecture, not a layer on it.
Built for healthcare and other regulated industries. Every system is designed against a written threat model, every control is enforced in code, in CI or in contract, and every control maps to the framework a client's auditor will ask about.
Threat model
- T1Prompt injection
- T2Data exfiltration
- T3Excessive agency
- T4Cross-tenant leakage
- T5Approval forgery
- T6Supply chain
HIPAA Security Rule · technical safeguards
| §164.312(a)(1) | Access control |
|---|---|
| §164.312(b) | Audit controls |
| §164.312(c)(1) | Integrity |
| §164.312(d) | Person or entity authentication |
| §164.312(e)(1) | Transmission security |
Frameworks
- HIPAABAAs where PHI is involved
- SOC 2Controls aligned to TSC
- NIST AI RMFGovern · Map · Measure · Manage
- OWASP LLM Top 10Controls per risk
Stack
How it is built.
The stack and the discipline, so an engineer can read this page and know what to expect.
- Language
- TypeScript end to end, strict mode. Shared schemas generate the types, the validators and the API contracts.
- Data
- Postgres with row-level security on every tenant table. Migrations rehearsed on a branch database before production. Point-in-time restore.
- Runtime
- Serverless and edge functions. Long-running agent work on durable background jobs with checkpoints and dead-letter queues.
- Models
- One gateway in front of every model call: allowlisted models, per-step routing, structured outputs, prompt caching, streaming, and per-tenant spend metering.
- APIs
- Versioned, date-pinned APIs. Idempotency keys on every write. Signed webhooks with replay protection. SDK generated from the same schemas.
- Delivery
- Trunk-based. A preview deployment per change. Build-stamped releases, automated visual, accessibility (WCAG 2.2 AA) and performance checks, one-step rollback.
- Observability
- Structured traces per run, per-tenant dashboards for uptime, latency, model spend and eval scores, and a daily live check on every deployment.
Frontier models, every lab
Openings
The books are closed.
We work with a few enterprise companies at a time. If you have a project and want to know whether an opening comes up, write to us. You will get a straight answer either way.
Not taking new clients