cognovís health
sdk · llm gateway

LLM gateway: governed intent routing.

@chp/llm-gateway routes LLM requests to the route assigned to a governed intent by the deployment binding, with PII type enforcement, intent catalog validation, and mandatory audit logging. Three endpoint types: POST /v1/chat/completions (text), POST /v1/audio/transcriptions (batch audio), and GET /v1/audio/transcriptions/live (WebSocket live transcription).

Overview

What the gateway does.

Callers name a governed intent and classify the data's PII mode. The gateway resolves the job through the deployment binding, enforces hard constraints defined in the intent catalog, and writes an immutable audit entry for every dispatch. No model names leak into application code.

Application Code
      |
      |  IntentModelRequest { intent, tuning, pii }
      v
┌─────────────────┐
│  llm-gateway     │   1. validate intent against catalog
│                  │   2. enforce PII constraints
│  routeRequest()  │   3. resolve the deployment job binding
│                  │   4. write audit entry
└─────────────────┘
      |
      |  selected ModelConfig + AuditEntry
      v
Upstream LLM API (OpenAI-compatible)

Intent-first SDK

import {
  createPolarisIntentChatClient,
  polarisIntentBoundary,
} from '@chp/sdk/llm'

const chat = createPolarisIntentChatClient({ baseUrl, machineAuthorization })
await chat.complete({
  messages: [{ role: 'user', content: anonymizedText }],
  gateway: { intent: 'patient-chat-proposal-extraction', pii: 'anonymized' },
})

const audio = polarisIntentBoundary({ baseUrl, machineAuthorization, workforceAuthorization: practitionerJwt })
await audio.transcribeAudio({
  intent: 'practitioner-audio-transcription',
  audio: audioBytes,
  contentType: 'audio/wav',
})

The older capability request, createPolarisChatClient, polarisBoundary, and their legacy types are not part of @chp/sdk 1.0.0 or @chp/llm-gateway 1.0.0. Pass the practitioner JWT as workforceAuthorization.

POST /v1/chat/completions

Text generation and transcription summarization. The deployment binding routes each governed intent to a configured model.

POST /v1/audio/transcriptions

Batch audio transcription. Uploads to AssemblyAI EU, polls to completion, best-effort deletes the transcript from the provider.

GET /v1/audio/transcriptions/live

WebSocket upgrade for live dictation. Proxies binary audio frames to AssemblyAI EU realtime and relays interim + final Turn events back to the client.

Canonical reference

All architectural decisions documented here are derived from ADR-020 v2. ADR-llm-gateway is the authoritative source — when this page and the ADR conflict, the ADR wins.

Capability vocabulary

19 capabilities across 4 axes.

Every capability is a member of the KnownCapability union type exported from @chp/llm-gateway. Catalog entries use capabilities to describe governance and readiness. New callers send a nonempty intent; deployment bindings own model selection.

import type { KnownCapability } from '@chp/llm-gateway'

// KnownCapability is a union of all 19 capability strings:
type KnownCapability =
  // Modality
  | 'text' | 'audioIn' | 'audioOut' | 'vision' | 'computerUse' | 'streaming'
  // Reasoning
  | 'reasoning' | 'longContext' | 'jsonMode' | 'toolUse' | 'agentic'
  // Domain
  | 'germanLanguage' | 'medicalGermanLanguage' | 'medicalCoding' | 'multilingual' | 'simplifiedLanguage'
  // Deployment
  | 'local' | 'lowLatency' | 'batch'

Modality axis

Capability Description
text Basic text generation; foundation capability required by all text-based intents
audioIn Audio transcription/speech-to-text input processing
audioOut Text-to-speech audio output generation
vision Image and document visual understanding
computerUse Autonomous GUI/computer interaction (PVS navigation)
streaming Streaming token-by-token response delivery

Reasoning axis

Capability Description
reasoning Extended chain-of-thought reasoning for complex clinical decisions
longContext Large context window (100k+ tokens) for long documents
jsonMode Guaranteed JSON-structured output
toolUse Function/tool calling capability
agentic Autonomous multi-step task execution

Domain axis

Capability Description
germanLanguage German language fluency for administrative content
medicalGermanLanguage Medical German: clinical terminology, Arztbriefe, Anamnesedokumentation
medicalCoding ICD-10, EBM, GOÄ, GOZ coding knowledge
multilingual Multi-language support beyond German
simplifiedLanguage Simplified/plain language output (patient-facing)

Deployment axis

Capability Description
local On-premises/self-hosted deployment (no data leaves the network)
lowLatency Optimized for real-time interactive use (<500ms first token)
batch High-throughput batch processing mode

PII type system

PseudonymizedPrompt and RealPrompt.

The gateway enforces PII classification at the type level. You cannot pass a RealPrompt to an intent that declares pii: 'anonymized' — the TypeScript compiler rejects it before the code runs. Both types are opaque: they can only be constructed via their from() factory, ensuring input validation runs on every prompt.

PseudonymizedPrompt

import { PseudonymizedPrompt } from '@chp/llm-gateway'
import type { PseudonymizedInput } from '@chp/llm-gateway'

// PseudonymizedInput requires anonymizedText —
// no name, no DOB, no insurance number fields
const input: PseudonymizedInput = {
  anonymizedText: 'Patient, 42J, Rückenschmerzen seit 3 Wochen.',
}

// Opaque prompt — carries a .text property
// but cannot be accidentally created from raw strings
const prompt = PseudonymizedPrompt.from(input)

// Use with intents that declare pii: 'anonymized'
// e.g. billing-coding-suggest, scribe-stt-summarize

RealPrompt

import { RealPrompt } from '@chp/llm-gateway'
import type { RealInput } from '@chp/llm-gateway'

// RealInput allows identified patient data
// Only valid for intents that declare pii: 'real'
// AND models with local or DSGVO-compliant deployment
const input: RealInput = {
  text: 'Full clinical note with patient context...',
  practitionerJwt: '<bearer-token-from-keycloak>',  // Required: Practitioner-JWT (ADR-llm-gateway)
}

const prompt = RealPrompt.from(input)

// Gateway rejects RealPrompt on anonymized intents —
// type error + runtime enforcement

PII modes

Anonymized and real PII modes.

The gateway enforces PII classification per request via the gateway.pii field. anonymized mode (the default) is required for all background agents. real mode requires a Practitioner-JWT and is only available for intents explicitly designed for identified patient context.

Mode Auth requirement Description
anonymized Service token (default) PHI must be removed or tokenized before forwarding to the LLM provider. phi_references enables tokenization. PII patterns must not appear in the post-tokenization prompt. When every route of the job is privacy-approved (local, or EU-hosted under a processing agreement), the original text is forwarded without detection or review; see privacy-approved routes.
real Practitioner-JWT required Patient-identified data is forwarded. Only valid on intents designed for identified patient context. Rejected with HTTP 401 when a service token is used.

PHI tokenization

In anonymized mode, callers may declare PHI strings via gateway.phi_references. The gateway replaces each declared value with an opaque token (e.g. [Patient-1]) before forwarding, and restores FHIR references in the response. The token-to-PHI mapping never leaves process memory. See the Prompt-Formulation Guide for declaration contract details and worked examples.

Declaration contract

Exhaustive declaration fast-path.

When a caller can enumerate all PHI in the request, it may set gateway.declaration: "exhaustive". The gateway then cross-checks the tokenized prompt for a residual insurance number (KVNR). If no residual PII is found, the request is forwarded without calling the model-routed detector — lower latency and resilient to detector-model downtime.

How the cross-check works

  • The caller declares all PHI strings via phi_references and sets declaration: "exhaustive".
  • The gateway tokenizes declared PHI, replacing names and identifiers with [Patient-N] tokens.
  • The tokenized prompt is scanned for a residual KVNR ([A-Z]\d{9}). Dates and postal codes are not residual patterns; they are forwarded unchanged.
  • No match: request is forwarded. Match: HTTP 422 caller_declaration_violation is returned.

422 error responses

Re-ID preflight blocked

HTTP 422
{
  "error": "Re-ID risk preflight: request blocked
            — quasi-ID combination uniquely
            identifies patient",
  "doc_url": "https://cognovis-health.de/docs/sdk/llm-gateway/prompt-formulation/"
}

Declaration violation

HTTP 422
{
  "error": "caller_declaration_violation:
            undeclared PII found after tokenization",
  "doc_url": "https://cognovis-health.de/docs/sdk/llm-gateway/prompt-formulation/"
}

Prompt-formulation guide

The Prompt-Formulation Guide documents the full declaration contract, the known-patient review contract, quasi-ID conventions for caller-supplied quasi_ids, classification-code pass-through, PHI declaration rules, PASS and FAIL examples, cross-check failure modes, and migration from the legacy combination_count API.

Known-patient review

Physician review before personal data leaves the gateway.

Known-patient requests (patient-chat, patient-chat-proposal-extraction, befund-extraction, befund-grounding) carry the patient's identifiers in gateway.patient_identity (patient_ref, name, address, birth date, insurance number) from an authenticated service caller; without it the gateway answers 400 patient_identity_required. The gateway matches these values deterministically, flags master-data patterns of any person (KVNR, street with house number, postal code with city, a date after a birth phrase), assesses other free text for person names only, and never changes other dates, codes, units or dosage. The envelope is never forwarded, logged or audited. A request with the envelope always takes this path: declaration: "exhaustive" or a non-empty phi_references next to it is refused with 400 invalid_chat_request.

A finding, or a names assessment that cannot complete, returns 428 pii_review_required with the findings (original excerpt, locations, proposed token) and no model call. The physician resends the identical request with gateway.pii_review: an explicit mask or include per finding, or cancel. Reviews are single use, expire after 15 minutes and are bound to tenant, consumer, physician, intent, route and the exact request (409 pii_review_invalidated, pii_review_consumed, pii_review_unknown). A confirmed request makes exactly one model attempt on the bound route, never a fallback; JSON answers receive JSON-escaped originals. Personal data never causes a 503. The review applies only when at least one route of the job, the first or any fallback, is not privacy-approved; otherwise the gateway forwards the original text and returns no pii_review_required. See Section 2a of the Prompt-Formulation Guide.

Tokenization fast-path

PHI tokenization pipeline.

When pii: 'anonymized' and phi_references are provided, the gateway runs the tokenization pipeline: PHI strings are replaced with opaque tokens before forwarding, and tokens in the LLM response are restored to FHIR references before returning to the caller. The token-to-PHI mapping lives in process memory only — it is never logged, never audited as values, and is dropped after each request.

Caller request                  Gateway pipeline
────────────────                ─────────────────────────────────────────────
messages: [{                   1. tokenizeMessages(messages, phi_references)
  content:                          → replaces "Erika Müller" with [Patient-1]
    "Erika Müller,                   → replaces "Dr. Schmidt" with [Practitioner-1]
     Altersgruppe 51-65,
     Hauptdiagnose E11.65"       2. (if declaration: "exhaustive")
}]                                 → cross-check tokenized content for residual PII
                                     KVNR pattern
gateway: {                         → 422 caller_declaration_violation if match found
  phi_references: [
    { resourceType: "Patient",  3. Forward tokenized messages to LLM provider
      id: "pvs-patient-12345",       phi_references + reid_preflight stripped
      values: ["Erika Müller",
               "Müller"] },     4. LLM response contains [Patient-1], [Practitioner-1]
    { resourceType:
      "Practitioner",            5. detokenizeText(response, tokenCtx)
      id: "pvs-pract-42",            → restores tokens to FHIR references
      values: ["Dr. Schmidt",        "Patient/pvs-patient-12345"
               "Schmidt"] }         "Practitioner/pvs-pract-42"
  ],
  declaration: "exhaustive"     6. dropContext(tokenCtx) — map cleared from memory
}

Security guarantee

The token-to-PHI mapping never touches disk, logs, or the audit trail. Only token counts and resource types are recorded. The mapping is cleared immediately after the response is detokenized.

Token format

[Patient-N], [Practitioner-N], [Encounter-N], [Condition-N]

Token numbers are randomized per request for cross-request unlinkability (ADR-privacy-and-agent-data-plane).

Audit entry

The audit entry records tokenization_audit.token_count and resource_types only — never PHI values or the token mapping.

TuningParams reference

Controlling model behavior.

TuningParams are optional hints attached to a ModelRequest. The gateway translates these into provider-specific parameters (temperature, max_tokens, response_format) before dispatch. Intents in the catalog declare their expected tuning — the gateway validates that caller tuning is compatible.

Parameter Type Description
effort 'low' | 'medium' | 'high' Reasoning depth and cost trade-off. Use high for coding/billing suggestions, low for classification tasks.
creativity 'deterministic' | 'balanced' | 'creative' Controls temperature. Clinical and billing intents must use deterministic.
responseFormat 'text' | 'json' | 'markdown' Output format hint. json activates jsonMode capability requirement automatically.
streaming boolean Enable streaming token delivery. Activates streaming capability requirement.
maxTokens number Maximum output token count. Omit to use model default.

Intent catalog

20 intents — 13 full, 7 stub.

The intent catalog is the runtime contract between callers and the gateway. Full intents have complete capability declarations, hard constraints, and tuning. Stub intents are reserved IDs with risk annotations — implementation is in-work. Specifying an unknown intent ID at runtime raises a validation error.

The catalog is loaded from integration-layer/packages/llm-gateway/intent-catalog.yaml via loadCatalogFromFile() or embedded inline via loadCatalogFromYaml(). Use validateIntent() to check an intent ID against the loaded catalog at startup.

Selected full intents

billing-coding-suggest full risk: green approval queue

Suggest ICD-10/EBM/GOZ billing codes from anonymized encounter documentation

Medical device risk: none — coding suggestion, physician decides

Requires

textmedicalCodinggermanLanguage

Prefers

reasoningmedicalGermanLanguage

Tuning

effort: highcreativity: deterministicresponseFormat: json

Hard constraints

  • input must be PseudonymizedPrompt — no PII permitted
  • output is suggestion only — physician must confirm before submission
  • must log to audit trail with capabilities_matched and model_used
scribe-stt-summarize full risk: green approval queue

Transcribe and summarize spoken consultation notes into structured FHIR-compatible documentation

Medical device risk: none — summary goes to approval queue before chart entry

Requires

audioIntextgermanLanguage

Prefers

medicalGermanLanguagestreaming

Tuning

effort: highcreativity: deterministicresponseFormat: markdown

Hard constraints

  • audio stream must be processed locally or via DSGVO-compliant processor
  • output must go to approval queue before any chart write
  • no direct FHIR write without physician approval
anamnese-translate full risk: green

Translate patient anamnesis from non-German language to German, verbatim without clinical interpretation

Medical device risk: none — verbatim translation, no clinical interpretation

Requires

textmultilingual

Prefers

simplifiedLanguage

Tuning

effort: mediumcreativity: deterministicresponseFormat: text

Hard constraints

  • verbatim translation only — no summarization, no clinical interpretation
  • no triage decisions — translator role only
  • source language must be identified and logged
patient-chat-proposal-extraction full risk: green approval queue

Extract explicitly authored actions from anonymized patient chat into structured JSON proposals

Medical device risk: none — review-only structured proposals, no execution

Requires

textjsonModegermanLanguage

Prefers

reasoningmedicalGermanLanguage

Tuning

effort: mediumcreativity: deterministicresponseFormat: json

Hard constraints

  • input must be anonymized before provider forwarding unless every bound route of the job is privacy-approved
  • output contains proposal candidates only and never executes an action
  • ambiguous, unsupported, or unauthored actions yield no proposal
practitioner-audio-transcription full risk: green

Transcribe practitioner-recorded clinical audio to plain text without summarization

Medical device risk: none — verbatim transcription without interpretation

Requires

audioIntextgermanLanguage

Prefers

medicalGermanLanguage

Tuning

creativity: deterministicresponseFormat: text

Hard constraints

  • real PII uses the authenticated audio boundary
  • output must not summarize, diagnose, code, or trigger an action
  • audit logging excludes audio and transcript content

Stub intents — reserved, in-work

rule-detect-administrative stub risk: green

Classify incoming documents/tasks as administrative vs clinical

Hard constraints

  • output restricted to allowlist of administrative categories — no free-text clinical output
scribe-fhir-resource-suggest stub risk: green

From consultation transcript, suggest structured FHIR resources

Hard constraints

  • output must go to approval queue — no direct FHIR write
  • physician must review and approve each suggested resource
voice-doc-gap-billing stub risk: green

Detect potential billing gaps

MDR: 5a Undercoding — detection only

Hard constraints

  • detection only — no automatic billing submission
  • must present evidence (documentation snippet) for each detected gap
voice-doc-gap-quality stub risk: green

Flag documentation quality gaps

MDR: 5b QM-Handbuch

Hard constraints

  • output is quality flag only — no treatment recommendation
  • reference to QM handbook section mandatory per finding
voice-doc-gap-jtbd stub risk: green

Identify jobs-to-be-done from transcripts

MDR: 5c JTBD

Hard constraints

  • anonymized input only
  • output is operational insight — not clinical decision support
admin-day-protocol-flag stub risk: green

Flag items in daily administrative protocols

Hard constraints

  • output restricted to allowlist of flag types
  • no clinical diagnosis in output
pvs-computer-use stub risk: yellow

Automate PVS UI interactions

MDR: yellow-operational (navigation) / green-medical (data entry only)

Hard constraints

  • BLOCKED: dicom-letter-from-text use case is explicitly excluded from this intent
  • scope limited to data entry and navigation — no autonomous clinical decision
  • requires explicit operator opt-in per session

Code snippets

Three complete examples.

All snippets use real types from @chp/llm-gateway. Each snippet shows how to set up the gateway with createApp and how to call it. New calls pass a nonempty intent and PII mode inside the body.gateway field.

Example 1 — billing-coding-suggest

Suggest ICD-10/EBM/GOZ billing codes. Input must be anonymized — the PseudonymizedPrompt type enforces this at compile time. Output goes to an approval queue before reaching the billing system.

import { createApp, PseudonymizedPrompt } from '@chp/llm-gateway'
import type { PseudonymizedInput, ModelConfig, LlmProvider, GatewayConfig } from '@chp/llm-gateway'

// 1. Configure models (provided by ops/platform team)
const models: ModelConfig[] = [
  {
    id: 'anthropic/claude-sonnet-4-5',
    provider: 'anthropic',
    modelName: 'claude-sonnet-4-5',
    capabilities: ['text', 'reasoning', 'medicalCoding', 'germanLanguage', 'medicalGermanLanguage', 'jsonMode'],
  },
]

// 2. Wire in your LLM provider (LiteLLM, Anthropic SDK, etc.)
const llmProvider: LlmProvider = async (model, request) => {
  // Forward to your actual provider implementation
  throw new Error('Replace with real provider implementation')
}

// 3. Create the gateway app
const config: GatewayConfig = {
  models,
  intentCatalogPath: './packages/llm-gateway/intent-catalog.yaml',
}
const app = createApp({ config, llmProvider })

// 4. Build an anonymized prompt (type-safe — no raw strings accepted)
const anonymizedInput: PseudonymizedInput = {
  anonymizedText: 'Patient klagt über anhaltende Rückenschmerzen seit 3 Wochen, keine Ausstrahlung.',
}
const prompt = PseudonymizedPrompt.from(anonymizedInput)

// 5. POST to the chat completions endpoint
const response = await app.fetch(
  new Request('http://localhost/v1/chat/completions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      messages: [{ role: 'user', content: prompt.text }],
      gateway: {
        intent: 'billing-coding-suggest',
        pii: 'anonymized',
      },
    }),
  }),
)
const result = await response.json()
// result.choices[0].message.content — JSON billing code suggestions
// Must be routed to physician approval before billing submission

Example 2 — scribe-stt-summarize

Transcribe and summarize spoken consultation notes. Audio must be processed locally or via a DSGVO-compliant processor. Output goes to an approval queue before any chart write.

// scribe-stt-summarize: audio transcription + summarization
// Note: audioIn capability requires the llmProvider to handle audio content.
// The prompt text here represents the transcribed audio content.

const app = createApp({
  config: {
    models: [
      {
        id: 'anthropic/claude-opus-4-5',
        provider: 'anthropic',
        modelName: 'claude-opus-4-5',
        capabilities: ['text', 'audioIn', 'germanLanguage', 'medicalGermanLanguage', 'streaming'],
      },
    ],
    intentCatalogPath: './packages/llm-gateway/intent-catalog.yaml',
  },
  llmProvider,
})

const transcribedNote = PseudonymizedPrompt.from({
  anonymizedText: 'Anamnese: Keine Vorerkrankungen. Aktuell Husten seit 5 Tagen, leichtes Fieber 37.8°C.',
})

const response = await app.fetch(
  new Request('http://localhost/v1/chat/completions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      messages: [
        { role: 'system', content: 'Erstelle eine strukturierte Dokumentation aus den folgenden Konsultationsnotizen.' },
        { role: 'user', content: transcribedNote.text },
      ],
      gateway: {
        intent: 'scribe-stt-summarize',
        pii: 'anonymized',
      },
    }),
  }),
)
// response includes X-Approval-Required: true header — must go to approval queue
const approvalRequired = response.headers.get('X-Approval-Required') === 'true'

Example 3 — anamnese-translate

Translate patient anamnesis from a non-German source language to German. Verbatim only — no summarization, no clinical interpretation, no triage decisions. Source language must be identified and logged.

// anamnese-translate: translate non-German patient anamnesis to German
const app = createApp({
  config: {
    models: [
      {
        id: 'anthropic/claude-sonnet-4-5',
        provider: 'anthropic',
        modelName: 'claude-sonnet-4-5',
        capabilities: ['text', 'multilingual', 'simplifiedLanguage'],
      },
    ],
    intentCatalogPath: './packages/llm-gateway/intent-catalog.yaml',
  },
  llmProvider,
})

const foreignLanguageText = PseudonymizedPrompt.from({
  anonymizedText: 'The patient reports chest pain that started two days ago, especially when breathing deeply.',
})

const response = await app.fetch(
  new Request('http://localhost/v1/chat/completions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      messages: [
        {
          role: 'system',
          content: 'Übersetze den folgenden Text wörtlich ins Deutsche. Keine Zusammenfassung, keine klinische Interpretation.',
        },
        { role: 'user', content: foreignLanguageText.text },
      ],
      gateway: {
        intent: 'anamnese-translate',
        pii: 'anonymized',
        // No approval queue — pure translation, no clinical decision
      },
    }),
  }),
)
// approvalQueue: false for this intent — result can be used directly
// Gateway audit entry records source language + model used

Operator configuration

LLM providers are configured in the Integration Layer only.

The Practice Workspace and other application code call the Integration Layer by a governed intent. Provider credentials (Anthropic, OpenAI, Scaleway, OpenRouter, Ollama) stay in the Integration Layer gateway configuration — not in Practice Workspace env vars.

Scaleway (preferred EU route)

Scaleway Generative APIs are EU-hosted and OpenAI-compatible at https://api.scaleway.ai/v1. The TypeScript policy gateway routes scaleway-glm-5-2 through LiteLLM when bound by the deployment; Other providers remain available when a reviewed deployment binding configures them through LiteLLM.

# Integration Layer operator env (not the Practice Workspace)
SCW_SECRET_KEY=...

# integration-layer/infrastructure/llm-gateway/model-catalog.yaml
models:
  scaleway-glm-5-2:
    title: Scaleway GLM 5.2
    attributes: [text, reasoning, germanLanguage, medicalGermanLanguage, jsonMode]

# Additional providers are declared in the deployment binding. The generated
# LiteLLM artifacts forward the credential and API-base environment names.

Application boundary

SDK consumers use @chp/sdk/llm types and the gateway HTTP API. They name a governed intent and PII mode; the deployment binding selects the model and the gateway records model_used, provider, and token counts in the audit trail. Missing provider credentials fail with llm_provider_error (502) — no silent fallback to another provider.

Discovery endpoint

Querying gateway capabilities at runtime.

The gateway exposes a capability discovery endpoint for aggregate feature availability and readiness diagnostics. It does not choose a model or prove that a specific intent is bound.

Request

GET /api/llm/capabilities
Authorization: Bearer <api-key>

Response shape

{
  "capabilities": [
    "text", "reasoning", "germanLanguage",
    "medicalGermanLanguage"
    // ... all capabilities any configured model covers
  ],
  "coverage": {
    "text": ["claude-sonnet-4-5", "claude-opus-4-5"],
    "reasoning": ["claude-opus-4-5"],
    "germanLanguage": ["claude-sonnet-4-5", "claude-opus-4-5"],
    "medicalGermanLanguage": ["claude-opus-4-5"],
    "local": []
  },
  "default": "text"
}

Use at startup

Use discovery during application initialization to show broad feature availability. Inspect the deployed catalog and binding for job readiness; request routing follows the intent's binding.

Execution receipt

Hash-only evidence of each executed invocation.

Every successful buffered POST /v1/chat/completions response and every successful batch POST /v1/audio/transcriptions response carries one additive top-level field, execution_receipt (receipt_version: "chp.execution-receipt.v1" from @chp/llm-gateway 2.0.0 on). The gateway sets it; a provider value under that key is replaced. Streaming chat, live audio and error responses carry no receipt, and all other response fields are unchanged.

The receipt is hash-only (content_mode: "hash-only"). It never contains the prompt sent upstream, a transcript, audio bytes, STT prompt or keyterm text, patient identifiers or credentials, and no request field enables such content. The gateway does not log receipts or write them to the audit trail. Validate a receipt with parsePolarisExecutionReceipt from @chp/sdk/llm; one call handles chat and transcription receipts and returns only the fields of the version it knows.

Fields

{
  receipt_version, invocation_id, correlation_id,
  capability,   // "chat.completions" | "audio.transcriptions"
  intent, pii_mode, started_at, completed_at,
  execution: {
    deployment, binding_fingerprint,
    gateway_release: { version, release_id, image_digest },
    route_alias, provider, model, residency,
    fallback_position, model_card,
    provider_reported_model  // upstream-reported model or null
  },
  parameters,   // temperature, max_tokens, response_format, language
  usage,        // chat token counts; null for transcription
  request_digest: { algorithm: "sha256",
    canonicalization: "chp-canonical-json.v1",
    basis, value },
  input_artifacts, // transcription: [{ kind: "audio", sha256,
                   //   byte_length, media_type, duration_seconds }]
  content_mode: "hash-only"
}

correlation_id is the request's correlation id in lower case when it is a canonical UUID and null for any other value; the X-Correlation-Id response header and the audit trail keep the id as sent. An empty or blank X-Correlation-Id counts as absent, so the gateway mints a UUID. input_artifacts[].media_type is the lower-case MIME essence (type/subtype, parameters dropped) of the media type sent upstream, or null when that value is not a media type. execution.model is the route's configured model; execution.provider_reported_model is the model the upstream response reported (AssemblyAI speech_model_used, the LiteLLM response model) when it is a single-line identifier of at most 256 characters, and null otherwise.

Request digest

SHA-256 (lowercase hex) over the canonical JSON — keys sorted at every depth, no whitespace, arrays in order, absent members omitted — of the provider request after routing, PII tokenization, prompt assembly and tuning normalization, for the fallback candidate that succeeded. basis names the hashed shape: litellm-chat (the chat body sent to LiteLLM), litellm-transcription (model, resolved language and joined prompt), assemblyai-transcription (the pass-through submission without the provider-assigned upload URL) or gateway-request (the gateway-normalized request handed to an adapter of another provider kind). Transcription digests represent the audio as audio: { sha256, byte_length, media_type } with the media type exactly as the adapter sends it (the upload Content-Type for AssemblyAI, the file part's Content-Type for LiteLLM); litellm-transcription adds the multipart filename. The gateway normalizes this metadata before transport, identically on every runtime: for both adapters the media type is sent without surrounding whitespace when it is a valid ASCII media type (type/subtype with optional ; parameters) and as application/octet-stream otherwise. The AssemblyAI upload keeps the caller's spelling, while the LiteLLM file part also sends application/octet-stream for a type with a tab. For LiteLLM the filename has ", CR and LF percent-encoded (%22, %0D, %0A) and the text fields have line breaks normalized to CRLF. The digest hashes this normalized request as the upstream receives it; for LiteLLM the gateway reads the file part's type and filename and the text fields back from the serialized multipart body. The filename is hashed and never returned.

Model Card

execution.model_card is the route's configured modelCard reference (an https: or urn: URI) exactly as the binding declares it, and null when the route configures none. On an AssemblyAI route it is also null when the upstream reports a speech model other than the route's, because the card would describe a model that did not run. The gateway never derives or invents one.

Receipt failure

The receipt is part of the success path. If it cannot be built, the gateway returns execution_receipt_unavailable (HTTP 500) and no model output.

Public exports

Quick-start exports.

The most-used exports are listed below. See the package's TypeScript types for the full export surface.

Types

import type {
  KnownCapability,    // 19-member union
  Capability,         // string (open for custom caps)
  TuningParams,       // effort / creativity / responseFormat
  IntentModelRequest, // nonempty intent + tuning + pii
  PseudonymizedInput,    // { anonymizedText: string }
  RealInput,          // { text: string }
  ModelConfig,        // per-model gateway configuration
  GatewayConfig,      // top-level gateway config
  AuditEntry,         // immutable audit log record
  AuditStore,         // storage interface for audit entries
} from '@chp/llm-gateway'

Values

import {
  // Prompt constructors
  PseudonymizedPrompt,       // .from(PseudonymizedInput): opaque prompt
  RealPrompt,             // .from(RealInput): opaque prompt

  // Audit stores
  NoopAuditStore,         // discards all entries (testing)
  CapturingAuditStore,    // in-memory capture (testing)

  // Gateway core
  routeRequest,           // low-level routing function
  scanForPii,             // PII detection utility

  // Intent catalog
  loadCatalogFromFile,    // load YAML catalog from disk
  loadCatalogFromYaml,    // load from YAML string
  validateIntent,         // validate intent ID against catalog

  // App factory
  createApp,              // creates the gateway Hono app
} from '@chp/llm-gateway'

Binary document processing

POST /v1/documents/process accepts multipart file (PDF, PNG or JPEG, up to 10 MiB) and an optional prompt (up to 32,000 characters). The single document-ocr job uses centrally bound plain or prompt routes and returns text, including JSON when requested. Applications validate the result. Model, tuning and output-schema fields are rejected.

Call createPolarisDocumentClient(...).processDocument(...) from @chp/sdk/llm or @chp/llm-gateway/client. Supply the workforce bearer and a same-tenant consumer bearer with llm:documents:invoke for integration-llm-gateway. Document egress requires approved local/EU, zero-retention, no-training, no-logging routes. Missing routes return document_route_unavailable (503); invalid or oversized input returns invalid_document_request (400) or document_payload_too_large (413). Expired or cancelled delegation returns document_processing_expired (504). Execution lasts at most 120 seconds and never outlives either bearer or an earlier SDK request delegationDeadline (epoch milliseconds).

Error Codes

All error responses from POST /v1/chat/completions use a structured envelope:
{ error: { code, errorClass, message, doc_url, details? } }

Code HTTP Error Class Description
invalid_json400RequestParseErrorRequest body is not valid JSON
practitioner_jwt_required401AuthenticationErrorpii: real requires a Practitioner-JWT, not a service token
unknown_intent400IntentValidationErrorIntent ID is empty, non-string, or not found in the catalog
red_risk_intent501IntentValidationErrorIntent is a known prohibited use case (red-risk denylist)
intent_catalog_unavailable503IntentCatalogErrorIntent provided but catalog not loaded — cannot validate
no_model_for_capabilities503CapabilityRoutingErrorNo configured model satisfies the required capabilities
reid_preflight_invalid_input400ReidPreflightErrorMissing or invalid quasi_ids / combination_count fields
reid_preflight_blocked422ReidPreflightErrorQuasi-ID combination is too unique — blocked to prevent patient re-identification
caller_declaration_violation422PiiDeclarationErrorUndeclared PII detected post-tokenization in declaration: "exhaustive" mode
output_pii_detected422OutputPiiFilterErrorResponse PII was detected and this deployment uses reject output filtering
llm_provider_error502LlmProviderErrorUpstream LLM provider returned an error or timed out
llm_budget_exhausted429LlmBudgetExhaustedErrorThe deployment's LiteLLM key or team budget is used up; not retried
llm_rate_limited429LlmRateLimitedErrorThe deployment's LiteLLM key or team rate limit is reached; not retried
llm_gateway_busy503LlmGatewayBusyErrorThe gateway's provider-wide ceiling and queue are full, or the request waited past the queue's maximum age; it never reached the provider. Retry after the seconds in Retry-After
invalid_request_class400RequestValidationErrorX-CHP-Request-Class is neither user nor optional; absent means user
patient_identity_required400RequestValidationErrorA known-patient intent was called without gateway.patient_identity
pii_review_required428PiiReviewRequiredFindings or an incomplete names assessment need a physician decision; nothing was transmitted
pii_review_decision_invalid400PiiReviewErrorThe decision is malformed (exact strings only) or does not name every finding exactly once with mask or include
pii_review_invalidated409PiiReviewErrorThe request, patient, physician, tenant or route differs from the reviewed request
pii_review_consumed409PiiReviewErrorThe review was already confirmed or cancelled
pii_review_unknown409PiiReviewErrorThe review is unknown or expired
execution_receipt_unavailable500ExecutionReceiptErrorThe provider call succeeded but the execution receipt could not be built; no model output is returned
invalid_chat_request400RequestValidationErrorChat request contains invalid messages or gateway metadata
invalid_audio_request400RequestParseErrorAudio transcription request must include an audio payload
audio_auth_required401AuthenticationErrorAudio transcription requires a Bearer token
audio_payload_too_large413RequestSizeErrorAudio transcription payload exceeds the configured size limit
audio_too_short422AudioTooShortErrorThe recording is too short to transcribe
audio_streaming_upgrade_required426WebSocketUpgradeErrorLive audio transcription requires a WebSocket upgrade request
machine_authorization_required401AuthenticationErrorA declared consumer machine credential is required for this service operation
machine_authorization_forbidden403AuthorizationErrorThe consumer is not authorised for this recipient, tenant, or operation
intent_auth_required401AuthenticationErrorIntent caller authentication and tenant context are required
intent_pii_mismatch422IntentValidationErrorRequested PII mode is incompatible with the governed intent
streaming_not_supported501StreamingNotSupportedErrorStreaming (stream: true) is not supported by this gateway deployment
pii_detector_unavailable503PiiDetectorErrorPII detector unavailable (legacy outcome of older gateways)
legacy_reid_fields_removed400ApiVersionErrorThe fields combination_count and practice_size were removed. Migrate to gateway.quasi_ids; see the prompt-formulation guide.

Canonical reference

ADR-llm-gateway v2 is the canonical source for all decisions documented here — intent governance, intent catalog schema, PII enforcement rules, audit trail requirements, and the discovery endpoint contract. When this page and ADR-llm-gateway conflict, the ADR wins.