Your gateway captures every prompt and response. Your auditors will still fail you. Here's the gap between request logging and inference audit. Closing it is an architecture problem, not a logging problem.

TL;DR: Gateway logs record the envelope of an LLM call (request, response, tokens, latency) but miss the decision process inside the model. Auditors under the EU AI Act and RBI guidelines need tamper-evident evidence of how each output was produced: model version, retrieval context, guardrail outcomes, and human overrides. Closing the gap requires a three-layer audit architecture (model internals, decision provenance, governance metadata) signed with hash chains and post-quantum signatures, built as serving middleware, not as in-model callbacks.

Key Takeaways: - Gateway logs are not audit trails. They capture what was asked; auditors need how the decision was made. - Synchronous in-model logging kills inference performance. Async observation substrates keep overhead under 7%. - A real audit trail layers model internals, decision provenance, and governance metadata, each answering a different auditor question. - Tamper-evidence comes from SHA-256 hash chaining plus ML-DSA-65 signatures, not from access controls alone. - EU AI Act Article 12 and RBI model risk guidelines both demand inference-level provenance that gateway dashboards cannot provide.

Most teams believe their cloud security stack already covers LLM audit. They point to gateway logs: every prompt, every response, every token count, every latency number. It looks complete on paper. It fails under auditor scrutiny.

The EU AI Act's Article 12 requires automatic event logging tied to every algorithmically driven decision. Gateway logs are tied to API calls, not decisions. The distinction matters because a single LLM call can produce downstream actions through function calls, agent loops, and tool invocations. The gateway never sees these.

The gateway logged the envelope. The decision happened elsewhere.

At the opposite end sits the NullLog pattern: zero data retention, total opacity to auditors. It satisfies privacy maximalists and fails every compliance review. Neither extreme produces an audit trail. One over-collects the wrong layer; the other under-collects everything.

This gap has surfaced as a recurring audit failure point in regulated industries. Teams arrive with petabytes of gateway logs and no usable evidence. When auditors ask "which model version made this decision," the answer is often "we can check the deployment log from last Tuesday, maybe."

But the gap isn't about logging more or less. It's about what happens inside the model between the prompt arriving and the token leaving. Auditors look at that layer when the envelope is not enough.

Gateway logs answer one question: what was asked and what was returned. Auditors need a different question answered: how was the decision made, and what version of the model decided it.

Auditors expect four event categories in any AI system: - System behavior: inference traces, tool calls, retrieval operations - Configuration changes: model swaps, prompt template edits, embedding model updates - Human actions: approvals, overrides, redline reviews - Quality measurements: eval scores, drift signals, bias checks

Not every logged inference qualifies as audit evidence. A trace capturing only the final output leaves compliance questions unanswered. Auditors want to see the chain of inputs that produced the output, not the output alone.

Gateway logs typically miss several things. They miss the model version hash used for this specific call. They miss the prompt template version that was active.

They miss the retrieval chunks that fed the context window. They miss the sequence of tool calls, the guardrail decisions at each step, and any human override events.

These gaps aren't logging bugs. They're structural. The gateway sits outside the model and the agent loop. It cannot see what it doesn't instrument.

Teams running ai training pipelines or model training workflows with versioned checkpoints feel this gap most. The absence of a per-inference model version hash is especially damaging to audit claims. Auditors cannot tie a decision to a specific model artifact if the gateway only records "gpt-4-class."

The obvious fix (turn on verbose logging at the inference layer) sounds simple. It breaks the system in ways most teams don't anticipate.

The instinct is to add logging callbacks inside the inference path. Every token, every attention weight, every KV-cache update gets written to disk synchronously. This works in a notebook. It collapses in production.

Synchronous logging inside the inference path adds latency that compounds at scale. A synchronous log write per token adds overhead that is negligible at low throughput.

At high request concurrency and token generation rates across multiple GPU devices, that overhead dominates inference time. The model spends more time writing logs than computing attention.

Beyond latency, naive capture creates storage explosions. KV-cache state, attention weight matrices, and embedding deltas per token per request generate volumes that no retention policy survives. The intermediate state captured during inference is typically orders of magnitude larger than the prompt and response text combined.

The fix is counterintuitive: observe from outside the hot path, not from inside it. DMI-Lib research shows a device-host memory staging mechanism with a dedicated asynchronous observation substrate can capture model internals at under 7% overhead.

The observation happens on a separate memory channel, decoupled from the inference compute path. The model never waits for the logger.

Observation architecture can be designed for longevity without performance decay when the substrate is decoupled from the hot path.

So the answer isn't "log more." It's "log what matters, at the right layer, with the right mechanism."

A usable audit trail decomposes into three distinct layers, each answering a different question:

Layer 1 - Model internals. Attention patterns, KV-cache snapshots at decision points, embedding shifts. Captured via async observation substrate, never via in-model callbacks. This layer answers: what did the transformer actually compute?

Layer 2 - Decision provenance. Prompt template version, retrieval chunks and their source documents, tool invocations, function-call chains, and their sequence. This layer answers: what context and actions produced this output?

Layer 3 - Governance metadata. Approver identity, guardrail triggers and their verdicts, redaction events, human override decisions, waiver references, risk classification. This layer answers: who authorized this, and under what policy?

The three layers compose into end-to-end cryptographic evidence. LiteLLM proxy signing handles the call layer (what was asked). Agent-level signing handles the action layer (what was done). Together they create a chain from prompt to side effect with no gaps.

The reason this decomposition matters: auditors don't ask one question. They ask dozens, organized by category. A single monolithic log doesn't answer them. Three semantically distinct layers, each with its own schema and signing strategy, do.

This three-layer decomposition is the minimum viable audit surface. Teams that try to collapse it into one log end up rebuilding the layers under auditor pressure. That rebuild is the most expensive time to discover what should have been designed from the start.

Tamper-evidence is a property of the data structure, not the access controls. Access controls can be bypassed. A hash chain cannot be silently modified.

The core mechanism:

Each entry links to the previous via `prev_hash`. Breaking the chain at any point reveals tampering. This is the same structure blockchain ledgers use, simplified for a single-writer audit log.

Add ML-DSA-65 post-quantum signatures on each proxy request for non-repudiation. The signing key is held in an HSM or KMS; the signature is attached to the entry. Auditors verify the chain and the signatures independently.

Build as serving middleware, not as model-internal callbacks. This keeps existing ml training and fine-tuning workflows unchanged. The model code stays clean.

The observability layer sits between the model server and the calling agent. It captures what flows in and out without modifying the model.

Deployment matters. Running this on kubernetes means the audit middleware deploys as a sidecar. It can also run as a separate service in the same pod, with its own persistent volume for the chain.

Retention policy should be framework-driven, not guessed: - GDPR: minimize, but audit-relevant data may require longer retention with justification - HIPAA: 6 years for healthcare-relevant AI decisions - SOC 2: audit trail retention aligned to the trust service criteria period - RBI: aligns with model lifecycle documentation requirements

Map each framework explicitly rather than defaulting to a single window. The window you choose is itself an audit decision auditors will review.

This architecture works for global frameworks, but India's regulatory landscape adds a specific wrinkle that most generic audit-trail guides ignore. The RBI demands evidence that gateway dashboards structurally cannot provide.

RBI's AI/ML model risk management guidelines require documented evidence of model behavior across the lifecycle, not just at the API boundary. Gateway dashboards with aggregate metrics don't satisfy this.

Indian regulators expect explainability artifacts tied to specific decisions. A loan rejection needs to show which model version ran, which features were considered, and which rules triggered the decision. Aggregate fairness scores across a population don't answer this; per-decision provenance does.

Bias and fairness monitoring must happen at inference time, against the actual population being served. Pre-deployment evaluation on a test set is necessary but insufficient.

The served population shifts. The audit trail must capture the shift and the model's response to it.

For rag pipelines, RBI scrutiny extends to retrieval sources and embedding model versions used per decision. If your embedding model changed last Thursday and a customer dispute references a decision from last Friday, the trail must show which embedding model was active. Gateway logs won't show this. The cost of reconstructing it after the fact is far higher than capturing it at inference time.

The fix is always the same: instrument inside the inference path, not at the boundary.

Done right, the benefits compound faster than the compliance effort.

The shift is operational, not just regulatory. When the audit trail is real: - Incident response changes. It shifts from "reconstruct what happened from partial logs" to "query the audit trail directly." The result: hours instead of days. The attention patterns that produced a bad output are queryable, not lost. - Model iteration speeds up. Behavior becomes a first-class engineering signal, not just a compliance artifact. Teams debug model regressions with the same tools they debug application crashes. - Regulatory audits become evidence retrieval, not evidence reconstruction. The auditor's question is a database query, not a multi-week investigation. - The audit trail itself becomes an asset for drift detection, customer trust, and model debugging. It stops being a cost center that gets archived and forgotten.

Security posture improves in parallel. When iam roles are tied to audit trail access, you can see who queried what, when, and why. The audit trail audits the auditors.

The choice isn't between compliance and performance. It was never either. It's between audit architecture designed for the decision layer, or another failed review and a scramble to retrofit.

What is an LLM inference audit trail?

An LLM inference audit trail is a tamper-evident, chronologically ordered record of every event that influenced a model's output during inference. It includes model version, prompt template, retrieval context, tool calls, guardrail decisions, and human overrides. Unlike gateway logs that capture the request envelope, an inference audit trail captures the decision process itself. This makes it usable as compliance evidence under frameworks like the EU AI Act and RBI guidelines.

How is inference logging different from LLM gateway logging?

Gateway logging captures the request-response pair, token counts, and latency at the API boundary. Inference logging captures what happened inside the model. It captures attention patterns, KV-cache state, retrieval sources, function-call chains, and the governance metadata attached to each decision. Gateway logs answer "what was asked." Inference logs answer "how was the decision made." For audit purposes, only the latter qualifies as evidence.

Does inference observability slow down model performance?

Naive synchronous logging inside the inference path degrades performance. However, techniques like DMI-Lib's asynchronous observation substrate and middleware-based instrumentation can capture model internals at under 7% overhead. They observe from outside the hot path. The performance cost is an architecture choice, not an inherent property of inference observability.

What does the EU AI Act require for LLM audit trails?

EU AI Act Article 12 mandates automatic event logging that captures every algorithmically driven decision, ensuring traceability and accountability from deployment onward.

Research and references cited in this article:

• AI Agent Governance: Policy and Compliance 2026 Guide
• Audit Evidence From LLM Traces (July 2026) - Openlayer
• AI Audit Trail Requirements: A 2026 Checklist for Finance, Healthcare ...
• AI Audit Trail Requirements by Regulation: What Each Regime Actually Asks For | DeepInspect
• AI Audit Trail: Compliance, Accountability & Evidence | Swept AI
• Enabling Performant and Flexible Model-Internal Observability for LLM Inference _(academic)_
• What is LLM Observability ? Complete Guide
• Master LLM Observability for Peak AI Performance & Security
• LLM Observability: Tutorial & Best Practices | LaunchDarkly
• LLM Observability Explained: Prevent Hallucinations, Manage Drift, Control Costs | Splunk
• 8 Real-World Responsible AI Examples + Best Practices in 2026
• Explainable AI in Finance: What Regulators Actually Require in 2026