The hardest part of bringing large language models into a regulated financial institution isn’t prompt engineering. It’s proving to Risk, Compliance, and eventually your supervisor that every inference is governed, logged, and reversible. In a CSSF and DORA context, “it usually works” is not an acceptable control.

Put a gateway in front of everything

No application talks to a model provider directly. A central LLM gateway owns authentication, prompt/response logging, PII redaction, and cost attribution per business unit. This gives you one enforcement point instead of dozens of scattered integrations to audit.

gateway:
  auth: entra-id-oauth2
  policies:
    - redact_pii: on
    - max_tokens_per_request: 4096
    - data_residency: eu-only
  audit:
    sink: immutable-append-log
    retention_days: 3650

Make the guardrails deterministic

Probabilistic “please don’t leak secrets” instructions in a system prompt won’t satisfy an auditor. Enforcement belongs in code: hard input/output filters that fail closed. If the PII redactor is unavailable, the request is rejected — never passed through unredacted.

In regulated environments, the model is the easy part. The control plane around it — identity, logging, residency, cost governance — is the real product.

Audit trails that stand on their own

Every request and response is written to an append-only store the application layer cannot modify. When compliance asks “what did the model tell this customer on the 3rd of March”, you produce the exact prompt, response, model version, and policy decisions — not a reconstruction.