AI Watch Tower: how we taught our AI to answer the question “why?”
We built AI Watch Tower — a control plane for enterprise AI systems. One gateway that all LLM traffic passes through, making every call observable, cost-attributed, policy-compliant and explainable. We built it for ourselves first. Today it runs in production and we are deploying it with clients in regulated industries.
TL;DR: We built AI Watch Tower — a control plane for enterprise AI systems. One gateway that all traffic to language models passes through, making every call observable, cost-attributed, policy-compliant and explainable. We built it for ourselves first. Today it runs in production, and we are working with clients in regulated industries.
The question that stops AI deployments
We have been working with large enterprises in Poland and across Europe for many years. Over the last two, almost every GenAI conversation has ended with the same question — asked by the board, the compliance team or an auditor:
“What exactly is our AI doing, how much does it cost, and can we prove it?”
It sounds simple, yet most organisations cannot answer it today. Assistants, agents and RAG systems are entering companies faster than companies can supervise them. In practice it looks like this: the AI invoice is a single line item with no attribution to cost centres. Prompts are changed ad hoc in production, with no trace of who, when or why. Nobody can answer “why did the system respond this way and not another”. Personal data ends up in logs. And when the audit comes — there is no evidence.
In regulated industries — banking, insurance, energy — this is not an inconvenience. It is a blocker. AI projects sit in the deployment queue because nobody can take responsibility for them.
We faced the same problem — with our own AI products. So we built a solution. For ourselves first.
What AI Watch Tower is
AI Watch Tower is a control plane for enterprise AI systems. The principle is simple: all traffic to language models passes through a single gateway. The application changes only the endpoint address and the key — nothing else. From that moment, every LLM call is:
Observable. A complete trace of every interaction: the prompt and its version, agent decisions, tools invoked, data sources from the RAG pipeline (with provenance: where from, which document version, when), cost, tokens, latency. All in one trace — the agent’s reasoning and the model’s generations together, not in separate systems.
Cost-attributed. Every call tagged with a cost centre, system, client — even the end user. An organisational hierarchy with cost roll-ups, allocation rules, monthly budgets with alerts and hard limits that physically cut off traffic once exceeded. Exports in formats the controlling department opens without a translator — including a CSV that opens correctly in a Polish Excel. That sounds like a detail. For a finance team it is the difference between “an engineers’ tool” and a system they use every day.
Policy-enforced. And this is the heart of it. Most tools on the market observe. Watch Tower can enforce. Prompts live in a central registry with versions and a promotion process (draft → staging → production, with required approval). If a prompt outside the registry appears in production, the system — depending on the mode — records the gap for the compliance report, or blocks the call before anything reaches a customer. A policy that works, not a policy that hangs on the intranet.
Explainable. A single endpoint assembles, from one trace, a complete answer to the question “why”: which prompt and in which version, which sources, which tools, which options the system considered and rejected — with reasons. And when an incident happens, it can be replayed: the same input, through the same gateway, optionally with a corrected prompt version — and the results compared side by side. An incident stops being a mystery and becomes a test case.
Security: fail-closed, EU-only
Two design decisions we made at the very beginning and have held to.
First, personal data is masked at the gateway before anything reaches the logs and the observability system — national ID numbers, tax IDs, IBANs, e-mail addresses, phone numbers, names. Crucially: if the masking component fails, the system does not let data through “because tough luck” — it redacts the content (fail-closed). Leaks through logs are one of the most common and most embarrassing AI incident scenarios. In our design it is eliminated structurally: since masking happens before write, every export and every audit artifact by definition contains no personal data.
Second, the whole platform is self-hosted and EU-only. It runs on the client’s infrastructure or ours, models in the Warsaw region, and no data flows to any third-party SaaS. For regulated industries this is not a feature — it is a condition of entry.
We also took the audit literally. Policy events go to an append-only ledger in files readable without special tools, and change records (e.g. promoting a prompt to production) land in an audit trail with a hash chain — each record contains the hash of the previous one, so tampering with history is detectable. The auditor does not have to trust our dashboard. They can read the ledger and verify it themselves.
Quality is measured through the same path as production
Governance without quality measurement is just bureaucracy. That is why Watch Tower has three measurement loops: offline (a golden set of questions and expectations, replayed through the gateway on every change — with a regression gate in CI: quality below the threshold blocks the merge), online (an LLM judge samples real traffic every hour, and low-scored answers automatically enter a human review queue) and traffic signals (error rates and latency spread per prompt version — an “uneven” prompt is visible before anyone scores it).
The key principle: we measure everything through the same path production traffic flows through. A benchmark that takes a different route than production measures a different reality.
What our own benchmark taught us
Frontier models are excellent — we use them daily ourselves, for writing code and running operations. Where a model works for us, let it be the strongest one.
But products follow different mathematics: every call is multiplied by thousands of users. So we ran three models through our own golden set — through the same gateway production flows through. The result: the cheapest and fastest model achieved the highest quality — 3.6/5, zero hallucinations, p95 at 1.2 seconds — beating larger, more expensive models. In chatbots, models of the Gemini 2.5 or Mistral class perform remarkably well.
The conclusion we keep repeating to clients: the question is not “which model is best”, but “which model is best for this task”. Companies that do not measure this overpay every month for big models “just in case”.
What about performance?
A fair question: how much does all this control cost on the path of every request?
We measured it with a methodology that can be defended: a model mock with constant latency, two test arms on the same virtual users — directly to the mock, and through the gateway to the same mock. Model time identical in both arms, so the difference is the pure gateway overhead. The result: ~31 ms median — with full tagging, governance and PII masking on the path. A user has no chance of noticing it.
Production, not a slide deck
Watch Tower runs in production on our DigitalFleet — a fleet-management platform used by real customers. Under the hood there is a multi-agent assistant and a RAG pipeline over documents. One trace links the agent’s decision, tool calls, retrieval with full data provenance and the model’s generations — with costs visible per client and per end user. The prompts of five agents live in the central registry: swapping a version in production is a label change in the registry, no deploy — the application picks up the new version within a minute. Enforce mode is enabled in production.
Onboarding another system onto the platform? Issuing one key with metadata. Zero changes in the application’s orchestration.
The conclusion we did not expect
We started out thinking of Watch Tower as a cost — a security tax you pay so that compliance lets you deploy. We ended up with the opposite.
Control does not slow innovation down. It sets it free. Once the layer of security, accounting and evidence is built, every next AI system enters the organisation faster — because it does not start the risk discussion from zero. When compliance sees evidence instead of promises, it stops saying “no” and starts asking “what’s next?”. Structure stops being a brake and becomes a runway.
Trust in AI is not declared. Trust in AI is measured.
We built AI Watch Tower for ourselves first. Today we work with clients in regulated industries — including finance and energy — and we make this technology available to them as well. If your organisation needs evidence, not promises — let’s talk.