Case UW-G08. A warehouse in wind zone C with a roof of 28 years. The agent calls find_similar_decisions and finds a close match: a similar building, same zone, roof of 28 years, recommended for approval and approved by an underwriter last year. The persuadable stand-in does the natural thing and writes approve, citing the precedent.
The precedent was real. It was also decided under the policy fixture's earlier version, 2025.4, when the roof limit of R-ROOF-01 was 30 years. On 2026-07-01 the insurer moved to version 2026.2, with a limit of 25 years. Under the current rules, 28 years in zone C is not eligible.
Nothing about the run is malicious. The model found true evidence and reasoned from it. The problem is that the rule changed, and the model has no reliable way to know which version of a rule a piece of evidence was decided under, or which version applies today. Code does, if the rules are code and the version is a value.
1. A rule that must hold is code#
Policy as code means writing the rules a system must follow as code or data that a program evaluates, each rule with an id and each set of rules with a version. Example: R-ROOF-01 is a row in policy_fixture.json with max_roof_age_years: 25 and wind_zone: "C", evaluated by check_eligibility in policy.py.
The policy fixture is this site's name for that rule set: UW-GUIDE, versions 2026.2 (current, effective 2026-07-01) and 2025.4 (earlier, effective 2025-01-01). It is illustrative: invented for teaching, not insurance practice, and it belongs to no real insurer.
Writing rules as code gives three properties that prose cannot. The same facts and version give the same decision every time (deterministic). Every decision names the rule ids it applied, so it can be cited and audited. And the rules can be tested like code: a table of facts and expected decisions, run on every change.
Here is the current version of the roof rule, as data:
{"R-ROOF-01": {"text": "Roof older than 25 years in wind zone C: not eligible.",
"max_roof_age_years": 25, "wind_zone": "C"}}
The text is for people. The numbers are for the code. A change to the limit is a change to one value, reviewed like any change to code.
2. The policy decision point#
A policy decision point is the one component that takes facts and returns a decision under the rules; everything else asks it instead of deciding itself. Example: check_eligibility(submission, version) in policy.py.
The term comes from access-control architectures, where a decision point decides and enforcement points (the places that act on the decision) enforce. This site's guards are the enforcement points. G-OUT-POLICY, the policy gate, asks the decision point and compares its answer with the model's. The decision point never sees the model's text. It sees typed facts: the roof age from the application and from the provider, the wind zone, the occupancy, the area, the sprinklers.
On UW-G08, the two versions give two answers:
check_eligibility under 2026.2 -> decline (R-ROOF-01)
check_eligibility under 2025.4 -> approve
That one line of difference is the whole case. The run is pinned to 2026.2, so the policy gate denies the model's approve:
44 DENY G-OUT-POLICY the candidate says approve; UW-GUIDE 2026.2 gives decline (R-ROOF-01)
45 settled what will be stored: decline (policy_fallback)
A historical approval is evidence about the past. It is not a reason today, and the gate does not need to argue the point: it compares one word with one rule result under one version.
Diagram: Deciding apart from enforcing · the decision point sees facts, the gate compares
Read left to right. The run is pinned to one version. Typed facts go to the decision point, which returns a decision with rule ids. The model's recommendation goes to the policy gate, which compares the two words. The decision point never reads the model's text, and every decision is logged with its version.
- Inputs
- Typed facts: Structured values from the submission and the provider; no free text.
- Pinned version: Chosen once at the start; carried by every audit event.
- Decides
- Decision point: Rules as data, evaluated by one function or a policy engine.
- The model's word
- Candidate: What the model recommends, possibly citing a precedent.
- Enforces
- Policy gate: Compares the two words; escalate is always allowed.
- Records
- Decision log: Facts, version, rule ids and result, joined to the run.
3. Versions, and pinning a run#
A policy version is a named, dated state of the whole rule set. Example: 2026.2, effective 2026-07-01.
Versions answer questions that otherwise have no answer. Which rules applied to this recommendation? The one in its policy_version field. Why did last year's case get approve? Because it was decided under 2025.4. Can we reproduce today's decision next year? Yes, by running the same facts under 2026.2.
Pinning means a run picks its version once, at the start, and every check in the run uses that version. Every audit event carries it. The output schema check refuses a recommendation whose policy_version is not the pinned one. Without pinning, a rule change in the middle of a run could make two checks of the same run disagree, and nobody could say afterwards which rules governed the recommendation.
The example's command line can run a case under another version, which is how you test what a change would do before it takes effect. UW-G08 under 2025.4 stores approve and goes to an ordinary underwriter; under 2026.2 it stores decline and goes to a senior underwriter. That comparison is the change review for the new limit.
Before AI. This is how pricing and tax rules are handled in a well-run backend: a rate table with an effective date, an invoice that records which table version priced it, and a recalculation that uses the recorded version, never today's. Nobody lets a salesperson's note about last year's price override the table.
4. A prompt instruction against the gate#
The roof rule could also be written in the system prompt: "Do not recommend approval for a roof older than 25 years in wind zone C." Four words showed why that is an instruction, not a guardrail. The lab makes the difference measurable inside the example.
Set the roof age, the zone, the version and whether the broker asks for an exception. The left side runs a vague prompt instruction over 20 scripted trials of the persuadable stand-in; the right side runs the policy gate once, because once is enough for a deterministic check.
| Preset | Policy gate | Vague instruction: trials that agree | Unsafe approvals | Needless declines |
|---|---|---|---|---|
UW-G02: 31 years, zone C, exception asked | decline | 8 of 20 | 12 | 0 |
UW-G08: 28 years, zone C | decline | 10 of 20 | 10 | 0 |
UW-G08 under 2025.4 | approve | 10 of 20 | 0 | 10 |
UW-G01: 8 years, zone A | approve | 20 of 20 | 0 | 0 |
| 30 years, zone B | approve | 16 of 20 | 0 | 4 |
The agent here is a scripted stand-in. The obedient variant is fooled every time by construction, so these results show how each control works, not how often a real model is fooled or how good any model is. The trial counts above come from the scripted sloppy variant, whose persuadability is fixed by construction; they are illustrative and measure nothing about a real model.
Read the third row closely. Under the old version, the vague instruction produces needless declines: the model is told "old roofs are bad" in words and errs in the safe direction. A sentence in a prompt has no version. It is wrong in both directions as soon as the rule changes, and nobody can tell from the outputs which version the model believed.
5. A policy engine#
As rules multiply and several services must apply them, teams move the decision point out of the application into a policy engine, a separate component that evaluates rules written in its own language. Open Policy Agent (OPA) is a widely used open-source one. Its documentation describes the shape: the calling service sends structured input; OPA evaluates it against policies written in Rego, OPA's policy language, and against data loaded beforehand; it returns a decision that can be any structured data, not only allow or deny. With decision logging on, each answer carries a decision_id, and OPA logs the input, the result and the revision of the policy bundle.
The roof rule in Rego, reading its limits from data so that a version change is a data change, is on Frameworks as documented shape, not run here. The same page names a trap worth repeating: when no rule matches, OPA's HTTP answer has no result key and still returns status 200, so a caller that treats a missing result as allow fails open. Write a default in the policy and treat a missing result as deny in the caller.
When is an engine worth it? When several services must apply the same rules, when the rule owners are not the people who deploy the agent, or when an auditor needs decision logs from a component the agent team does not control. For one agent with five rules, a function in the same process, like check_eligibility, gives the same answer with no extra process and no network hop. The idea that matters is the separation of deciding from enforcing, and both shapes keep it.
6. The decision log#
A decision log is the record of each policy decision: the input, the rules or version applied, the result, and an id that joins it to the request. Example: the audit event of G-OUT-POLICY in UW-G08 records the run id, the trace id, policy_version: 2026.2, the decision deny and the reason with R-ROOF-01.
The log is what makes the rules accountable. When an underwriter asks why a recommendation was decline, the log shows the facts, the version and the rule. When the rule owner asks how many submissions the new limit affected, the log can be counted. When an auditor asks whether the agent ever stored a recommendation against the rules, the policy gate's denials and the stored rows answer it. Production is the home of the audit record as a whole.
7. Change control for rules#
If rules are code, rule changes are code changes, and they get what code changes get: a review, a test, an owner's approval and a recorded release.
| Step | In this example | Who acts |
|---|---|---|
| Propose | A new version, 2026.2, with max_roof_age_years: 25 | The rule owner (the head of underwriting) |
| Test | Every case run under the old and new version; differences listed | The platform team |
| Approve | The owner signs off the differences, not only the text | The rule owner |
| Release | The version gets an effective date; new runs pin it | Automated, at the date |
| Keep | The old version stays loadable for audits and replays | Forever, or per retention |
This is where governance touches code: governance decides who owns R-ROOF-01 and who may change it; policy as code makes that decision enforceable, because a change that skipped the owner cannot be released.
What this is not. Policy as code here means the insurer's business rules, evaluated deterministically. It is not the instructions in the system prompt, which belong to Prompt Engineering, and it is not a model that judges compliance, which is a model-based check and the subject of the next page.
Diagram: Changing a rule · from 2025.4 to 2026.2 under its owner
Read left to right. The rule owner proposes a new version. The platform team runs every case under both versions and lists the differences. The owner approves the differences, not only the text. The new version gets an effective date and new runs pin it. The old version stays loadable for replays and audits.
- Propose
- New version: A new rule file with a changed value.
- Test
- Run both: Every case under the old and the new version; differences listed.
- Approve
- Sign off: The owner approves the changed decisions.
- Release
- Effective date: New runs use the new version from the date.
- Keep
- Old version kept: Past decisions can be reproduced exactly.
8. The same idea for a coding agent#
A repository's merge rules are policy as code too: branch protection, required reviews and required checks are configuration with an owner, evaluated by the server on every push, and versioned in the repository's settings. A coding agent's instruction file may say "always run the tests"; the required check is what refuses the merge when they fail. Diff to Deploy covers those gates in delivery.
For the Principal engineer: facts, precedence and replay
The decision point is only as good as its facts. check_eligibility is deterministic over the facts it is given. If the agent supplies the facts (for example, a roof age it extracted from free text), the model is back in the decision path. In the example, the facts come from typed fields and from the provider's typed roof_age_years, never from the description. Make the provenance of every fact the decision point reads explicit, and keep model-extracted facts out of it, or mark them and route to a person.
Missing facts must have a rule. R-ROOF-02 exists because "unknown" is a common state. A rule set without an explicit rule for missing inputs falls through to its default, which is often "approve" by accident. Test every rule with each of its inputs missing.
Precedence between rules is policy too. check_eligibility evaluates in a fixed order: provider unavailable, then unknown age, then conflicting ages, then the roof limit, then sprinklers. That order is a decision (a conflict between two ages beats a roof limit computed from one of them), and it should be written down and tested, not left to the order of if statements. Advanced discusses conflicts between rules from different owners.
Pin, record and replay. Pinning only helps if the version is recorded with every decision and the old version remains loadable. Then any decision can be replayed exactly, which turns "the agent got this wrong" into a reproducible test. Keep the rule files immutable per version; a fix is a new version, never an edit.
Effective dates are about the submission, not the run. Which version applies to a submission received on 2026-06-30 and processed on 2026-07-02? The business must decide (date received, date quoted, date bound), and the pinning logic must implement that decision. The example pins by run for simplicity; a real system should derive the version from the governing date in the record.
Checklist#
| Before you rely on policy as code | Check |
|---|---|
| Every rule that must always hold is code or data, with an id | Not only a sentence in the prompt |
| Rules are grouped into versions with effective dates | 2025.4, 2026.2 |
| Each run is pinned to one version, recorded in every audit event and in the output | The output schema refuses another version |
| One decision point decides; the guards only compare and enforce | The decision point never reads the model's text |
| Facts the decision point reads come from structured, controlled sources | Not extracted from free text by the model |
| Every rule has a test, including missing inputs | R-ROOF-02 for the unknown roof age |
| Rule changes are reviewed by their owner on the list of changed decisions | Old version kept loadable for replay |
| A policy engine's missing answer is treated as deny | No fail-open default in the caller |
In one breath#
A rule that must always hold is written as code or data with an id, grouped into versions with effective dates, and evaluated by one decision point that sees typed facts and never the model's words. Each run is pinned to one version, every decision is logged with that version, and the policy gate compares the model's recommendation with the decision point's answer, so a precedent from 2025.4 cannot carry an approve past 2026.2. Whether the decision point is a function or a policy engine like OPA matters less than keeping deciding apart from enforcing, and rule changes under their owner's review.
Hand-off#
Every check so far has been deterministic: the same input, the same decision. Some things you want to check have no rule you can write: whether a sentence is an instruction in disguise, whether a reply is abusive, whether a claim is supported by a paragraph. For those, the check itself is a model. The next page, Model-based checks, asks how far such a check can be trusted, and where on the path it may stand.