Skip to main content

Testing your policies

A policy is the only thing between an agent and a real action. Unit-test it.

from watchlight import govern

govern.load("watchlight.policy.json")
assert govern.has_policies, "no policies loaded — every call would be denied"

report = govern.test([
    {"name": "under limit allows", "action": "book",
     "context": {"amount": 200, "limit": 500, "refundable": True}, "expect": "Allow"},
    {"name": "over limit denies", "action": "book",
     "context": {"amount": 800, "limit": 500, "refundable": True}, "expect": "Deny"},
    {"name": "big wire needs a human", "action": "wire",
     "context": {"amount": 5000}, "expect": "NeedsApproval"},
])
assert report["failed"] == 0, report

Each fixture asserts the verdict for a (principal, action, resource, context). A wrong expectation fails the suite; a fixture missing action or expect raises.

const report = await govern.test([
  { name: "over limit denies", action: "book",
    context: { amount: 800, limit: 500, refundable: true }, expect: "Deny" },
]);
if (report.failed) throw new Error(`${report.failed} policy tests failed`);

Those fixtures exercise these policies:

watchlight.policy.json
[
  { "name": "funded-book",
    "code": "permit(principal, action == Action::\"book\", resource) when { context.amount <= context.limit && context.refundable };" },
  { "name": "big-wire-needs-human",
    "code": "@enforcement_effect(\"require_approval\")\npermit(principal, action == Action::\"wire\", resource) when { context.amount > 1000 };" }
]

govern.test(...) drives the engine's decision core directly, so it holds no decision logic of its own and writes nothing to the audit trail. A thousand fixtures in CI leave no residue in .watchlight/.

Each result is { name, expected, actual, ok, reason } — reason carries the engine's explanation, so an unexpected Deny tells you why.

Two more fixture keys​

Approving a hold​

"approved": true mints a single-use token and asserts the NeedsApproval → Allow downgrade, so you can prove both halves of an approval gate:

{"name": "the wire proceeds once approved", "action": "wire",
 "context": {"amount": 5000}, "approved": True, "expect": "Allow"}

Asserting obligations​

"obligations" asserts the obligations an Allow carries:

{"name": "readable with the SSN redacted", "action": "read",
 "context": {"record_type": "customer"},
 "expect": "Allow", "obligations": {"redact": ["ssn"]}}

The match is exact — a key you leave out must be absent, and {} asserts no obligations at all. Suite files accept both spellings (maxItems / max_items), so one JSON file runs under both CLIs.

Run it in CI​

Put policies and fixtures in one suite.json — { policyFile?, policies?, tests: [...] } — and run it:

watchlight policy test suite.json                                 # Python
npx --package @watchlight/sdk watchlight policy test suite.json   # Node
watchlight policy test — suite.json

  ✓ under limit allows → Allow
  ✓ over limit denies → Deny

2 passed, 0 failed (2 total)

The CLI exits 0 when every fixture passes, and 1 on a verdict mismatch. It exits 2 on a malformed suite: bad JSON, a missing file, a fixture missing action or expect, or an unimplemented @enforcement_effect.

.github/workflows/policy.yml
- run: pip install watchlight        # or: npm i @watchlight/sdk
- run: watchlight policy test policy/suite.json

Add a fixture for the behaviour you are changing and for the behaviour you are not, so a fix cannot quietly widen something else.

Worth knowing​

  • An unrecognised @enforcement_effect fails at load. Anything outside attenuate, escalate, observe, quarantine, require_approval, revoke, sever_subtree, terminate raises PolicyError, and load adds nothing from that file. A typo in the annotation name only warns.
  • govern.load(path) is idempotent per source, keyed on the resolved path, not on content: edit a loaded file and pass force=True to load it again.
  • govern.allow(code) is always additive. The same code twice is two policies.
  • A test that calls authorize() or a governed tool is a governed call like any other and appends a record. Set WATCHLIGHT_AUDIT_FILE=0 for the test run to keep those verdicts out of the application's trail.

See also​