Runbook¶

Operational procedures for the ClyCites kernel. Written to be followed by somebody who did not build it, at three in the morning, under pressure.

References to sections are to the Data Protection and Privacy Act, 2019 (Act 9 of 2019).

The contact rows in §1 are empty. That is a real gap, not a template placeholder. Fill them before they are needed, because the one thing §1 does not permit is time spent looking up who to call.

1. Data breach — s.23¶

Trigger: any belief that personal data has been accessed, altered, disclosed, or destroyed without authorisation. Belief, not proof. A suspicious access pattern, a leaked credential, a lost laptop with a database password on it, an unexplained gap in the audit log.

s.23(1) requires notification to the Personal Data Protection Office immediately. Not within seventy-two hours. Not once the investigation concludes. Immediately, on belief.

Do these concurrently, not in sequence¶

Containment does not wait for notification, and notification does not wait for containment. Two people, two tracks. If there is only one person, notify first — it is the shorter task.

Track	Action
Contain	Rotate the credentials that could have been used (§3). Revoke sessions. Block the source if it is identifiable.
Notify	Contact the Authority. See below.
Preserve	Snapshot logs and the audit schema now, before rotation destroys the evidence.

Do not delete anything¶

s.36 makes unlawful destruction of personal data a criminal offence. The append-only log is the evidence of what happened and when. Deleting affected records to "limit exposure" converts a data breach into a second offence and destroys the only account of the first.

The log cannot be deleted from anyway — decision 0001 removed DELETE from the application role — but nothing stops somebody dropping a schema as the owner. Do not.

The Authority decides whether data subjects are notified¶

s.23(2). Not us. Prepare the notification text so it can go out the moment it is authorised, and do not send it unilaterally. Notifying subjects before the Authority has ruled can prejudice an investigation and is not within our discretion to choose.

Contacts¶

Role	Name	Phone	Email
Incident owner (decides, and is accountable)
Deputy (when the owner is unreachable)
Personal Data Protection Office
Legal counsel
Infrastructure / hosting provider

Record it¶

Every breach, and every belief of one that turned out to be nothing, gets an entry in §7. Under s.33(2) a documented, dated record of how we responded forms part of a reasonable-care defence. An undocumented correct response is worth less than a documented imperfect one.

2. Restore from backup¶

Verify a backup without touching production¶

This is the drill. Run it monthly (§6). It never touches the live database.

export BACKUP_PASSPHRASE=...        # from the secret store, never from a file
scripts/restore.sh backups/<stamp> \
  postgres://clycites_owner:...@localhost:5433/clycites_restore

Exit 0 means every row count, every primary-key fingerprint, every constraint (including its validated flag), and every table grant matches the database that was dumped. Anything else prints a diff and exits non-zero.

restore.sh drops and recreates its target, and refuses any database whose name does not contain restore, test, or scratch.

Restore production¶

Stop the kernel. Writes during a restore are lost silently and there is no reconciliation path afterwards.
Restore into a scratch database first and confirm it verifies. Do not restore straight over production — a corrupt backup then leaves you with nothing.
Rename or point the connection at the verified scratch database.
Run pnpm migrate. It is idempotent and will report already applied.
Start the kernel and check /ready, then /metrics.
Note the actual RPO and RTO in §7 while it is fresh.

What the verification does not cover¶

The manifest is one we wrote ourselves. It proves the restored database matches the dump; it cannot prove the dump matched reality at the moment it was taken.

Once anchoring ships (work order I), completeness becomes checkable against published Merkle roots, which is independently strong because we do not control them. That is the intended upgrade to this section.

Recovery objectives¶

	Target	Last measured	Date
RPO — data we accept losing
RTO — time to serving again

Measure these during the monthly drill. A target with no measurement beside it is a hope.

3. Rotate a secret¶

Rotate on any breach belief, on any staff departure with access, and on any credential that has ever existed in a file, a terminal history, or a chat.

`kernel_app` password¶

Change it in the secret store.
ALTER ROLE kernel_app WITH PASSWORD '<new>' as the owner.
Update KERNEL_APP_PASSWORD and DATABASE_URL in the deployment environment.
Restart the kernel. Confirm /ready returns 200 — it fails when Postgres is unreachable, which is exactly what a half-rotated password looks like.

`clycites_owner` password¶

Same, plus MIGRATOR_DATABASE_URL, plus anywhere scripts/backup.sh runs.

`BACKUP_PASSPHRASE`¶

Backups taken under the old passphrase stay readable only with the old passphrase. Keep it until every backup encrypted with it has aged out of retention, and record the date it becomes safe to destroy. Losing the old passphrase while old backups are still the only copies means losing the data.

After rotating, take a fresh backup and verify it (§2) before relying on it.

After any rotation¶

Run bash scripts/check-secrets.sh. It scans everything git tracks.

4. Roll back a migration¶

There is no down-migration, by design. Decision 0001 makes the log append-only; a mechanism able to undo a schema change would also be a mechanism able to reach the data behind it.

A schema change to undo: write a new forward migration that undoes it. Migrations are immutable once applied — the migrator refuses a file whose checksum changed — so editing the original is not an option and will fail loudly if attempted.
A migration that failed mid-run: it ran in its own transaction and rolled back. The database is on the last complete migration. Fix the file and re-run.
Record data that is wrong: this is not a deployment problem. Supersede or retract through the API, which is what those record types exist for. If the data is wrong because of a defect, it is an incident (§1), not a rollback.

5. Deploy¶

pnpm typecheck && pnpm lint && pnpm test — all three, no exceptions.
pnpm openapi && git diff --exit-code apps/kernel/openapi.json. A drifted contract means applications are coding against a document that is no longer true.
Take a backup and verify it (§2) before running migrations.
pnpm migrate.
Deploy, then check /ready and /metrics.

Watch after a deploy:

Metric	What a change means
`kernel_assumed_conversion_share`	Tonnage resting on an unverified conversion factor
`kernel_thin_sample_kg_total`	Tonnage on a `measured` factor with a thin sample (0018)
`kernel_records_by_lawful_basis{basis="unstated"}`	Should only ever be pre-0013 rows. Growth means a write path is bypassing the basis check
`kernel_financial_records_without_special_consent`	Should be zero for anything written after 0013

6. Monthly verification — s.20(2)¶

s.20(2) requires regularly verifying that safeguards are effectively implemented. A one-off hardening pass does not satisfy it. This table, dated and signed, is what does — and under s.33(2) it forms part of a reasonable-care defence.

Run on the first working day of each month.

#	Check	How
1	A backup from the last 24h exists	`ls backups/`
2	That backup restores and verifies	§2, into a scratch database
3	RPO and RTO measured and recorded	§2
4	No credential in the tree	`bash scripts/check-secrets.sh`
5	Config guards pass	`pnpm test` — `test/ops/`
6	No data-tier port reachable from off-host	`nmap` or `nc -vz` from another machine
7	OS security updates applied	`unattended-upgrades` log
8	Access list still correct	Who holds DB, host, and secret-store access — remove departures
9	§1 contacts still correct	Ring one of them
10	Audit log shipping off-box	`AUDIT_SHIP_URL` is set, and the collector holds entries dated within the last 24h
11	Audit log is still write-only to the app	`select privilege_type from information_schema.role_table_grants where grantee='kernel_app' and table_schema='audit'` returns INSERT and nothing else
12	No unexplained DDL	`select occurred_at, reason, detail->>'object', detail->>'role' from audit.entry where action='schema.ddl' and occurred_at > now() - interval '1 month'` — every row should match a deploy in §5
13	Denial rate has not shifted	`select date_trunc('day', occurred_at) d, reason, count(*) from audit.entry where outcome='denied' group by 1,2 order by 1` — a change is a signal, not noise

Verification log¶

Date	Who	1	2	3	4	5	6	7	8	9	10	11	12	13	Notes

7. Anchoring has stopped¶

Alert: kernel_anchor_stale == 1, or scripts/anchor-cron.sh check exiting non-zero.

Not live yet. No topic exists and no root has been published, so this section does not currently fire — it describes the state to run in once ANCHOR_TOPIC_ID (or ANCHOR_PUBLISH_URL) is set. Until then, tamper evidence is not a property this deployment has, and must not be described as one to a lender, a regulator or a farmer. See 0038.

What it means¶

No Merkle root has been published in three days on a kernel configured to publish one. Every record written since the last root has no public commitment, so nothing about it can be shown to predate today. The data is intact; the evidence that it is intact is not being produced.

This is the only failure in the kernel that is silent by construction. Everything else breaks in front of somebody — a refused write is a 4xx a caller sees, a broken read is a support ticket. A batch that stops running produces nothing at all, and the absence is normally noticed the first time someone asks for a proof, which is exactly the moment it cannot be repaired. That is why the alert is wired to the absence of an expected root and not to a job's exit code: a scheduler that was uninstalled leaves no failed run behind it.

Diagnose¶

scripts/anchor-cron.sh check

Read the line it prints:

Field	Meaning
`configured=false`	No topic. Not a fault — the alert is gated on this and will not fire. If you expected a topic, the environment is wrong, not the job.
`last_root=none`	A topic is set and nothing has ever been published. The first run has never succeeded. Check credentials before anything else.
`age_days` large	Anchoring ran once and stopped. Check the crontab first: `crontab -l \\| grep anchor`.
`pending` > 0	Batches were built but never reached the topic. Credentials, network, or an operator account out of hbar.
`failed` > 0	Publication exhausted its retries. Read `last_error` on `kernel.anchor_batch`.
`unanchored_age_days` large with a fresh `last_root`	The quieter failure. The batch is running and covering almost nothing. Do not close this as healthy because a root exists.

Repair¶

Missed days are not lost. Each day is its own batch keyed on batch_date, and a day already anchored is a no-op, so the fix is to run the missing days:

cd apps/kernel
for d in 2026-07-30 2026-07-31 2026-08-01; do
  ./node_modules/.bin/tsx src/anchoring/anchor-cli.ts --date "$d"
done

Then confirm:

scripts/anchor-cron.sh check

Roots published late are still roots. What was lost is the tightness of the bound — a record from 30 July anchored on 5 August is proved to predate 5 August, not 31 July. Say so plainly if anyone asks; do not backdate a batch to close the gap, and note the window in the incident log below.

Do not¶

Do not widen ANCHOR_STALE_AFTER_DAYS to silence the alert. It is in anchor.service.ts and not in configuration precisely so that the person who can quiet the alarm has to change the code and be seen doing it.
Do not delete a failed batch to make the metric read zero. The leaves are the record of what was in that day's tree.

8. Incident log¶

Every incident, every drill, every restore. Append only — correct an entry by adding another, which is the same discipline the log itself keeps.

Date	What	Data subjects affected	Authority notified	RPO/RTO observed	Outcome

9. Known gaps¶

Recorded here rather than left implicit, because a gap somebody wrote down is a decision and a gap nobody wrote down is a surprise.

Gap	Blocked on
§1 contact rows are empty	Nothing. Fill them.
Answering a s.24(1)(c) request has no tooling	Nothing. The data and the grants are right (0025); a person runs the query as the owner.
Un-shipped audit entries are lost if the process dies	A shipper outside this process, which needs its own role. See 0025 finding 1.
Retention expiry and destruction, s.18(4)	The 2021 Regulations, and counsel
Cross-border transfer position	Counsel
External penetration test	Budget. Before the first real farmer record.
Restore completeness verified independently	Work order I (anchoring)
Seed plot area, crop mix and delivery frequency are invented	The UNPS microdata, which is behind a manually approved login. See `docs/data-sources.md`.
`UG.KIRYANDONGO` and `UG.NEBBI` are not in `registry.admin_region`	Nothing. The seed uses both district codes anyway; the season calendar could not be scoped to them.
Three observation types carry no citation	A field request for any of them, or their eventual withdrawal. See 0028.
A farmer cannot read their own Planting, Harvest or Observation	Closed. Those types reach a party through one declared hop — `plot` for Planting and Harvest, `subject_ref` for Observation. Where the hop resolves to nobody the read is denied as `subject_unresolvable`. See 0029.
A cooperative can classify as member body on records it is not a party to	Nothing, but know both bounds: every party the record resolves to must be its member at `occurred_at`, and it must have asserted the record or hold the delegation it was written under. A harvest a farmer recorded independently stays third party. See 0029.
Whether s.9(3)(c) reaches a member's data unrelated to the body's own dealings	Counsel, with the s.9 flag question. The nexus condition assumes not, which is the conservative reading. If counsel says the provision is broader, the nexus limb comes out of `classify`. See 0029.
Erasure under s.16 and s.18 is not implemented	Counsel, open decision D3. An objection is not erasure and the response says so in as many words, so nobody is told their records are gone when they are not.
A subject-access answer is capped at 1000 records and reports `truncated`	An export path for subjects above the cap. Nobody has one today, and the flag is the only thing keeping the shortfall visible. See 0031.
`audit.disclosures_to` is the application's only read of the audit log	Nothing, but it is the thing to look at first if the audit grants are ever revisited. A compromised app can call it per subject; it cannot read the log. See 0031.
A party whose `kind` is unrecorded is redacted as a person	Party records for every counterparty. Over-redaction is recoverable, under-redaction is a disclosure, so the default is set that way. See 0031.
Parties who already received data are not notified when an objection is lodged	Nothing here. J4 landed notification for corrections and retractions, which is what s.16(4) reaches; an objection is neither. A party who read a record before the subject objected is not told, and the objection response says the objection stops future disclosure only. See 0032.
Nothing sends a disclosure notification	An adapter, outside the kernel. J4 raises the obligation and counts it; `kernel_disclosure_notifications_outstanding` will sit above zero and climb until something discharges it, which is the intended reading. See 0032.
`audit.recipients_of` is the second application read of the audit log	Nothing, but it is now two functions rather than one that would have to be reviewed if the audit grants are revisited. Neither can read the log; each is scoped to one subject or one record. See 0032.
A party reached only through a hop may be notified about a correction they are close to	Nothing. The exclusion list is what the two records name directly, because a spurious notification is noise and a missing one is a breach. See 0032.
`S9_CONSENT_REQUIRED_FOR_MEMBER_BODY` is set true on a guess	Counsel. One question: does s.9(3)(c) permit a cooperative to process its members' financial information without separate explicit consent? The answer sets one flag. See 0029.
Aggregate and anonymised access has no rule at all	Counsel, and a de-identification standard. Tangled with s.37 on selling personal data. The next consent question, not part of 0029.