Skip to content

Core operations

Everything on this page is read-only unless it is under Change, Recover or Retire. Read-only actions never write a tag.

Daily read

./lk check status

One command answers: state, reason, earliest credential expiry and the next due item. Then:

./lk check verify      # service probes
./lk check backup      # newest set, age, RPO
./lk check drift       # containers vs declared pins
./lk check incident    # host, container count, command trail
Check Answers
status Is it live, degraded, changing or suspended? What is due next?
verify Do the four services answer?
drift Do running containers match the declared pins?
secrets Are stored values readable, and what expires soon?
access Do the grant file, policy and declared roles agree?
backup Is the newest set complete and inside the RPO?
cost What is the instance consuming?
incident What is the host doing right now?

Command map

I want to Command
Re-check the host before a change ./lk check preflight --tools --facts --host
Read the derived state ./lk check status
View container logs Dozzle: https://<slug>-dozzle.<tailnet_dns>
Review or apply access declarations Edit access/grants.yml, then ./lk change access --reason "..."
Preview an access change ./lk change access --explain
Rotate one secret ./lk change rotate --set <NAME> --reason "..."
Patch the host ./lk change patch --reason "..."
Upgrade one accessory ./lk change upgrade --component <name> --reason "..."
Stop the instance ./lk change suspend --reason "..."
Start it again ./lk change resume --reason "..."
Take a backup now ./lk recover backup --reason "..."
Test the newest backup ./lk recover drill --reason "..."
Restore a set ./lk recover restore --set <id> --reason "..."
Export before leaving ./lk retire export --reason "..."
Tear it down ./lk retire decommission --confirm <slug> --scope all --reason "..."
Hand over ./lk transfer handover

Users and access

access/grants.yml is the only place a grant exists. Each entry maps one principal to a Forgejo team and role, a set of Tailscale Service grants, and an Infisical project role.

Role Tailscale Forgejo Infisical
Owner Tailnet admin; owns the policy Instance admin Organization admin
Operator All four Services + SSH to the host Org owner; repo write on the core Project admin
CI identity Tag-scoped grants only Repo read + evidence:tag Read-only, TTL and max uses
Collaborator Named Service grants Write on named repos None by default
Auditor None Repo read None 
./lk check access                  # declared table
./lk check access --effective      # declared CI scope, not a live Forgejo read
./lk check access --identities     # machine identities and TTLs
./lk change access --explain       # preview
./lk change access --reason "review access"

change access applies the tailnet policy and ensures the Forgejo teams exist. Adding or removing a person's actual Forgejo or Infisical membership is a console step: create the account or seat in Forgejo and Infisical, then record the role in access/grants.yml.

Console-only steps: Tailscale ownership transfer, Forgejo instance admin, and the Infisical organization admin. They are done once and are named in ACCESS.md.

Secrets

Values live in Infisical. Git holds names and coordinates, never values. .kamal/bootstrap.env stays on the laptop at mode 0600 because it must be readable while Infisical is down.

./lk check secrets --expiry
./lk change rotate --set FORGEJO_JWT_SECRET --dry-run
./lk change rotate --set FORGEJO_JWT_SECRET --reason "quarterly rotation"

Rotation order is always create → apply → verify → revoke:

  1. Generate and save the new value.
  2. Write it to Infisical and the bootstrap file.
  3. Reboot the consumer and verify.
  4. The old value is gone once the store write succeeds.

Two values are not routine rotations:

  • ENCRYPTION_KEY protects every Infisical secret.
  • PGBACKREST_CIPHER protects every backup set.

Both require a fresh backup, a passing drill, the old value offline, and a maintenance stop. Read ROTATION.md before touching either.

S3_ACCESS_KEY, S3_SECRET_KEY and TS_AUTHKEY are provider-owned. Rotate them in the provider console, then re-run install ops or reboot the affected component.

Backups

Fact Default
Backup schedule @daily
Drill schedule 0 5 * * 1 (weekly)
RPO 24 hours
Warm restore target 60 minutes
Retention 30 days

What is backed up:

  • forgejo-postgres and infisical-postgres through pgBackRest base + WAL.
  • forgejo_data and infisical_data through stop-copy-start.
  • One manifest.json per set, uploaded last.
./lk check backup                     # newest set and age
./lk check backup --json              # machine-readable
./lk check backup --coverage          # five declared objects
./lk recover backup --reason "before an upgrade"
./lk recover drill --reason "weekly"

A drill restores both clusters into /srv/lk-drill-<slug>/, runs an integrity query in a throwaway container, removes the root and records restore_seconds. Production is never stopped.

The first drill inside install ops writes an install-ops tag, not a recover-drill tag. check status reads recover-drill tags, so run one laptop drill to populate the drill line:

./lk recover drill --reason "first measured drill"

A restore is different:

./lk check backup --json
./lk recover restore --set <id> --reason "forgejo data loss"
./lk check verify

--set is required. There is no "latest" shortcut because choosing a set is the decision that loses data after it. A failed restore leaves the instance stopped; it does not restart half the stack.

Patching and upgrades

Images are pinned by tag and digest in config/instance.yml.

./lk check verify --pins

The upgrade path is:

  1. Edit the pin in config/instance.yml and commit.
  2. ./lk change upgrade --component <name> --reason "...".
  3. ./lk check verify --component <name> and ./lk check drift.

Stateful components refuse an upgrade unless a complete backup set is newer than the pin edit. Re-run ./lk recover backup first.

Rollback is the pin, reverted:

git revert <pin-edit-commit>
./lk change upgrade --component forgejo --reason "rollback to the previous pin"
./lk check verify --component forgejo

The core has no recover rollback and no app version to roll back to.

Host packages:

./lk change patch --dry-run
./lk change patch --reason "monthly patches"
./lk change patch --reboot --reason "kernel security update"

Suspend and resume

./lk change suspend --reason "maintenance window"
./lk change resume --reason "maintenance complete"

Suspend stops all ten accessories and disables cron. Resume starts each accessory by name and re-enables cron. It does not run a reboot, because a reboot would try to read Infisical while Infisical is starting.

Alerts and status

  • The host runs config/host/check.sh every 15 minutes.
  • A red check posts one line to notify_channel / notify_url.
  • Backups run from host cron, not from CI.
  • ./lk check status --publish prints the eight-line status issue body.
  • ./lk check incident --cron shows the drop-in and its last run.

The host check asserts containers, pins, loopback ports, disk and backup age. It does not read git, tags or facts from the laptop.

Handover and retirement

Hand over with one attestation:

./lk check status
./lk check preflight --facts --host --store --tailnet
./lk transfer handover
git tag -l 'evidence/transfer-handover-*' --format='%(contents)'

Retire in two steps:

./lk retire export --reason "end of life"
./lk retire decommission --confirm <slug> --scope all --reason "end of life"

--scope is workload, data or all. --confirm takes the slug in full; there is no --yes. retire export must run first, and the decommission refuses without it.

Reading outcomes

Code Meaning Tag
0 Changed One evidence/* tag, result: ok
10 Nothing to change None
11 Drift or red check None
2 Refused before mutating One tag, result: refused
1 Mutated and failed One tag, result: failed 
./lk check status --history
git tag -l 'evidence/*' --sort=-creatordate | head

Rules of thumb

  • Do not re-run install setup to repair a container. Use change upgrade --component.
  • Do not delete a lock file by hand. check status names the holder.
  • Do not rotate ENCRYPTION_KEY or PGBACKREST_CIPHER as a routine change.
  • Do not restore without a chosen set id.
  • Do not treat a backup as proven until a drill has passed.