App operations¶

App actions follow the same six intents as the core, with two differences:

Mutations run on the core's Forgejo runner, not on your laptop.
The app action set includes release and rollback; ERPNext also has migrate.

Where commands run¶

Command	Runs on	Why
`./lk collect`	Your laptop	One attended step: create the repo, store project and secrets
`./lk check *`	Laptop or runner	Read-only
`./lk change release`, `patch`, `upgrade`, `migrate`	Runner only	A laptop dispatch exits `2`
`./lk recover *`	Runner	Production restore and drill
`./lk change rotate`	Laptop or runner	The one attended exception; needs a human write token and `--confirm <slug>`
`./lk retire *`	Runner	Export and teardown

Dispatch a workflow in Forgejo:

Open the app repository.
Open Actions.
Select install, change, recover, transfer, retire or check.
Choose Run workflow, set the action, add a reason, and run.

Runner capacity is 1. A drill or release waits behind the current job.

Daily read¶

./lk check status
./lk check verify
./lk check backup
./lk services

Command	Answers
`status`	State, backup age, drill age, next due item
`verify`	App-specific probes
`drift`	Rendered shape, DNS, WAF and Services; not the image digest
`secrets`	Secret names and due dates; never values
`cost`	Object-store and edge usage
`incident`	Read-only incident bundle
`services`	Every URL for this instance
`report`	Operations report

Container logs are in Dozzle at https://<slug>-dozzle.<tailnet_dns>. Dozzle is loopback-bound, read-only and never public.

Command map¶

I want to	Command
Deploy a new app version	`./lk change release --version <v> --reason "..."`
Upgrade the host	`./lk change patch --reason "..."`
Check template differences	`./lk change upgrade --notes`
Migrate the ERPNext schema	`./lk change migrate --reason "..."`
Rotate one secret	`./lk change rotate --name <NAME> --confirm <slug>`
Stop the app	`./lk change suspend --confirm <slug> --reason "..."`
Start it again	`./lk change resume --reason "..."`
Take a backup now	`./lk recover backup --reason "..."`
Restore a set	`./lk recover restore --set latest --confirm <slug> --reason "..."`
Test the newest set	`./lk recover drill --confirm <slug> --reason "..."`
Roll back the image	`./lk recover rollback --version <v> --confirm <slug>`
Export before leaving	`./lk retire export`
Tear it down	`./lk retire decommission --confirm <slug> --scope instance,volumes,edge`

Release and update¶

flowchart TB
  P[Pin or template change] --> R[change release]
  R --> B[pre-deploy: guard, backup, ensure]
  B --> D[Kamal deploy]
  D --> V[post-deploy verify]
  V -->|"verify: green"| L[Live]
  V -->|"verify: red"| K[Explicit rollback]

change release:

Checks the Kamal pin.
Exits 10 if the requested version is already running.
Builds and pushes the app image to the core's Forgejo registry.
Deploys with Kamal; pre-deploy runs the guard, ensures edge/tailnet inputs, takes a backup when the app exists, and checks template notes.
Verifies after deploy.

A red post-deploy verify does not roll back automatically. The new version stays live; recover rollback is the explicit undo.

Facts-only changes do not deploy

If you change a fact but keep the same app.version, change release exits 10 before Kamal runs. Apply it with a version change, or run the specific edge ensure, tailnet ensure or accessory reboot.

Per-app update notes¶

App	Update
CryptPad	`change release --version <v>`; OnlyOffice mounts are part of the image
ERPNext	`change release --version <v>`, then `change migrate`. A release does not migrate the schema
Jitsi Meet	`change release --version <jitsi-release>`; accessory pins are rebooted with `kamal accessory reboot <name>`
Zipline	`change release --version <v>`; a PostgreSQL pin change recreates the database accessory with downtime

Backups¶

Default schedule is 0 3 * * *; retention is 30 days; the RPO target is 24 hours and the warm RTO target is 60 minutes.

App	Backup	Restore and drill
CryptPad	Tar of volume directories + SHA-256 manifest; app stopped	Verify hash, swap volumes, start. Drill uses a separate root
ERPNext	All-database `mariadb-dump` + `sites/` tar	Stop roles, load dump, replace sites, restart. Drill on `127.0.0.1:3307`
Jitsi Meet	Tar of seven state directories; lldap and Continuwuity paused	Verify hash, replace tree, restart six services. Drill checks the databases
Zipline	pgBackRest full/diff + WAL; uploads tar when `storage=local`	Stop app and Postgres, restore PGDATA and uploads. Drill on `127.0.0.1:5433`

./lk recover backup --reason "before an upgrade"
./lk recover restore --set latest --confirm <slug> --reason "restore"
./lk recover drill --confirm <slug> --reason "monthly"

A restore refuses without --set and --confirm <slug>.
Restores are destructive and keep no automatic pre-restore copy.
--set latest means the label in /var/lib/<project>/last-backup.json. For CryptPad it means the newest object in the store.
A failed restore leaves the app stopped and records result: failed.

Users¶

There is no ./lk action that creates an application user.

App	How to add a user
CryptPad	No account system; share document URLs
ERPNext	Create Frappe users in the ERPNext UI
Jitsi Meet	Create meeting accounts in the lldap UI; no self-registration
Matrix	Register with `MATRIX_REGISTRATION_TOKEN`; the admin is created by `install admin`
Zipline	Admin is created by `install admin`; further users through the Zipline UI

Read admin passwords with:

./bin/lk-secret ADMIN_PASSWORD
./bin/lk-secret LLDAP_ADMIN_PASSWORD
./bin/lk-secret MATRIX_ADMIN_PASSWORD

Tailnet access¶

Tailnet grants are declared in policy/tailnet.yml.

grants:
  - src: ["group:<core>-admins"]
    dst: ["svc:<slug>-main"]
    ip: ["tcp:443"]

CI=true ./lk tailnet ensure     # reconcile
./lk transfer access            # review; 0 converged, 11 drift

tailnet ensure always requires CI=true. It merges grants; deleting a grant from the file does not remove the live ACL entry. Remove it in the Tailscale console, or use retire decommission --scope edge.

Rollback¶

./lk recover rollback --version <v> --confirm <slug>

Rollback uses Kamal's rollback when the image is retained, otherwise it deploys the pinned version. It does not downgrade a schema.

ERPNext: an older image cannot start on a newer schema. Restore a dump taken before the migration.
Zipline: a newer image may migrate on start; going back does not undo it.
Jitsi: the Matrix server_name cannot change for the life of the data. Restoring Continuwuity data under a different name does not adopt the old rooms.

Suspend and resume¶

./lk change suspend --confirm <slug> --reason "maintenance"
./lk change resume --reason "maintenance complete"

Suspend puts the app into Kamal maintenance mode. Resume makes it live again.

Retirement¶

./lk retire export > export-<slug>.txt
./lk retire decommission --confirm <slug> --scope instance,volumes,edge --reason "retire"

--scope is a comma-separated subset of exactly:

Scope	Removes
`instance`	Containers, proxy route, host bodies, cron, env file, lock directory
`volumes`	`/var/lib/<project>` and the drill root
`edge`	Tailnet Services, owned grants and tags, public DNS and pull zone

Always kept: the S3 backup prefix, evidence tags, the Infisical project, the Forgejo repository and provider credentials. Revoke those separately.

On a host shared with another app, set LK_SHARED_HOST=1 so teardown does not remove the shared proxy or network.

Reading outcomes¶

Code	Meaning
`0`	Changed or green
`10`	Nothing to do
`11`	Red read-only check
`2`	Refused before mutating
`1`	Mutated and failed

Every terminal mutating outcome writes one tag:

./lk evidence
./lk evidence --verify
git tag -l 'evidence/*' --sort=-creatordate | head

Per-app notes¶

App	Watch for
CryptPad	Two hostnames for public. The first admin URL is stored at `/etc/lk/<project>.admin-url` on the app host; open it once, then re-run `install admin` to prove it is dead. First boot installs OnlyOffice, so it is slower
ERPNext	Release and migration are separate. `sites/` is a shared bind mount. A restore reverts the MariaDB root password to the dump's value
Jitsi Meet	Public needs UDP `10000` direct to the origin. Three hostnames for public. Matrix `server_name` is immutable. lldap accounts only; no self-registration
Zipline	`storage` defaults to `s3`; pass `storage=local` for host uploads. Escrow `PGBACKREST_CIPHER_PASS` — losing it makes every backup unreadable

Public origin¶

Bunny terminates TLS and fronts the app. The template prints the Bunny IP ranges for origin allowlisting but does not apply a firewall rule. Restrict the origin yourself; otherwise the origin IP is directly reachable.