Skip to content

Admin portal - Backup, restore, and reset

Purpose

Every tenant's data is backed up automatically, operators can prove it, and when a customer loses or breaks their data an operator can bring it back, or return the tenant to a clean default setup, without hand-run database work.

Users and permissions

All roles can view backups. Owner and Operator can take an on-demand backup. Only Owner can restore or reset. See the permission matrix.

Requirements

ID Priority Requirement
BKP-01 Must The platform backs up each Active or Suspended tenant at least once a day: tenant database and tenant files together, as one backup set.
BKP-02 Must Each backup set records: backup ID, tenant, type (Scheduled, On-demand, Pre-restore, Pre-reset, Pre-deletion), started and finished time, size, schema version, result (Running, Succeeded, Failed), expiry date, and who triggered it.
BKP-03 Must Tenant detail, Backups tab, lists the tenant's backup sets newest first, with the last successful backup time shown prominently.
BKP-04 Must A failed scheduled backup sends an alert to the operations channel and appears on the operations dashboard until a later backup for that tenant succeeds.
BKP-05 Must Owner or Operator can take an on-demand backup of a tenant.
BKP-06 Must Owner can restore a tenant from a successful backup set, with a reason and by typing the subdomain. The platform first takes a Pre-restore backup, puts the tenant in Maintenance, restores database and files, applies schema migrations if the backup is older than the current version, then returns the tenant to its previous state.
BKP-07 Must Owner can reset a tenant to defaults, with a reason and by typing the subdomain. The platform takes a Pre-reset backup, puts the tenant in Maintenance, removes all business data and files, re-applies the default configuration, keeps the subdomain, the tenant owner account, and the subscription, and returns the tenant to its previous state.
BKP-08 Must Restore and reset show progress by step and end with a clear result. If any step fails, the tenant stays in Maintenance, an alert is sent, and the screen offers "restore from the pre-action backup".
BKP-09 Must Restore and reset end all tenant user sessions; the tenant owner is emailed when the action starts and when it finishes.
BKP-10 Must Backups are kept according to a retention policy (default: daily for 14 days, weekly for 8 weeks). Pre-* backups are kept for at least 30 days. Expired backups are removed automatically.
BKP-11 Should Platform-wide backup view: for each tenant, last successful backup time and result, filterable to tenants with no successful backup in the last 26 hours.
BKP-12 Should Each backup set has a checksum verified after upload; a backup that fails verification is marked Failed.
BKP-13 Could Restore into a new, separate tenant for inspection, without touching the original.
BKP-14 Must An operator with backup:create saves one schedule per tenant: daily at HH:mm in Asia/Ho_Chi_Minh (default 00:00) or an interval of a whole number of hours from 1 to 24. The API rejects any other clock or hour value. Saving recalculates the next slot and writes an audit record. Changing the schedule does not change the expiry of backups already taken.
BKP-15 Must The operator chooses postgres or zip. The suggestion is packaging only: zip when the tenant has files, postgres when it has none. Both formats contain the same set: a custom-format dump of that tenant's database and a file manifest. With no file storage, the manifest lists no files.
BKP-16 Must At most one backup runs per tenant. A second on-demand request is rejected and audited as a failure. A scheduled slot that finds a run already going is skipped, and the next slot moves forward. A run whose worker lease expires is marked Failed and its temporary files are removed so another run can start. If the scheduler observes several missed slots, it runs once and returns to the original grid.
BKP-17 Must A successful daily or on-demand backup expires 14 days after it succeeds. A successful interval backup expires 48 hours after it succeeds. Expiry is set when the run succeeds. The expired file is deleted and the row stays, marked purged.

Screens

Tenant detail, Backups tab. Banner: last successful backup time and next scheduled backup. Schedule form (daily time or interval hours, format, suggestion, save) and Backup now (format only, without changing the saved schedule), both shown to Owner and Operator. Table, 10 rows per page, newest first: time, type, format, size, schema version, result, expiry, requested by. A failed row shows a short reason. Support can view the tab and cannot save or start a backup.

Restore dialog. Shows the tenant, subdomain, backup time, format, and schema version, the warning "All data and files changed after

Reset dialog. Lists what is removed (all customers, contacts, opportunities, contracts, products, transactions, tasks, imported data, and files) and what is kept (subdomain, owner account, subscription, audit history), reason field, and subdomain confirmation.

Action progress. Step list with status taken from the restore job, not from a timer: pre-action backup, maintenance on, restore, migration, verify, email the tenant owner, maintenance off. The admin portal keeps showing this progress while the tenant workspace is in Maintenance.

Backups (platform-wide). Table for BKP-11.

Business rules

  • Only one backup, restore, or reset can run per tenant at a time. A backup already running when the tenant enters Maintenance finishes; a new backup does not start while the tenant is in Maintenance or any state other than Active or Suspended.
  • The scheduler runs inside the API process. It creates the default daily 00:00 schedule for an Active or Suspended tenant that has none.
  • Each backup dumps that tenant's own database. The artifact is stored in a temporary directory on the API host. Checksum is SHA-256 of the finished artifact, hashed twice; a mismatch or a dump failure marks the run Failed and deletes the temporary directory.
  • Restore is available only from a succeeded backup of the same tenant whose file is still inside retention, whose checksum and manifest match, and whose schema version is read from that backup. A newer schema is blocked. An older schema is migrated with the tenant migration folder. The destination database is the tenant's own database name.
  • The Owner types a reason of at least 10 characters and the subdomain. Operator, Support, and workspace users are rejected by the API.
  • A restore job holds the tenant lock, including while its pre-restore backup runs. The pre-restore backup is kept for 30 days; that period is not configurable downward. From the moment the job is accepted, tenant API calls are temporarily unavailable, so data cannot change between the snapshot and Maintenance. The tenant record itself switches to Maintenance only after the pre-restore backup succeeds.
  • The screen list used to put "notify owner" after "maintenance off", while BKP-08 says a failed step leaves the tenant in Maintenance. Those two disagree if the finish email can fail. The finish email is therefore sent while Maintenance is still on, and Maintenance is cleared only after that send succeeds. A failed finish email leaves the tenant in Maintenance, with recovery from the pre-restore backup. A failed start email or a failed pre-restore backup does not enter Maintenance and does not overwrite data.
  • A worker whose lease no longer matches cannot mark the job succeeded or change its files. An expired lease is failed by a later sweep; the tenant stays in Maintenance if Maintenance had already started.
  • Tenant users see a maintenance page during Maintenance; API calls get "This workspace is temporarily unavailable. Retry shortly." with a retry hint. Existing sessions for that tenant are deleted. The tenant owner, not the admin Owner, is emailed when the action starts and when it finishes. If email is not configured, the job fails instead of being reported as a success.
  • Recovery from the original pre-restore backup is available to the Owner while the tenant is in Maintenance after a failed restore. It returns the tenant to the state saved before the restore, Active or Suspended.
  • Operators cannot download backups through the portal.
  • Daily and on-demand retention is 14 days. Interval retention is 48 hours. Weekly retention is not applied yet.
  • Outside production, an empty tenant registry receives one sample tenant, Northwind Traders (northwind, northwind.owner@example.com). Every tenant with no backup rows receives 22 sample runs, newest first, covering scheduled and on-demand, postgres and zip, one failure, and one purged file. An Active or Suspended tenant with no schedule receives a daily 00:00 schedule. The first Active or Suspended tenant also keeps two extra succeeded backups, one postgres and one zip, with a small fixture file so Restore can be opened. The fixture dump contains one table, restore_probe, whose row is from-backup. Those two files are rewritten when they are missing. Other tenants do not get a restorable file. Production does not insert any of them.

Acceptance criteria

  • BKP-01: A tenant in Active or Suspended with no schedule receives a daily 00:00 schedule. A tenant in Maintenance does not start a backup.
  • BKP-03: The Backups tab lists sets newest first and shows the last successful backup time.
  • BKP-05: Owner or Operator can start an on-demand backup. Support cannot.
  • BKP-12: A checksum mismatch marks the run Failed and removes the temporary directory.
  • BKP-14: 02:30 and whole hours 1 through 24 are accepted. 24:60, 0, decimals, and values above 24 are rejected. Saving an interval schedule leaves an existing backup's expiry unchanged.
  • BKP-15: With no files, both formats succeed and the manifest lists no files. The dump command targets that tenant's database name.
  • BKP-16: Two overlapping on-demand requests produce one run and one rejection. A missed hourly grid runs once and the next slot stays on the grid. An expired lease is Failed, and a new run can start.
  • BKP-17: A successful daily or on-demand backup expires after 14 days. A successful interval backup expires after 48 hours. The expired file is removed and the row remains.
  • BKP-04: Making one tenant's scheduled backup fail produces an alert and a dashboard item; the item clears after the next successful backup.
  • BKP-06: After creating a record, restoring a backup taken before it removes the record, and a Pre-restore backup appears that still contains it.
  • BKP-06: Restoring a backup from an older schema version leaves the tenant on the current schema version and usable.
  • BKP-07: After reset, the tenant has no business records or files, the owner can sign in with the same password, and the default configuration is present.
  • BKP-08: Forcing a failure mid-restore leaves the tenant in Maintenance, the progress and audit show the reason, and recovering from the pre-restore backup returns the original data. An operations-channel alert is still not sent.
  • BKP-09: A tenant user signed in during a restore is signed out, and the tenant owner receives the start and finish emails. A failed send does not mark the job succeeded.
  • Operator (not Owner) sees no restore action, and the server rejects those calls. Reset is still Owner-only and not built.

Not yet implemented

  • BKP-04 operations alert and dashboard item. BKP-08 does not send that alert yet; the failure stays on the job, the progress screen, and the audit log.
  • BKP-07 reset, including the reset dialog.
  • BKP-11 platform-wide backup view and the 26-hour filter.
  • BKP-10 weekly retention of 8 weeks. Pre-restore backups are kept for 30 days. Pre-reset and pre-deletion retention is not built.
  • Encryption of backup files at rest.
  • Durable object storage for tenant files. A backup manifest can list files, and restore copies those files into TENANT_FILES_ROOT (or the API host temporary directory). The backup dumper still writes an empty file list until file storage exists.
  • Operator approval before restore. Only the admin Owner can restore.

Out of scope

  • Restoring a single record or a single table.
  • Tenant users restoring their own data from inside the workspace.
  • Downloading or exporting backups.
  • Backups of the control plane and platform services; those are covered by infrastructure runbooks, not the portal.

Open questions

  1. Point-in-time recovery for paid plans is not in this slice. The daily schedule remains the default; an operator can also choose an interval of 1 to 24 hours (BKP-14, BKP-17).
  2. Weekly retention of 8 weeks is not applied yet. Daily and on-demand backups expire after 14 days; interval backups expire after 48 hours.
  3. Should reset also remove non-owner user accounts, or keep them?
  4. Operator approval before restore is not in this slice. Only the admin Owner can restore.