Skip to content

Admin portal - Provisioning monitor

Purpose

New tenants are created automatically after sign-up, with no manual work. When that automation is slow or fails, operators see it immediately, understand which step broke, and finish the job without touching servers or databases by hand.

Users and permissions

All roles can view. Owner and Operator can retry and cancel runs. See the permission matrix.

Provisioning steps

A provisioning run executes these steps in order. Every step must be idempotent so a retry can safely run it again.

# Step Done when
1 Register tenant Tenant row exists in the control plane with state Provisioning
2 Create database Isolated tenant database exists and is reachable
3 Apply schema Tenant database is at the current schema version
4 Seed default configuration Default roles, settings, and reference data exist
5 Create owner account Tenant owner user exists with the owner role
6 Activate tenant Tenant state is Active and its subdomain resolves to the workspace
7 Send welcome email Welcome email with sign-in details is accepted by the email provider

If step 7 fails, the tenant stays Active and the run ends as Succeeded with warnings; the email can be resent (see HLT).

Requirements

ID Priority Requirement
PRV-01 Must Every provisioning attempt is recorded as a run with: run ID, tenant, trigger (sign-up or retry), triggered by (system or operator), start and end time, duration, and result (Running, Succeeded, Succeeded with warnings, Failed, Cancelled).
PRV-02 Must Each run records every step with status (Pending, Running, Done, Failed, Skipped), start and end time, and for failures an error message readable by an operator plus a technical detail field.
PRV-03 Must Provisioning list shows runs newest first, with filters for result, time range, and tenant. Failed runs and runs running longer than the threshold are highlighted at the top.
PRV-04 Must Run detail shows the step timeline from PRV-02 and updates live (at most 5 seconds delay) while the run is in progress.
PRV-05 Must Owner or Operator can retry a failed run. The retry is a new run that starts from the failed step and skips steps already Done.
PRV-06 Must A step that fails is retried automatically up to 3 times with backoff before the run is marked Failed.
PRV-07 Must When a run fails, or runs longer than the threshold (default 10 minutes), the platform sends an alert to the operations channel configured for the deployment.
PRV-08 Must A tenant whose last run failed shows state Provisioning failed; its owner sees a friendly "we are preparing your workspace" page with a support contact, not an error.
PRV-09 Should Owner or Operator can cancel a Running or Failed run with a reason. Cancelling never deletes created resources; it stops further steps. Cleanup happens through tenant deletion.
PRV-10 Should Summary at the top of the list: runs in the last 24 hours, 7 days, and 30 days; success rate; median and 95th percentile duration.
PRV-11 Could When a failure is fixed by retry, the tenant owner gets an email that their workspace is ready (the normal welcome email).
PRV-13 Must The provisioning page is a sample screen and does not call the API or change a real tenant. It shows a title and four summary cards (total, completed with a percent, running, failed) counted from all runs, not from the active filter. Status colors are consistent on row badges, filter-chip dots, and summary cards: Completed green, Running blue, Failed red, Waiting amber; the Total card has no colored dot. The list has search by name or email (no duplicate page heading inside the list card), status chips All, Completed, Running, Failed, and Waiting with counts, and returns to page 1 when the query or chip changes. Columns are Tenant, Status, Progress (slim bar colored by status plus fraction text), and Updated. There is no Database column. At most 10 rows show per page, with Previous and Next. Clicking a row selects it. The right detail panel is sticky below the app header with its own scroll for the step list. The tenant name links to /tenants/:tenantId. View log opens a sample log of time, action, and error for the selected run. Completed runs show only View log; failed runs highlight the failed step (red icon and Failed label), show a short error under that step, and offer primary Retry from this step plus secondary View log. Surface colors follow the selected theme, including dark mode. Each finished or failed step shows that step's own duration (not cumulative); those durations sum to the run Duration field. A running step shows elapsed seconds, and each simulated running step stays for 10 seconds. The detail panel shows the seven steps; a skipped database branch reads as not applicable, and skipped steps count as finished in the progress fraction.

Screens

Provisioning. A sample screen (PRV-13): summary cards with status colors, a filterable run list (progress bars, no inner duplicate title), and a sticky detail panel for the selected run. There is no live tenant list. Summary cards and the live run table (PRV-10, PRV-03) are not fed by the API yet.

Run detail. Header with tenant name linking to /tenants/:tenantId, result, and actions (Retry from this step for failed runs, View log). Step timeline: each step with status icon and own duration; failed steps show a short error underneath. Link to previous and next runs of the same tenant.

Tenant detail, Provisioning tab. All runs for the tenant, using the same table.

Business rules

  • Only one run per tenant can be Running at a time; retry is disabled while a run is in progress.
  • A run is Running while any step is Running; a run with no progress for 30 minutes is marked Failed with the error "stalled" and can be retried.
  • Retry is allowed only for the tenant's latest run.
  • The threshold in PRV-07 and the alert destination are deployment settings, not hard-coded.

Acceptance criteria

  • PRV-01: A new sign-up creates one run with trigger sign-up and triggered by system.
  • PRV-02: Forcing the database step to fail shows that step as Failed with an operator-readable message, and later steps as Pending.
  • PRV-05: Retrying that run creates a new run in which steps 1 and 2 are Skipped or re-checked, the remaining steps run, and the tenant ends Active.
  • PRV-05: Running a retry twice in a row never creates a second database or a second owner account.
  • PRV-06: A step that fails once and then succeeds does not mark the run Failed.
  • PRV-07: A failed run produces one alert containing the tenant subdomain, the failed step, and a link to the run detail.
  • PRV-08: The owner of a tenant in Provisioning failed sees the preparing page, not a server error.
  • A Support operator sees runs but no retry or cancel action, and the server rejects those calls.
  • PRV-13: Given the provisioning page, when it opens, then summary counts come from every sample run. Filtering by Failed does not change the total card. The list has no Database column, shows at most 10 rows, and Next shows the rest. Searching part of a name shows only matching runs. Clicking a shared-database run shows the other database step as not applicable and shows the step that moves the tenant to Active. Failed runs highlight the failed step with Retry from this step and View log; the tenant name links to /tenants/:tenantId; per-step durations sum to Duration.

Not yet implemented

  • PRV-01
  • PRV-02
  • PRV-03
  • PRV-04
  • PRV-05
  • PRV-06
  • PRV-07
  • PRV-08
  • PRV-09
  • PRV-10
  • PRV-11

Out of scope

  • The live tenant list, stage filters, and search previously tracked as PRV-12. That ID is not reused.
  • Editing provisioning steps from the portal.
  • Provisioning a tenant by hand without a sign-up.
  • Building alert routing; the portal only sends to the configured destination.

Open questions

  1. Where do alerts go (chat channel, email, on-call tool)? This decides the integration to build.
  2. Is 10 minutes the right threshold for a slow run?
  3. Should a failed sign-up whose provisioning cannot be fixed be automatically cleaned up after some days?