NotusBase

Runbooks

One section per problem the server warns about. Each warning on the dashboard (Home → Overview) has a How to fix link to its section here, and each alert in alerts.yml carries the same section in its runbook_url annotation. Every section says what the warning means, how to confirm it, and the fix, step by step. Where no command fixes it yet, the section says so under No command yet. Tested says whether the runbook was rehearsed, by causing the problem on a throwaway server and typing its steps, or only read against the code.

The background, the limits and the metrics are in Operations. This page is what to type.

Before any runbook

The commands assume the Linux package with systemd (First server), a root shell on the server (sudo -i), and these lines pasted once:

cd /var/lib/notusbase
NB="sudo -u notusbase /opt/notusbase/current/notusbase"
DIR=/var/lib/notusbase/pb_data
URL=http://localhost:8091
# One metric, by name: metric notusbase_wal_bytes
metric() { printf 'Authorization: Bearer %s\n' "$($NB metrics-key --dir $DIR)" | curl -fsS -H @- $URL/api/notusbase/metrics | grep "^$1"; }
# The warnings of the status screen, one per line: title, then runbook
notices() { printf 'Authorization: Bearer %s\n' "$($NB deploy-key --dir $DIR)" | curl -fsS -H @- $URL/api/notusbase/status | grep -o '"title":"[^"]*"\|"runbook":"[^"]*"' | paste - -; }
# A folder for logs kept for a report
install -d -o notusbase -g notusbase -m 700 /var/lib/notusbase/kept

URL is the admin port. If NOTUSBASE_ADMIN_HOSTS is set, it must include localhost; empty, it refuses no host. notices lists what the dashboard shows right now, so a fix can be confirmed with it too. Three things recur:

A warning leaves the screen by itself once its condition stops holding at the next minute's sample. An event warning, like a restart, stays for a fixed time and then leaves.

NotusBaseDown

Alert: NotusBaseDown (page). On the dashboard: nothing, the dashboard is down with the server. With NOTUSBASE_ALERT_PRIMARY and NOTUSBASE_ALERT_URL set on the standby, it sends Primary is not answering; with only the first, it only writes that to its log.

Tested: Rehearsed on 2026-11-11 in a Linux container without systemd: the forced stop of step 6 (QUIT: goroutine dump, restart, then the unclean-restart warning). Read against the code, not rehearsed: the alert itself, systemd and the journal.

What it means. Prometheus could not read the metrics for 1 minute: the process is down, hung, or the admin port does not answer.

Confirm.

systemctl status notusbase --no-pager
curl -s $URL/api/health
journalctl -u notusbase -n 100 --no-pager | grep -iE 'panic|fatal|error'

Fix.

  1. If the unit is failed after 10 crashes in 5 minutes, go to NotusBaseRestarting first.
  2. If it is only stopped: systemctl start notusbase, then curl -s $URL/api/health.
  3. If it started right after an update: /opt/notusbase/update.sh --rollback.
  4. If the log says the disk is full, see NotusBaseDiskAlmostFull.
  5. If the log points at the database (malformed, corrupt): $NB check --dir $DIR. If it does not print ok, restore: Restoring.
  6. If the process is up but does not answer (hung): systemctl kill --kill-who=main -s QUIT notusbase. This is a forced stop, not a clean one: the binary does not catch QUIT, so Go prints every goroutine to the journal and exits at once, without the wait for calls in flight that a stop or restart has. The last WAL frames are not copied to the backup before it exits (the next chain picks them up), the next start shows Restarted without a clean stop, and confirmed writes stay in the database. --kill-who=main keeps the signal off the use node workers. systemd starts it again within 2 s and clients reconnect by themselves. Keep that log for the bug report: journalctl -u notusbase --since -10min --no-pager > /var/lib/notusbase/kept/hung-$(date +%F-%H%M).log.
  7. /opt/notusbase/doctor.sh.

Switching to the standby: notusbase promote, which moves the route by itself with NOTUSBASE_PROMOTE_HOOK (Standby).

Standby cannot follow the bucket

Alert: none in alerts.yml; the standby sends it by webhook (NOTUSBASE_ALERT_URL) after 10 minutes in a row of not following. On the dashboard: nothing; the standby has no dashboard.

Tested: Read against the code, not rehearsed: no standby or bucket in the rehearsal.

What it means. The standby's passes fail: it cannot read the bucket, what it read does not apply, or what came directly from the primary does not apply (the error starts with direct segment N: see Standby is not applying). The copy off the primary is still in the bucket, but a switch now would start from where the standby stopped.

Confirm. On the standby, where /folder/standby is its folder:

notusbase replica-status /folder/standby

Then the output of the replica process (its journal, if it runs as a unit): the line replica: stopped at <chain>: <error> says why.

Fix.

  1. AccessDenied, InvalidAccessKeyId, NoSuchBucket, a wrong endpoint: fix the standby's NOTUSBASE_S3_*, the same as the primary's, and start replica again.
  2. The bucket does not answer: the provider. The standby resumes by itself; the log says replica: back.
  3. The primary started a new chain the standby cannot apply, or the error repeats with good keys: run notusbase replica in a new empty folder, which downloads the newest base, and delete the old folder once replica-status shows it caught up.
  4. Confirm: replica-status shows the last applied point close to the last point seen in the bucket.

No command yet: nothing tests the standby's keys before a pass, and the package has no unit for the standby; how it runs is up to whoever set it up (Standby).

Standby is not receiving

Alert: none in alerts.yml yet; the warning is on the primary's dashboard. On the dashboard: Standby is not receiving, and the Standby tile goes red.

Tested: Read against the code, not rehearsed: the switchover rehearsal (switchover_test.go) only checks the metrics.

What it means. The direct standby has synced nothing for over a minute while the primary has newer points, or has not asked at all for 5 minutes (notusbase_standby_silent_seconds; on a quiet primary that is the only sign of a dead standby). A switchover now loses that time and more. While the standby waits for the base of a new chain from the bucket, the lag does not count for the first hour: the tile shows "Waiting for the base" instead.

Confirm. On the primary:

metric notusbase_standby_

On the standby, notusbase replica-status /folder/standby and the replica process's log: standby: direct copy stopped: <error> says why.

Fix.

  1. notusbase_standby_connected is 0: the standby does not reach the primary. Check that it runs, its NOTUSBASE_ALERT_PRIMARY (the primary's public address) and the network between them.
  2. The standby's log says signature or 401: the two NOTUSBASE_S3_KEY differ. Use the primary's on the standby and start replica again.
  3. The log says segment N is only in the bucket now: the standby fell behind past what the primary keeps raw (notusbase_standby_raw_cap_bytes). It catches up through the bucket by itself; nothing to do but wait for notusbase_standby_lag_seconds to fall.
  4. The log says a single point of N MiB ... comes through the bucket, and notusbase_standby_bucket_only_bytes is above 0: one transaction (an import, an index built over a big table) is bigger than the 256 MiB one answer carries. The direct copy goes on from the next segment once the bucket has this one closed (the next upload round after it closes, plus its upload time); until then a switchover loses what the bucket lacks. Nothing to do but wait; split big imports to avoid it.
  5. A standby taken out of service: the primary forgets it after 24 hours without a request, and the warning leaves; restarting the primary forgets it at once.
  6. Confirm: notices no longer lists it, and metric notusbase_standby_lag_seconds is under a few seconds.

No command yet: nothing on the primary shows the standby's own log.

Standby is not applying

Alert: none in alerts.yml yet; the warning is on the primary's dashboard. On the dashboard: Standby is not applying.

Tested: TestStandbyRule checks when it fires; not rehearsed.

What it means. The direct standby received a closed segment over 5 minutes ago and did not apply it to its database, or applied only part of the segment it is on while the next one already came (notusbase_standby_unapplied_seconds). The bytes are on its disk and promote applies them, so a switchover does not lose them, but its database is behind and its daily check does not see them.

Confirm. On the standby, the replica process's log: replica: stopped at <chain>: <error>. Either its pass over the bucket fails (the direct copy is applied only after a pass that worked), or the apply itself fails: then the error starts with direct segment N. Both count as the replica stopped: after 10 minutes the standby's own watch sends Standby cannot follow the bucket, and its daily check waits.

Fix.

  1. replica: stopped at with a bucket error: check NOTUSBASE_S3_* on the standby and that the bucket answers; the direct copy keeps receiving meanwhile.
  2. replica: stopped at ...: direct segment N: read the error. does not continue the previous segment or invalid frame mean the tail and the bucket disagree: stop replica, delete /folder/standby/.replica/tail and start it again; it receives again from the bucket's point.
  3. Confirm: metric notusbase_standby_unapplied_seconds on the primary is 0.

No command yet: nothing on the primary shows the standby's own log.

Writes are stopped: no lease

Alert: none in alerts.yml yet; the dashboard, the webhook and the e-mail send it. On the dashboard: Writes are stopped: no lease.

Tested: TestLeaseStopsWritesAndResumes, TestStandbyRenewsTheLease, TestLeaseGuardsPocketBaseRoutes and TestLeaseStandbyNotForgottenWhileCut check the refusal, the renewals, PocketBase's routes and when the standby is forgotten; the partition rehearsal (partition_test.go, bench/partition.sh) cuts the bucket and the standby off with proxies, and freezes the primary.

What it means. This server has a standby, and for the last 30 s it reached neither the bucket (its beat) nor the standby (the direct copy). It cannot know whether the standby was promoted, so it refuses every write with Writes are stopped: ..., before running it: nothing was written. Reads go on. It resumes by itself on the first renewal (see The write lease in Operations).

Confirm. On the server:

metric notusbase_lease_

notusbase_lease_required 1 and notusbase_lease_held 0; notusbase_lease_refused_total counts the refused writes. The log says lease: WRITES STOPPED when it stopped and lease: writes resume (renewed by the bucket) (or standby) when it came back.

Fix.

  1. Check whether the standby was promoted (notusbase replica-status on it, or its log). If it was, this server is the old primary: leave it stopped. Once the bucket answers again, its beat reads the new epoch and it stops.
  2. If not, look at the network: the log's backup: lines say why the bucket fails (NOTUSBASE_S3_*, the provider, DNS), and metric notusbase_standby_connected says whether the standby reaches this server. Any one of the two coming back is enough.
  3. The standby was turned off on purpose and the bucket is down: writes stay stopped until the bucket answers, however long ago the standby last asked. To run without the lease, stop the server, delete .notusbase_standby from the data folder and start it again; without a standby, a cut-off primary keeps writing, and a switchover then loses what it confirmed alone.
  4. Confirm: metric notusbase_lease_held is 1 and notices no longer lists it.

Route hook failed after promotion

Alert: none in alerts.yml; the warning is on the new primary's dashboard. On the dashboard: Route hook failed after promotion.

Tested: Read against the code, not rehearsed here: TestPromoteHookFailure runs a failing hook and checks that the server stays up.

What it means. notusbase promote started the new primary, and the command in NOTUSBASE_PROMOTE_HOOK failed. The server is up, but clients may still go to the old address.

Confirm. The output of promote (or the server log) has the hook's output and exit code: metric notusbase_promote_hook_failed is 1.

Fix.

  1. With route-cloudflare.sh: check CF_API_TOKEN (Zone.DNS edit on the zone), CF_RECORD_NAME (the record must exist) and that the machine reaches api.cloudflare.com.
  2. Fix the variable in the standby's environment file, then run notusbase promote again on the same folder: on a folder already promoted it only runs the hook again.
  3. If the hook cannot be fixed now, point DNS, the IP or the proxy at this machine by hand.
  4. Confirm: notices no longer lists it, and the domain answers from this machine.

Integrity check failed after promotion

Alert: none in alerts.yml; the warning is on the new primary's dashboard. On the dashboard: Integrity check failed after promotion.

Tested: Read against the code, not rehearsed: promote --check quick on a damaged file was not caused.

What it means. promote --check quick started serving after quick_check, and the integrity_check that ran next found a defect in the promoted database.

Confirm. The server log has the lines of integrity_check; metric notusbase_promote_check_failed is 1.

Fix.

  1. Keep this folder as it is: it has what the standby received directly, which the bucket may not have.
  2. Restore the bucket's newest chain into a new folder (Restoring) and run notusbase check on it.
  3. If the restored copy is good, stop this server and serve the restored folder in its place; the writes since the promotion that are only here are lost, so say which ones from this server's log if it matters.
  4. Confirm: notusbase check prints ok on the folder that serves.

NotusBaseRestarting

Alert: NotusBaseRestarting (page). On the dashboard: the next start shows Restarted without a clean stop.

Tested: Read against the code, not rehearsed: a crash loop needs systemd's restarts.

What it means. The process started 3 or more times in 15 minutes: a crash loop. After 10 crashes in 5 minutes systemd stops trying.

Confirm.

systemctl show -p NRestarts --value notusbase
journalctl -u notusbase --since -30min --no-pager | grep -iE 'panic|fatal|signal|out of memory'
journalctl -k --since -30min --no-pager | grep -i 'killed process'

Fix.

  1. Right after an update: /opt/notusbase/update.sh --rollback.
  2. Killed for memory (Killed process in the kernel log): see Headroom: memory, then systemctl reset-failed notusbase && systemctl start notusbase.
  3. A panic with the same stack every time: keep the log (journalctl -u notusbase --since -30min > /var/lib/notusbase/kept/crash-$(date +%F-%H%M).log) and report it. If it comes from a deploy, publish the previous commit again with notusbase deploy from your machine.
  4. The database: $NB check --dir $DIR. Not ok means Restoring.
  5. systemctl reset-failed notusbase && systemctl start notusbase, then /opt/notusbase/doctor.sh.

No command yet: nothing says which of the causes above it was. The log is read by hand.

NotusBaseUncleanRestart

Alert: NotusBaseUncleanRestart (page). On the dashboard: red, Restarted without a clean stop. Also by webhook and email. It stays 15 minutes.

Tested: Rehearsed on 2026-11-11 in a Linux container without systemd: caused with kill -9 and with QUIT; warning, metric, log line and check. journalctl, the kernel log and last were not run.

What it means. At startup the server found pb_data/notusbase.running, which only a clean stop deletes. The previous run crashed, was killed (kill -9, out of memory, QUIT) or the machine lost power. When only the process died, no confirmed write is lost: the database recovers from the WAL. When the machine lost power, the last second of writes may be lost (synchronous=NORMAL; see the fsync measure in Checking the machine).

Confirm.

metric notusbase_unclean_start
journalctl -u notusbase --since -1h --no-pager | grep -iE 'panic|fatal|signal'
journalctl -k --since -1h --no-pager | grep -i 'killed process'
last -x reboot | head -3

Fix.

  1. Find the cause in the lines above: a panic (report it with the log), out of memory (see Headroom: memory), or a reboot or power loss (nothing to fix in NotusBase).
  2. Check the database: $NB check --dir $DIR prints ok. It works with the server up, but it holds a snapshot for as long as the integrity check runs, so the WAL grows meanwhile (NotusBaseWalGrowing): on a large database, run it at a quiet hour.
  3. /opt/notusbase/doctor.sh.
  4. If it repeats, it is NotusBaseRestarting.

NotusBaseWriterFailed

Alert: NotusBaseWriterFailed (page). On the dashboard: red, Writer lost a whole batch. Also by webhook and email.

Tested: Rehearsed on 2026-11-11 in a Linux container without systemd: caused by filling the data disk to 100%; warning, metric, df and check, then freeing the disk brought writes back. The kernel-log step was not run.

What it means. A commit of the single writer failed as a whole, so every mutation in it returned an error. It is the disk or the database, not the code: a function that throws fails alone.

Confirm.

metric notusbase_writer_errors_total
df -h /var/lib/notusbase
$NB check --dir $DIR

Fix.

  1. Disk full: NotusBaseDiskAlmostFull. Writes come back by themselves once there is room.
  2. check not ok: the database is damaged. Stop writing to it and restore: Restoring.
  3. Neither: look at the kernel log for I/O errors, journalctl -k --since -1h --no-pager | grep -iE 'i/o error|nvme|ext4|xfs'. A failing disk means moving to another machine (Standby).
  4. Confirm: metric notusbase_writer_errors_total stops growing, and a mutation works.

No command yet: the server log only says why the batch failed for a disk I/O error (a writer: line); otherwise the cause comes from df, check and the kernel log.

NotusBaseWritesPaused

Alert: NotusBaseWritesPaused (page). On the dashboard: red, Writes are paused after a disk error. Also by webhook and email.

Tested: By TestCutRefusedFramesLeavesOnlyCommitted (the cut on a WAL with a refused commit frame), TestWritesPauseWhenTheCutFails (the cut made to fail) and by TestGhostAfterKill on a small full volume, on macOS and Linux. Not rehearsed on a server.

What it means. A commit failed with "disk I/O error" after writing its frames into the WAL file. The server cuts those frames off before it answers, so the write is refused for good. This time the cut failed too. Until the frames are gone, a restart could bring the refused write back as committed, so the writer refuses every mutation with "Writes are paused after a disk error". The pause holds the writes of functions only; writes PocketBase makes by itself (records API, sign-in, the admin dashboard) are not held (Limits). The calls that hit the failed commit got "Write outcome unknown": their write may or may not appear. Queries and subscriptions go on.

Confirm.

metric notusbase_writes_paused
df -h /var/lib/notusbase
journalctl -u notusbase --since -1h --no-pager | grep 'writer:'

Fix.

  1. Free space on the data disk (NotusBaseDiskAlmostFull). The writer tries the cut again on the next mutation, at most once a second, and resumes by itself when it works: the log says "writes resume".
  2. The disk has room and the log still says the cut failed: look at the kernel log for I/O errors, journalctl -k --since -1h --no-pager | grep -iE 'i/o error|nvme|ext4|xfs'. A failing disk means moving to another machine (Standby).
  3. Do not restart or stop the server while it is paused: the stop cannot cut the frames either, and the next start may take the refused write as committed.
  4. Confirm: metric notusbase_writes_paused is 0 and a mutation works. For the calls that got "Write outcome unknown", read the data before sending them again.

NotusBaseWalGrowing

Alert: NotusBaseWalGrowing (page). On the dashboard: red, WAL above 1 GiB and growing. Also by webhook and email.

Tested: Rehearsed on 2026-11-11 in a Linux container without systemd: an outside sqlite3 holding a read; the lsof line found it and left the server out, the WAL grew, and after kill it stopped growing. The WAL reached 155 MB, not the 1 GiB of the warning.

What it means. The checkpoint cannot reset the WAL. Usually a reader holds an old snapshot: a long query, a tool left open on data.db, or the backup check. With no backup chain, the WAL also grows until 8 GiB. At 2 GiB the writer brakes.

Confirm.

metric notusbase_wal_bytes
metric notusbase_backup_check_running
metric notusbase_backup_chains
lsof -t /var/lib/notusbase/pb_data/data.db | grep -vx "$(systemctl show -p MainPID --value notusbase)" | xargs -r ps -o pid,etime,user,args -p

Fix.

  1. The lsof line lists every process that has data.db open except the server itself; it prints nothing when there is none. A sqlite3, a backup tool or an editor there holds the WAL: end it, kill <pid>. A notusbase check or notusbase sql there was started by someone: let it end, or end it.
  2. A backup check is running (notusbase_backup_check_running 1): it gives up by itself at NOTUSBASE_BACKUP_CHECK_WAL_MB and resumes on the next run. Wait.
  3. No chain (notusbase_backup_chains 0): NotusBaseBackupStalled.
  4. Confirm: metric notusbase_wal_bytes stops growing while writes go on: the WAL starts over from its beginning. The file itself shrinks back to about 200 MB at that restart (journal_size_limit), which needs writes, so on an idle server the number stays where it was until the next write.

No command yet: nothing lists the reads in flight inside the server or cancels one, and nothing stops a backup check that is running.

NotusBaseBrakeActive

Alert: NotusBaseBrakeActive (page). On the dashboard: red, Writer is throttled. Also by webhook and email.

Tested: Read against the code, not rehearsed: it needs 2 GiB of WAL.

What it means. The WAL passed 2 GiB and the writer waits 3 times the length of each batch, so the checkpoint catches up. Mutations are about 4 times slower.

Confirm.

metric notusbase_brake_active
metric notusbase_wal_bytes
iostat -xz 1 5

Fix.

  1. Follow NotusBaseWalGrowing: the brake turns off by itself when the WAL falls.
  2. iostat with %util near 100 on the data disk: the disk is the limit. See Why is it slow, step 1.

NotusBaseBackupStalled

Alert: NotusBaseBackupStalled (page). On the dashboard: red, Backup is behind. Also by webhook and email.

Tested: Rehearsed on 2026-11-11 in a Linux container without systemd: caused by a full backup disk, and by a base larger than the free space of a disk under 85% full; warning, log line and, for the first, the fix.

What it means. There have been writes with no copy in the backup for more than 5 minutes. The backup folder refuses writes: permission, full disk or a disk that is not mounted. Writes to the database go on.

Confirm.

metric notusbase_backup_lag_seconds
journalctl -u notusbase --since -1h --no-pager | grep 'backup:'
ls -ld /var/lib/notusbase/backup
df -h /var/lib/notusbase/backup

Fix.

  1. Permission (permission denied in the log): chown -R notusbase:notusbase /var/lib/notusbase/backup. On a separate disk, its folder also has to be in ReadWritePaths of the unit: systemctl edit notusbase, then systemctl restart notusbase.
  2. Full (no space left on device in the log): NotusBaseDiskAlmostFull. The backup disk needs room for a new base, the size of data.db, beside the chain it replaces, so a disk with more than 15% free can still refuse it: compare du -sh /var/lib/notusbase/pb_data/data.db with the free space of df. Then free space or grow that disk.
  3. Not mounted: mount -a, or mount it by hand.
  4. Wait. Within 1 minute the server tries a new base. Do not restart for this.
  5. Confirm: journalctl -u notusbase --since -10min --no-pager | grep 'backup: chain' shows the new chain, and metric notusbase_backup_lag_seconds drops near 0.

The full story, with the WAL ceiling while there is no chain, is in Backup stopped.

NotusBaseNoBackup

Alert: NotusBaseNoBackup (page). On the dashboard: red, No backup. Also by webhook and email.

Tested: Rehearsed on 2026-11-11 in a Linux container without systemd: a server started without NOTUSBASE_BACKUP; the warning came after 31 minutes, and the fix made a chain and took it away.

What it means. For 30 minutes the server has had no backup at all: NOTUSBASE_BACKUP is not set, or it is set and no chain was ever made.

Confirm.

metric notusbase_backup_enabled
metric notusbase_backup_chains
grep NOTUSBASE_BACKUP /etc/notusbase/notusbase.env
journalctl -u notusbase --since -1h --no-pager | grep 'backup:'

Fix.

  1. notusbase_backup_enabled 0: add NOTUSBASE_BACKUP=/var/lib/notusbase/backup to notusbase.env, install -d -o notusbase -g notusbase -m 700 /var/lib/notusbase/backup, then systemctl restart notusbase.
  2. Enabled with no chain: the folder refuses writes. Follow NotusBaseBackupStalled.
  3. Confirm: metric notusbase_backup_chains is 1 or more, then /opt/notusbase/doctor.sh.

NotusBaseBackupCheckFailed

Alert: NotusBaseBackupCheckFailed (page). On the dashboard: red, Last backup verification failed. Also by webhook and email.

Tested: Rehearsed on 2026-11-11 in a Linux container without systemd: a check that had no room on the backup disk; steps 1 to 3 and 5, with the check moved to another disk, passed and took the warning away.

What it means. The last check restored the backup beside the database and either could not finish (no space, an unreadable chain) or found different data. The backup may not restore.

Confirm.

$NB backup-check --last --dir $DIR --url $URL
grep NOTUSBASE_BACKUP_CHECK_DIR /etc/notusbase/notusbase.env
df -h /var/lib/notusbase/backup

The check restores into NOTUSBASE_BACKUP_CHECK_DIR, or into the backup folder when it is not set; df that one.

Fix.

  1. Keep every chain before anything else. The server keeps only the two newest, and the next chain deletes the oldest. Copy the folder to another disk with room for it (du -sh /var/lib/notusbase/backup/ gives the size), for example cp -a /var/lib/notusbase/backup/ /mnt/other-disk/backup-kept-$(date +%F). Keep the slash at the end: when backup is a link to another disk, without it du measures the link and cp -a copies only the link. The server goes on appending to the newest chain while it copies, so the copy's last segment may be cut short; the sealed segments before it and the older chain are whole, and that is what this copy is for.
  2. The error says space: free it (NotusBaseDiskAlmostFull) or point NOTUSBASE_BACKUP_CHECK_DIR at another disk, and restart. The folder must be owned by notusbase and, being outside /var/lib/notusbase, listed in ReadWritePaths of the unit (systemctl edit notusbase, [Service], ReadWritePaths=/other/disk/folder): the unit has ProtectSystem=strict, and the check fails without it.
  3. Run it again: $NB backup-check --dir $DIR --url $URL. It exits 0 when it passes.
  4. Different data twice is an incident. Check the live database, $NB check --dir $DIR, and keep the S3 copy as it is (backup-s3-status from Backup to S3).
  5. Confirm: $NB backup-check --last --dir $DIR --url $URL shows "ok":true, and the warning leaves within a minute.

No command yet: keeping the chains aside is a cp by hand, and nothing says which side is wrong when the data differ.

NotusBaseRestoreCheckFailed

Alert: NotusBaseRestoreCheckFailed (page). On the dashboard: red, Last restore check failed. Also by webhook and email.

Tested: Read against the code, not rehearsed: a wrong restore needs a damaged chain or bucket.

What it means. The last restore-check restored the backup the way you would after losing the server (restore of the local chain and, with S3, restore-s3), and the copy did not give back the live data: different rows, a _storage file with another size or SHA-256, or a copy that did not start. A check that could not run (no room, a bucket or copy that does not answer, the chain changed) is not a failure; it ends without a verdict and leads to NotusBaseRestoreCheckStale if it lasts.

Confirm.

$NB restore-check --last --dir $DIR --url $URL
journalctl -u notusbase --since -8d --no-pager | grep 'RESTORE CHECK DID NOT PASS'

The problem of the result names the source (local or s3) and the step that went wrong.

Fix.

  1. Keep every chain before anything else, as in step 1 of NotusBaseBackupCheckFailed: cp -a /var/lib/notusbase/backup/ /mnt/other-disk/backup-kept-$(date +%F). Delete nothing in the bucket.
  2. Tell a bad chain from a bad restore path: $NB backup-check --dir $DIR --url $URL. If it fails too, the chain is bad: go to NotusBaseBackupCheckFailed. If it passes and only s3 failed, the bucket's copy is the one that differs; keep it as it is and check backup-s3-status (Backup to S3).
  3. Check the live database too: $NB check --dir $DIR.
  4. Run it again: $NB restore-check --dir $DIR --url $URL. It exits 0 when it passes. A wrong restore twice is an incident: report it with the result of both runs.
  5. Confirm: $NB restore-check --last --dir $DIR --url $URL shows "ok":true, and the warning leaves within a minute.

No command yet: nothing says which row or file differed beyond the problem line, and keeping the chains aside is a cp by hand.

NotusBaseBackupCheckStale

Alert: NotusBaseBackupCheckStale (ticket). On the dashboard: orange, Backup not verified in 2 days.

Tested: Read against the code, not rehearsed: it takes 2 days.

What it means. The backup is on and no check finished in 2 days, while the server has been up longer than that.

Confirm.

$NB backup-check --last --dir $DIR --url $URL
grep NOTUSBASE_BACKUP_CHECK_HOURS /etc/notusbase/notusbase.env
journalctl -u notusbase --since -3d --no-pager | grep -i 'check'

Fix.

  1. Run one now: $NB backup-check --dir $DIR --url $URL.
  2. NOTUSBASE_BACKUP_CHECK_HOURS=0 because the standby checks: look at the standby, whose result reaches this server through the bucket. Otherwise set it back to 24 and restart.
  3. It gives up half way every time: NotusBaseBackupCheckPartialRepeated.

NotusBaseRestoreCheckStale

Alert: NotusBaseRestoreCheckStale (ticket). On the dashboard: orange, No restore check passed in two intervals.

Tested: Read against the code, not rehearsed: it takes 15 days with the weekly default.

What it means. No restore-check passed in two intervals and a day (15 days with NOTUSBASE_RESTORE_CHECK_HOURS=168), while the server has been up that long. Nobody knows whether the backup restores.

Confirm.

$NB restore-check --last --dir $DIR --url $URL
grep 'NOTUSBASE_RESTORE_CHECK_HOURS\|NOTUSBASE_BACKUP_CHECK_DIR' /etc/notusbase/notusbase.env
df -h /var/lib/notusbase/backup

The problem of the last attempt says why it did not pass. The usual one is did not run: needs ... MiB and keeps ... MiB free (15%): the check needs room for a copy of the database (twice that with S3) and leaves the disk at least 15% free. It restores into NOTUSBASE_BACKUP_CHECK_DIR, or into the backup folder when it is not set; df that one.

Fix.

  1. No room: free it (NotusBaseDiskAlmostFull) or point NOTUSBASE_BACKUP_CHECK_DIR at another disk, owned by notusbase and in ReadWritePaths of the unit, and restart.
  2. gave up after, or a bucket or copy that does not answer: a run gives up after 6 hours without a verdict. Look at the bucket (NotusBaseS3Failing) and run it at a quieter hour.
  3. not compared: the restore reached chain ...: the server started a new chain during the run; the next one compares. Run it again.
  4. Run one now: $NB restore-check --dir $DIR --url $URL, or Run now in the Backup block of the System screen.
  5. Confirm: $NB restore-check --last --dir $DIR --url $URL shows "ok":true, and the warning leaves within a minute.

No command yet: nothing picks the hour of the weekly run.

NotusBaseBackupCheckPartialRepeated

Alert: NotusBaseBackupCheckPartialRepeated (ticket). On the dashboard: not a warning; the Backup page shows the last check as partial.

Tested: Read against the code, not rehearsed: it takes 2 days of partial checks.

What it means. For 2 days every check gave up half way to protect the WAL: the database is too big for the writes at the hour the check runs.

Confirm.

$NB backup-check --last --dir $DIR --url $URL
metric notusbase_backup_check_partial

Fix.

  1. Put the copy on another disk: NOTUSBASE_BACKUP_CHECK_DIR=/other/disk/folder in notusbase.env, owned by notusbase, and in ReadWritePaths of the unit.
  2. Or let the WAL grow more during the check: raise NOTUSBASE_BACKUP_CHECK_WAL_MB.
  3. systemctl restart notusbase, then $NB backup-check --dir $DIR --url $URL.
  4. Or move the check to the standby (Standby, last paragraph).

No command yet: nothing picks the hour of the daily check.

NotusBaseS3Failing

Alert: NotusBaseS3Failing (page). On the dashboard: red, S3 upload is failing. Also by webhook and email.

Tested: Rehearsed on 2026-11-11 in a Linux container without systemd: an endpoint that refuses connections, with the keys in setup.env; warning, metric and the backup-s3-status line as the service user. The fixes need a real bucket and were not run.

What it means. Every upload round to the bucket failed for 10 minutes. The local backup goes on; the copy off the machine stopped.

Confirm.

metric notusbase_backup_s3_failing
journalctl -u notusbase --since -1h --no-pager | grep 'S3 DOWN' | tail -3
sudo /opt/notusbase/run.sh backup-s3-status

The last line reads the bucket with the server's own variables, as the service user: run.sh gives the command the unit's environment, and the binary applies setup.env itself, with the same checks as at startup (Commands with the service's environment). Never source setup.env in a root shell; the service user writes it.

Fix.

  1. AccessDenied, InvalidAccessKeyId, SignatureDoesNotMatch: fix NOTUSBASE_S3_ACCESS_KEY and NOTUSBASE_S3_SECRET_KEY where they were set (notusbase.env, or setup.env if the setup page saved them), then systemctl restart notusbase. Edit setup.env as the service user (sudo -u notusbase nano /var/lib/notusbase/setup.env): it must stay owned by notusbase, a regular file, one NAME=value per line, only the names the setup page writes, no control characters, no double quote or backslash in a value, and single quotes only around a whole value. Otherwise the server refuses it and does not start; the journal says why.
  2. NoSuchBucket, a wrong endpoint or region: fix NOTUSBASE_S3_BUCKET, NOTUSBASE_S3_ENDPOINT, NOTUSBASE_S3_REGION the same way.
  3. Quota, connection refused or a timeout: the provider. Nothing to do on this machine but wait; the upload resumes where it stopped.
  4. A database of about 58 GiB or more does not upload at all: see Backup to S3.
  5. Confirm: metric notusbase_backup_s3_failing returns 0, and backup-s3-status shows a recent last instant.

No command yet: nothing tests a write to the bucket with new keys before the restart.

NotusBaseS3Lagging

Alert: NotusBaseS3Lagging (page). On the dashboard: red, S3 bucket is behind. Also by webhook and email.

Tested: Read against the code, not rehearsed: no bucket in the rehearsal.

What it means. The upload works but cannot keep up: the bucket is behind by more than 2 segment deadlines plus 5 minutes. What a failover would lose has grown by the same amount.

Confirm.

metric notusbase_backup_s3_lag_seconds
metric notusbase_backup_s3_seal_seconds
metric notusbase_backup_s3_sent_bytes_total; sleep 60; metric notusbase_backup_s3_sent_bytes_total

The difference between the last two, divided by 60, is the upload rate in bytes per second.

Fix.

  1. The rate sits near NOTUSBASE_S3_MBS (64 MB/s by default): raise it in notusbase.env and restart.
  2. The rate is far below it: the network or the provider is the limit. Measure from the machine to the endpoint, or use a closer region.
  3. Confirm: metric notusbase_backup_s3_lag_seconds falls below the seal period.

NotusBaseDiskAlmostFull

Alert: NotusBaseDiskAlmostFull (page), with dir data or backup. On the dashboard: red, Data disk below 15% free or Backup disk below 15% free. Also by webhook and email.

Tested: Rehearsed on 2026-11-11 in a Linux container without systemd: both disks filled to under 15% free; df, both du lines and step 1 took the warning away. vacuum was run on a disk without room for it and refused, leaving the database as it was.

What it means. Less than 15% free on that disk. The backup check needs free space equal to the database size, and a full data disk stops writes.

Confirm.

df -h /var/lib/notusbase /var/lib/notusbase/backup
du -xsh /var/lib/notusbase/* /var/lib/notusbase/pb_data/* 2> /dev/null | sort -h | tail
# -x stays on one disk: the backup disk is measured on its own
du -sh /var/lib/notusbase/backup/* 2> /dev/null | sort -h | tail
metric notusbase_disk_free_bytes

Fix.

  1. Delete what is not NotusBase's on that disk (logs, old packages, other services' data).
  2. Leftovers of NotusBase that are safe to delete: pb_data.old once a backup check passed after the restore that made it, tmp/copy-* left by notusbase copy (an isolated copy kept with --keep, or a development copy), and old /opt/notusbase/releases/<version> other than current and the previous one.
  3. A database that had a lot deleted shrinks by itself: past 10% of the file and 64 MiB free, the server gives the space back at about 4 MB/s ("Space to give back" in $NB capacity). It cannot when the database predates this version ("only with notusbase vacuum") or when the disk cannot wait: systemctl stop notusbase (it returns once the calls in flight finish, up to 30 s), $NB vacuum --dir $DIR, systemctl start notusbase. It needs free space equal to the database while it runs, the search index is rebuilt at startup, the backup starts a new chain, and from then on the database gives space back by itself.
  4. Otherwise the disk must grow, or the volume must leave the database: Capacity, step 3.
  5. Confirm: df -h shows more than 15% free. The warning leaves within a minute.

No command yet: growing the disk is the provider's, and nothing moves a table to the archive or a customer to a child app. A disk already at 100% is not seen on the dashboard; the alert by webhook and email still goes out while the process lives.

NotusBaseDiskFilling

Alert: NotusBaseDiskFilling (page). On the dashboard: not shown; it needs 6 hours of history, which the screen does not keep.

Tested: Read against the code, not rehearsed: it needs 6 hours of Prometheus history.

What it means. At the pace of the last 6 hours, that disk fills in under 24 hours.

Confirm.

df -h /var/lib/notusbase /var/lib/notusbase/backup
du -xsh /var/lib/notusbase/* /var/lib/notusbase/pb_data/* 2> /dev/null | sort -h | tail
# -x stays on one disk: the backup disk is measured on its own
du -sh /var/lib/notusbase/backup/* 2> /dev/null | sort -h | tail
metric notusbase_wal_bytes

Run du again an hour later to see what grows.

Fix.

  1. The WAL grows: NotusBaseWalGrowing.
  2. The database grows: NotusBaseDatabaseFilling, with less time.
  3. Anything else: NotusBaseDiskAlmostFull.

No command yet: nothing tells which folder grows; du is run twice by hand.

NotusBaseOpenFilesHigh

Alert: NotusBaseOpenFilesHigh (ticket). On the dashboard: orange, 80% of the open files limit.

Tested: Read against the code, not rehearsed: 250 sessions under a limit of 300 files stopped at the automatic session cap, at 45% of the files, as the text says.

What it means. The process holds 80% of the files it may open. Each session is a socket. The server already caps sessions by itself, leaving a tenth of the files and at least 1,024 (half, under 2,048) for the rest (notusbase_sessions_limit), and gives a new session 503 past that; the client retries. So sessions alone reach 80% only with a limit above about 10,000 files, and below that a high count means something else holds files.

Confirm.

metric process_open_fds
metric process_max_fds
metric notusbase_sessions
metric notusbase_sessions_limit

Fix.

  1. Sessions went up with real traffic: raise the limit. systemctl edit notusbase, add [Service] and LimitNOFILE=2097152, then systemctl restart notusbase. The package's unit already sets 1,048,576, so a low process_max_fds means another unit or a container.
  2. Sessions did not go up: something leaks files. Keep ls -l /proc/$(systemctl show -p MainPID --value notusbase)/fd | awk '{print $NF}' | sort | uniq -c | sort -n | tail and report it.
  3. To refuse new sessions earlier than the automatic cap: a lower NOTUSBASE_MAX_SESSIONS in notusbase.env, and restart. A value above the automatic cap falls back to it.

NotusBaseFleetUnreachable

Alert: NotusBaseFleetUnreachable (ticket). On the dashboard: orange, One of the servers is not answering.

Tested: Covered by tests with fake servers (slow, garbage, redirect, wrong key, metadata address, a refresh cut short); not rehearsed with a real fleet.

What it means. A server registered on Home → Servers has not answered GET /api/notusbase/fleet/health for about 5 minutes (5 to 7: this server asks once a minute, and the warning reads the minute before). A restart of this server does not reset that time. The warning is about that server, not this one: it may be down, unreachable from here, or its metrics key may have changed.

Confirm. Open Home → Servers on this server's dashboard: the row is red, "No answer", and says since when and why (no answer in 3 s, connection refused, name not found, TLS certificate not valid, address refused, key refused, redirected, not a NotusBase answer, isolated copy, outbound locked). A yellow "Failed just now, trying again" is the same failure, younger than 5 minutes: not yet this warning. Follow the same steps if it stays; it turns red on its own. Then, from this machine:

curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer <that server's metrics key>" https://<that server>/api/notusbase/fleet/health

Fix.

  1. Connection refused or no answer: that server is down. Follow NotusBaseDown on that server.
  2. Key refused: that server's secret changed (a restore with --new-secret, a new machine). Take the row out on Fleet and add it again with the output of notusbase metrics-key on that server.
  3. Redirected: register the address it answers on, with https:// and without the path the proxy redirects from.
  4. Not a NotusBase answer, or HTTP 404: the address points at something else, or that server runs a version from before the fleet.
  5. Address refused: the name resolves to a cloud metadata or link-local address (169.254.0.0/16, fe80::/10, 100.100.100.0/24, 168.63.129.16, fd00:ec2::/32, fd20:ce::/64), which the fleet never calls; or http:// by a name that does not resolve to loopback. Register the private, Tailscale or public address of that server.
  6. The server was retired: take it out on Fleet. The warning leaves within 2 minutes.
  7. Confirm: the row shows the server's version and warnings again, and the warning leaves within 2 minutes.

No command yet: the fleet is registered on the screen only; nothing registers a server from the command line.

NotusBaseSlowWrites

Alert: NotusBaseSlowWrites (ticket). On the dashboard: orange, Slow writes.

Tested: Read against the code, not rehearsed: it needs a slow disk for 10 minutes.

What it means. The p99 of a commit was above 500 ms for 10 minutes.

Confirm.

iostat -xz 1 5
metric notusbase_brake_active
metric notusbase_backup_check_running

Fix. Follow Why is it slow in order. The usual ones:

  1. The disk at 100% with the backup check running: lower NOTUSBASE_BACKUP_CHECK_MBS, or move the check with NOTUSBASE_BACKUP_CHECK_DIR, and restart. The folder must be owned by notusbase and, being outside /var/lib/notusbase, listed in ReadWritePaths of the unit (systemctl edit notusbase, [Service], ReadWritePaths=/other/disk/folder): the unit has ProtectSystem=strict, and the check fails without it.
  2. The brake: NotusBaseBrakeActive.
  3. A disk that is slow at fsync: doctor.sh measures it. Over 20 ms the disk has to change.

NotusBaseReactivityStalled

Alert: NotusBaseReactivityStalled (page). On the dashboard: red, Live queries stopped. Also by webhook and email.

Tested: Rehearsed on 2026-11-11 in a Linux container without systemd: only the forced stop of step 1. Read against the code, not rehearsed: the stall itself, which is a bug.

What it means. There are writes and subscriptions, but no epoch finished: the screens of the systems stopped updating. This is a bug in NotusBase.

Confirm.

metric notusbase_epochs_total; sleep 60; metric notusbase_epochs_total
metric notusbase_live_queries
journalctl -u notusbase --since -1h --no-pager | grep 'epoch:'

Fix.

  1. Keep the state: systemctl kill --kill-who=main -s QUIT notusbase. This is a forced stop, not a clean one: the binary does not catch QUIT, so Go prints every goroutine to the journal and exits at once, without the wait for calls in flight that a stop or restart has. The last WAL frames are not copied to the backup before it exits (the next chain picks them up), the next start shows Restarted without a clean stop, and confirmed writes stay in the database. --kill-who=main keeps the signal off the use node workers. systemd starts it again within 2 s and clients reconnect by themselves.
  2. Save the log for the report: journalctl -u notusbase --since -1h --no-pager > /var/lib/notusbase/kept/stalled-$(date +%F-%H%M).log.
  3. Confirm: metric notusbase_epochs_total grows again.

No command yet: nothing dumps the state without stopping the server.

NotusBaseSlowEpoch

Alert: NotusBaseSlowEpoch (ticket). On the dashboard: orange, Slow live queries.

Tested: Read against the code, not rehearsed: it needs slow subscribed queries for 10 minutes.

What it means. An epoch (rerunning the subscribed queries after a write) had a p99 above 1 s for 10 minutes. A write takes that long to show on the screens.

Confirm.

metric notusbase_live_queries
metric notusbase_live_reruns_total; sleep 60; metric notusbase_live_reruns_total

Then the dashboard's Functions screen, under Slowest and Live screens: the slow query with many subscribers is the one.

Fix.

  1. Give that query an index, or read fewer documents, in the project, and notusbase deploy.
  2. A table that many write and many subscribe to: move the hot part out of the live page.

No command yet: no command lists the subscribed queries by cost; it is the Functions screen.

NotusBaseRuntimeContractMismatch

Alert: NotusBaseRuntimeContractMismatch (page). On the dashboard: red, Scheduled functions are stopped. Also by webhook and email.

Tested: Read against the code, not rehearsed: it needs a bundle from another runtime version.

What it means. The published bundle was built for another runtime contract than this binary's, usually right after an update. Crons and scheduled functions wait, with no loss, until the next deploy. Calls from clients still work.

Confirm.

metric notusbase_runtime_contract_mismatch
journalctl -u notusbase --since -1d --no-pager | grep 'bundle with contract'

Fix.

  1. From the project, with the same version of the CLI as the server: NOTUSBASE_DEPLOY_KEY=... notusbase deploy --url http://localhost:8091 (through the tunnel in Dashboard and second factor).
  2. Confirm: metric notusbase_runtime_contract_mismatch is 0, and $NB job --dir $DIR --url $URL shows the late jobs running.

NotusBaseScheduledFailing

Alert: NotusBaseScheduledFailing (page). On the dashboard: red, Scheduled function keeps failing. Also by webhook and email. It leaves after one success.

Tested: Rehearsed on 2026-11-11 in a Linux container without systemd: a cron that throws; every step, and the warning left after the fix.

What it means. A scheduled function or a cron failed 3 times in a row. The count lives in memory and resets on restart.

Confirm.

$NB job --view failed --dir $DIR --url $URL
$NB logs --since 1h --dir $DIR --url $URL | grep -A5 ERROR
$NB cron --dir $DIR --url $URL

The dashboard's Logs → Scheduled has the same, with each error.

Fix.

  1. Fix the function in the project and notusbase deploy.
  2. A cron: run it now to see it pass, $NB cron run <name> --dir $DIR --url $URL.
  3. A call outside that is down (an API, email): it is not NotusBase. The next runs retry on schedule.
  4. Confirm: the next run passes and the warning leaves.

No command yet: a failed job cannot be run again by command (job run only takes a pending one); it has to be scheduled again by the code.

NotusBaseSchedulerLagging

Alert: NotusBaseSchedulerLagging (ticket). On the dashboard: orange, Scheduled work is late.

Tested: Read against the code, not rehearsed: it needs a queue of late jobs.

What it means. For 5 minutes the most overdue job has been more than a minute late: more work than workers, or slow jobs holding them.

Confirm.

metric notusbase_scheduler_lag_seconds
$NB job --dir $DIR --url $URL | head -50
metric notusbase_runtime_contract_mismatch

Fix.

  1. The contract mismatch is 1: NotusBaseRuntimeContractMismatch.
  2. Many pending jobs of one function, each slow: make it faster or split it, and notusbase deploy.
  3. Many short jobs: raise NOTUSBASE_JOB_WORKERS (64 by default) in notusbase.env and restart.
  4. Confirm: metric notusbase_scheduler_lag_seconds falls under 60.

NotusBaseContextPerCall

Alert: NotusBaseContextPerCall (ticket). On the dashboard: orange, Code runs with one context per call. The Functions page shows the same in red, with the variables.

Tested: Rehearsed on 2026-11-11 on a throwaway server on macOS, without systemd: a deploy with a written module variable gave notusbase_context_per_call 1 and the warning with this runbook; with the variable moved inside the function, deploy --plan printed no warning, and after deploy the metric was 0 and the warning left within 2 minutes. Read against the code, not rehearsed: the alert itself.

What it means. A function of the deployed code writes a module variable (a let at the top of a file that a function changes). A value like that would pass from one call to the next, so the server stops reusing V8 contexts: every call of every function builds a new one, 14 to 264 times less throughput. Nothing is wrong yet; the server is only slower. It is the project's code, not the server.

Confirm.

metric notusbase_context_per_call

1 is the warning. The dashboard's Functions page names each variable as file: name.

Fix.

  1. In the project, move each named variable inside the function that uses it. A value that must survive between calls goes into a table, and a counter into a document that the mutation patches.
  2. notusbase deploy --plan from the project first: it prints warning: <file: name> is a module variable written by a function for each one still there, and publishes nothing. With none left, notusbase deploy, as in NotusBaseRuntimeContractMismatch, step 1.
  3. Confirm: metric notusbase_context_per_call is 0 and notices no longer lists the warning.

A client of a component created at the top of a file (new RateLimiter(components.rateLimiter, ...)) is not counted; see Module state in What behaves differently.

NotusBaseNewError

Alert: NotusBaseNewError (ticket). On the dashboard: orange, New error in followed by the function, one warning per function, for 30 minutes. It also goes out by webhook and email, with the functions' names and not the message: the first new error at once, and the ones born in the next 10 minutes together, as N new errors in ..., when those 10 minutes end.

Tested: Rehearsed on 2026-11-12 on a throwaway server on macOS, without systemd, with a local webhook: a query that throws with an id and an email in the message, called with two ids, gave one group with both runs (<id> and <email> in place of the values), notusbase_error_groups_new_total 1, the warning New error in items:boom with the message about a minute later, and a webhook message with the function and without the message. The batches of a bad deploy, two cycles of the same function one function filling its 20 groups, a stale group giving its place and many full functions were tested in code (TestNewErrorsGoOutInBatches, TestNewErrorsTwoCycles, TestErrorGroupsCapPerFunction, TestErrorGroupsTakeStalePlace, TestErrorGroupsManyFullFunctions), not on a server. Read against the code, not rehearsed: the Prometheus alert.

What it means. A function failed with a message not seen since the server started. A ConvexError, arguments that do not match the function's args, a name that is not a public function, a write the server redid after a conflict and a call refused for load do not count. A write the server gave up redoing does. Usually a deploy brought a bug, or a call reached a case the code did not expect. The groups live in memory: they reset when the server restarts, and an error that keeps happening is announced again once. A deploy does not restart the process and keeps them.

Confirm.

metric notusbase_error_groups_new_total
$NB logs --since 1h --dir $DIR --url $URL | grep -F '[ERROR]'

On the dashboard, Functions with no function picked: Errors by function and message lists each error with how many times, the first and the last time. latest run opens that run on Logs → History, with its lines and the [Deploy: N] that ran it.

Fix.

  1. Read the latest run: the message, the stack in notusbase logs --since, and the deploy number.
  2. If it came with the last deploy: fix it in the project and notusbase deploy, or publish the previous commit again.
  3. If the caller sends something the function does not expect: validate the arguments, or throw a ConvexError with a message for the caller; a ConvexError is not counted as an error, and neither are arguments that do not match the function's args: both are refusals, grey on Logs → Refused.
  4. Confirm: the group's last time stops moving on the Functions page.

No command yet: a group cannot be marked as solved; it leaves on a restart. A function keeps 20 groups; when full, a new error takes the place of one not seen for 24 hours, or else counts in its "other errors" row and raises no warning. A function that puts a value in its message fills its 20 places again each day and warns again, one or two messages a day. A jump of the clock of more than 24 hours frees every place at once. There is no cap on the whole: about 7 MB with a thousand functions all full. The normalization is best effort: an apostrophe inside single or typographic quotes, a backslash before the closing quote, about one in nine random words of 12 letters and a key of two pieces like X7K2-P9QL still pass.

NotusBaseCallsRefused

Alert: NotusBaseCallsRefused (ticket). On the dashboard: orange, Calls refused. It stays 10 minutes.

Tested: Rehearsed on 2026-11-11 in a Linux container without systemd: NOTUSBASE_VMS=1 and 20 queries at once; 15 got 429, the warning came, and with NOTUSBASE_VMS=8 the same 20 passed.

What it means. A call waited 5 s for a free runtime and was refused. Slow functions hold every isolate.

Confirm.

metric notusbase_calls_refused_total
metric notusbase_isolate_busy_ratio
grep -E '^NOTUSBASE_(VMS|ACTION_ISOLATES)=' /etc/notusbase/notusbase.env

notusbase_isolate_busy_ratio is the share busy right now, so it only shows the peak while the peak lasts. The client got HTTP 429 with Too many concurrent requests; the function log ($NB logs) does not list refusals. Then the dashboard's Functions screen, under Slowest, and Logs → History with the Slow filter. Queries and mutations wait 5 s for one of NOTUSBASE_VMS; an action waits 10 s for one of NOTUSBASE_ACTION_ISOLATES, and the text says which kind it was.

Fix.

  1. A slow function: fix it in the project and notusbase deploy.
  2. A real peak: raise NOTUSBASE_VMS (or NOTUSBASE_ACTION_ISOLATES for actions) in notusbase.env and restart. Each isolate costs memory: check Headroom: memory afterwards.
  3. Confirm: metric notusbase_calls_refused_total stops growing.

NotusBaseIsolatesNearCeiling

Alert: NotusBaseIsolatesNearCeiling (ticket). On the dashboard: orange, V8 isolates near their ceiling, after 5 minutes above 90%. The Limits page shows the same in the V8 isolates alive row.

Tested: Not rehearsed on a server. Read against the code and its tests: three retired generations of action isolates held by long actions, a deploy refused with 429 and the message below, and the swap coming once the actions ended.

What it means. Every V8 isolate alive counts against one ceiling: the pools (NOTUSBASE_VMS, the 4 of reactivity, the 4 shared action isolates, NOTUSBASE_ACTION_ISOLATES) plus room for two retired generations of shared action isolates and one probe. The pools alone never pass 90%. What fills the rest is one of two things: actions with isolates of their own near NOTUSBASE_ACTION_ISOLATES, or long actions started before one or more deploys or environment changes, which keep their isolates alive until they end (at most 10 min). Nothing has failed yet. With no room, a deploy or an import waits 10 s and then fails with 429 No room for new V8 isolates: N actions started before the last changes (deploy or environment variable) are still running..., and an environment change reaches actions and the auth.config.ts providers only when one of those actions ends.

Confirm.

metric notusbase_v8_isolates
grep -E '^NOTUSBASE_(VMS|ACTION_ISOLATES)=' /etc/notusbase/notusbase.env

notusbase_v8_isolates_live against notusbase_v8_isolates_ceiling. On the dashboard's Logs screen, filter by duration: long actions that started before the last deploy or environment change are the retired generations; many actions at once with module state are the isolates of their own.

Fix.

  1. Long actions on old code: wait for them to end before the next deploy, or shorten them (split the work into scheduled steps). The room comes back 2 s after the last one ends.
  2. Actions with isolates of their own: remove module state from them (see NotusBaseContextPerCall for how to spot it), or lower NOTUSBASE_ACTION_ISOLATES in notusbase.env and restart if memory is short. Each isolate costs about 2.3 MB new and 8.8 MB warmed up.
  3. Confirm: metric notusbase_v8_isolates shows live back under 90% of the ceiling, and notices no longer lists the warning.

Environment change not loaded yet

Alert: none: it goes out by no channel. On the dashboard: on the Limits page, orange while the swap waits, red when the code did not load with the new environment. notusbase env set and the dashboard's Environment screen say the same at the moment of the change.

Tested: Not rehearsed on a server. Read against the code and its tests: a change with the margin taken by held actions, answered with the time they end by; a change that auth.config.ts cannot load with, kept and shown, with the action isolates loading it anyway.

What it means. process.env inside a function already has the new value. What the code read when it loaded does not: a client built at the top of a module with a key, and the auth.config.ts providers. Those reload with a swap of the action isolates and a probe, which needs room among the V8 isolates (see NotusBaseIsolatesNearCeiling). Until then logins go on with the providers read before: a provider removed, or a key swapped, still lets its tokens in, at most 10 min + 2 s after the swap found no room. In red: a module or auth.config.ts threw while loading with the new values. The previous values stay until the next change or deploy, and it is not tried again on every call.

Confirm. The notice's text: how many actions are in the way and until when, or the error. notusbase logs has the same, after environment:.

Fix.

  1. Orange, and the change must not wait (a key that leaked): let the long actions end, or restart the server; a restart loads everything with the current environment.
  2. Red: fix the variable (notusbase env set again) or the code that reads it at load, and notusbase deploy.
  3. Confirm: the notice leaves the Limits page, and notusbase env set answers applied: the next call of any function reads the new value.

Test settings in use

Alert: none: it goes out by no channel. On the dashboard: orange, Test settings in use, on the Overview and in /api/notusbase/status, from the first read of one until the process stops. doctor.sh warns too, from the running process's environment.

Tested: Not rehearsed on a server. Read against the code and its tests: a setting read shows with its name, and notusbase dev shows nothing.

What it means. The server read a NOTUSBASE_TEST_<name>: a value that is fixed in production, which the tests and the bench change. Some of them turn a safety limit off: NOTUSBASE_TEST_NODE_PERMISSION=off drops Node's own fence, NOTUSBASE_TEST_NODE_MEM_MB=0, NOTUSBASE_TEST_FN_BYTES_MB=0 and NOTUSBASE_TEST_ACTION_BYTES_MB=0 remove a memory ceiling, NOTUSBASE_TEST_BACKUP_SAMPLE_MB=0 stops the standby's comparison, NOTUSBASE_TEST_BACKUP_WAL_MAX_MB=0 lets the WAL grow without a ceiling while the backup falls behind, and NOTUSBASE_TEST_PPROF opens a debug port. The notice names each one the server read.

Confirm.

bash /opt/notusbase/doctor.sh
systemctl cat notusbase
grep -E '^NOTUSBASE_TEST_' /etc/notusbase/notusbase.env

Fix.

  1. Remove every NOTUSBASE_TEST_ line from notusbase.env, the unit and its drop-ins (systemctl cat notusbase lists them all).
  2. systemctl daemon-reload if the unit changed, then systemctl restart notusbase.
  3. Confirm: the notice is gone after the restart, and doctor.sh no longer says has test settings.

A function refuses most of its calls

Alert: none in alerts.yml. On the dashboard: orange, the function's name and refuses most of its calls, on the Overview, on Limits and in /api/notusbase/status, while more than half of the function's calls of the last 10 minutes were refused, with at least 20 calls. At most three functions show, the ones that refused the most. Also by webhook and email, only when the server itself refused more than half of the calls (arguments that do not match what the function takes): a refusal of your own code may be what the function is for, and the names that are not a function come from the caller, so those two stay on the dashboard.

Tested: Seen on 2026-11-12 on a throwaway server on macOS: 26 calls of one function with arguments it does not take gave the orange notice with "26 of 29 calls" and the live deploy, and Logs → Refused listed the reason. TestNoticeWhenAFunctionRefusesMostOfItsCalls checks the two thresholds and which notice goes out by webhook and email. Not rehearsed on Linux.

What it means. A refusal is a call turned down on purpose: the code threw a ConvexError, the arguments did not match what the function declares in args, or the name is not a public function. One refusal is the protection working: it is not a failure, it is grey on the screens and it warns nobody. Most of a function's calls refused is something else: usually an app that still sends what the code took before a deploy (a browser tab open since before it, a mobile app not updated), or a caller with a bug. The notice says which deploy is live.

Confirm. On the dashboard, Logs → Refused: one row per reason, with who refused (your code or the server), where and how many times. See calls opens the calls of that function that were refused.

$NB logs --since 15m --dir $DIR --url $URL | grep -F '[ERROR]' | grep -F 'ArgumentValidationError'
$NB deploy --history --dir $DIR --url $URL

Fix.

  1. The reason is ArgumentValidationError and it began with a deploy: the deploy changed args and old clients still send the shape of before. Accept both shapes for a while (v.optional, v.union) and notusbase deploy, or go back with notusbase deploy --rollback.
  2. The reason is a message of your own code (ConvexError): the code is saying no to most callers. Read the message; if that is what the function is for (a check that mostly answers no), the notice is noise for this function: return the answer instead of throwing. This case is on the dashboard only: it sends no webhook and no email.
  3. The reason is a function that is not public: a client calls an internal function, or a name that no longer exists. Fix the caller.
  4. Confirm: the notice leaves by itself within 10 minutes of the refusals stopping.

No command yet: the thresholds (half, 20 calls, 10 minutes) are fixed, and a function cannot be left out of the rule.

NotusBaseConflicts

Alert: NotusBaseConflicts (ticket). On the dashboard: orange, Many write conflicts.

Tested: Rehearsed on 2026-11-11 in a Linux container without systemd: 64 calls at a time on one counter (28% conflicts); the warning came after 16 minutes, and a deploy that writes one row per call brought it to 0.

What it means. More than 10% of mutations ran again for 15 minutes, because another one wrote the same document first. A key everyone writes: a single counter, a settings document.

Confirm.

metric notusbase_commit_conflicts_total; metric notusbase_committed_mutations_total
sleep 60
metric notusbase_commit_conflicts_total; metric notusbase_committed_mutations_total

The growth of the first over the growth of the second, in that minute, is the share; the warning is above 0.1.

Fix.

  1. In the project, split the document everyone writes (a counter per user or per shard), or move the count to a scheduled job, and notusbase deploy.
  2. Confirm: the conflicts stop growing faster than a tenth of the mutations.

No command yet: nothing shows which document or which function conflicts.

NotusBaseDatabaseLarge

Alert: NotusBaseDatabaseLarge (ticket). On the dashboard: orange, Database above 80% of the size limit.

Tested: Rehearsed on 2026-11-11 in a Linux container without systemd: NOTUSBASE_DB_LIMIT_GB=1 and 85% of it; warning and capacity. The fixes have no command.

What it means. The database passed 80% of NOTUSBASE_DB_LIMIT_GB, the size agreed for this server. Above it, a switch to the standby takes over 15 minutes.

Confirm.

$NB capacity --dir $DIR --url $URL

It shows each table's size and growth, and the next step.

Fix.

  1. Delete or shorten what the system no longer needs, with a mutation of the project. The server gives the space back by itself; on a database from before that, $NB vacuum --dir $DIR with the server stopped (NotusBaseDiskAlmostFull, step 3).
  2. A limit that was set too low for this machine: raise NOTUSBASE_DB_LIMIT_GB and restart. The failover gets slower.
  3. Otherwise climb the ladder in Capacity: step 3, volume out of the main database.

No command yet: no command moves old rows to an archive, or a customer to a child app.

NotusBaseDatabaseFilling

Alert: NotusBaseDatabaseFilling (ticket). On the dashboard: orange, Database reaches the size limit within 90 days.

Tested: Read against the code, not rehearsed: it needs the daily points of the backup check.

What it means. At last month's pace the database reaches NOTUSBASE_DB_LIMIT_GB within 90 days. The pace comes from the daily backup checks.

Confirm.

$NB capacity --dir $DIR --url $URL

Fix. The same as NotusBaseDatabaseLarge, with more time. Start with the table that grows most.

NotusBaseDatabaseLargerThanRam

Alert: NotusBaseDatabaseLargerThanRam (ticket). On the dashboard: orange, Database larger than RAM.

Tested: Read against the code, not rehearsed: it needs a database larger than the RAM.

What it means. The database no longer fits in the machine's memory. Random writes get up to 9 times slower.

Confirm.

$NB capacity --dir $DIR --url $URL
free -g

Fix.

  1. More RAM: the ladder's step 1 in Capacity.
  2. Or take volume out: NotusBaseDatabaseLarge.

No command yet: resizing the machine is the provider's.

NotusBaseWriterBusy

Alert: NotusBaseWriterBusy (ticket). On the dashboard: orange, Writer busy more than half the time.

Tested: Read against the code, not rehearsed: it needs sustained writes for 10 minutes.

What it means. The single writer spent more than half of 15 minutes in commits. The next peak may hit the ceiling.

Confirm.

$NB capacity --dir $DIR --url $URL
iostat -xz 1 5

Fix. Step 0 of Capacity, in the project: fewer writes per call, bigger batches, no write that changes nothing. Then a faster disk (step 1). The same as Headroom: writer.

NotusBaseSessionsAboveMeasured

Alert: NotusBaseSessionsAboveMeasured (ticket). On the dashboard: orange, More sessions than ever tested.

Tested: Read against the code, not rehearsed: it needs more sessions than the measured ones.

What it means. More than 4,000 sessions for 15 minutes. The biggest test was 5,000; above that, nobody knows where the ceiling is.

Confirm.

metric notusbase_sessions
metric notusbase_sessions_limit

Fix.

  1. Put a ceiling where it was measured: NOTUSBASE_MAX_SESSIONS=5000 in notusbase.env, and restart. Above it a new session gets 503 and the client retries.
  2. To serve more: split the customers into child apps, the ladder's step 4 in Capacity.

No command yet: creating a child app and moving customers to it.

NotusBaseNoHeadroom

Alert: NotusBaseNoHeadroom (ticket), with piece writer, isolates, live queries, cpu or memory. On the dashboard: orange, one warning per piece, below. The Headroom card on the status screen shows the five.

Tested: Read against the code, not rehearsed: the load tried did not reach 80% of any resource.

What it means. That piece was busy more than 80% of the time for 15 minutes. The next peak will queue.

Confirm.

metric notusbase_writer_busy_ratio
metric notusbase_isolate_busy_ratio
metric notusbase_epoch_busy_ratio
metric notusbase_cpu_busy_ratio
metric notusbase_memory_busy_ratio

Headroom: writer

On the dashboard: Writer above 80% busy. Batch writes into fewer mutations, spread rows everyone writes, or move to a faster disk. See NotusBaseWriterBusy.

Headroom: isolates

On the dashboard: Isolates above 80% busy. Slow queries and mutations hold the isolates. Add indexes or read fewer documents in the project and notusbase deploy. Or raise NOTUSBASE_VMS in notusbase.env and restart, if there are cores and memory left.

Headroom: live queries

On the dashboard: Live queries above 80% busy. Epochs follow one another with little gap. Index the subscribed queries, or move heavy ones out of live pages. See NotusBaseSlowEpoch.

Headroom: cpu

On the dashboard: CPU above 80%. Find the functions that cost most on the Functions screen, and top -H -p $(systemctl show -p MainPID --value notusbase). Check the clock source (cat /sys/devices/system/clocksource/clocksource0/current_clocksource): with hpet or acpi_pm, see Why is it slow, step 4. Otherwise a machine with more cores.

Headroom: memory

On the dashboard: Memory above 80% of the limit. The limit is the lowest of GOMEMLIMIT, the service's (MemoryMax) and the machine's RAM. Open Home → Technical details, Processor and memory, to see whose share grows: V8, SQLite or the log. Then lower NOTUSBASE_VMS or NOTUSBASE_LOG_MEMORY_MB in notusbase.env, or raise the limit (systemctl edit notusbase, MemoryMax=), and restart. Memory that rises without stopping is a leak: keep metric process_resident_memory_bytes an hour apart and report it.