A backup of the live database runs beside sak's writes and never fails one #176
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "task/3860-backing-up-the-live-database-with-vacuum"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Tracker: #3860
The deploy's backup (
sqlite3 sak.db "VACUUM INTO …"on the NAS host) made sak's writes fail after the 5 s busy timeout for minutes. VACUUM INTO only reads its source, and a WAL reader takes no lock a writer waits on. What it does is load the NAS volume: it reads and writes 13.6 GB as fast as the disk takes it. A commit's fsync then runs for seconds while it holds the write lock, and every other writer reaches its 5 s busy timeout. A plaincpof the finished backup, which never touched the live file, failed one write the same way (comment 18498).This PR delivers:
sak backup --db <file> --to <path> [--rate <MiB/s>]. It copies the database through SQLite's online backup API, in steps, inside one read transaction it holds on its own connection. That transaction is the copy's snapshot: the copy is the database as it stood when the backup began, and a write sak makes between two steps never restarts it. The backup takes no lock a writer waits on. It first copies the uploaded files the snapshot records, from<file>.images/to<path>.images/, and lists any whose bytes were deleted before it reached them.--rate(default 20 MiB/s, about 11 minutes for the live 13.6 GB, the time VACUUM INTO took). A copy that falls behind that rate makes up at most a quarter of a second of it (CATCH_UP_NS). That covers a sleep that overshoots, and a stall of the copy (slow files, a busy disk) is not followed by a burst at the disk's full speed.PRAGMA quick_checkon the copy at the same pace, and ends only once it answersok. A progress handler (sqlite3_progress_handler) runs after each page the check visits, inside the check's btree walk too, and spends the bytes of the pages the check read from the copy's file since its last call (SQLITE_DBSTATUS_CACHE_MISS× page size). quick_check reads every page and then each table's pages again as it reads each row, so the check takes up to twice as long as the copy. The binding is SX #3861's (sx af8455c3).--tothat is already there, never writes the source and never migrates it. A backup that fails removes what it wrote, the directory it made included. A backup is laid out as sak serves it, sosak serve --db <path>serves it.BUSY_TIMEOUT_MS(src/db.sx) is 10 s, not 5 s. It stays inside the 15 s handler budget, so a write commits or fails before its request is answered 504.scripts/deploy_nas.shpulls the new image, then backs up with that image'ssakin a container of its own, beside the sak still serving the volume:docker compose run --rm --no-deps -T --name sak-backup sak backup --db /data/sak.db --to /data/backup-<UTC time>/sak.db. It starts the new image only once the backup passed, so a failed backup starts nothing. The backup runs from the pulled image because the running one may predatesak backup, as 0.3.10 does.docs/deploy.mdand the README say so, and the guide says the script runs silently for the backup's 11 minutes, and its check's up to twice that, and that a backup stays in/datauntil an operator deletes it.sx
The paced check needs sx af8455c3 (SX #3861, PR sx/sx#249): sx's sqlite binding gained
progressHandleranddbStatusthere. sak's gates pin af8455c3 since 2026-10-28 08:13Z (#3862); before that, at 49e83601,sx build src/main.sxfailed on them (unresolved 'progressHandler' (in src/backup.sx fn quick_check_ok), gate run 661). The deploy job builds in the runtime imagedeploy/runtime/lockbinds, which is still on the toolchain at 49e83601 (#3840), so the release that carries this PR needs that image rebuilt at af8455c3 or later first (docs/deploy.md, "The job's runtime", steps 1–4). #3871 covers it.Before the pin moved, at af8455c3 on m5, from a
git archivearm of this head merged onto master8b9a135f(tree089d0750):dist/sak,dist/sak-runneranddist/sak-runner-livebuild, andnpm run buildpasses;fullgate'stests/*.sxloop: all 68 files print OK (run four at a time, 5 min 14 s);scripts/release.shbuilds every platform's tarball,tests/step9_release.sxprints STEP9 OK (39 ok), andtests/deploy_contract.shpasses;memories.spec.ts:315phone-dark "paints Edit project's memory switch" (18,802 pixels). That shot does not wait for the project's task list to load, and this PR touches nothing underweb/. The runner config: 10 passed.Regressions
All but the paced check's were run on s32 from
git archivearms with the gate pin's sx of the day (49e83601). The paced check's were run on m5 at sx af8455c3, and again on s32 at af8455c3: the head printed BACKUP OK (20 ok), and the head without itsprogressHandlercall printed the one FAIL below.tests/backup.sx). The backup of the seeded database at 4 MiB/s takes at least 90% of the time its files, its copy and its check's second read of every page take at the rate.progressHandlercall, which checks asa01f2351did, printedFAIL it paces its copy, and its check of the copy, to the rate: the check reads every page again,BACKUP FAIL (1 checks).MAX_RATE_MIBprinted the same single FAIL.dist/sak backup --rate 50of a 310 MB database (75,677 pages of 4 KiB, 150,000 comments) took 17.7 s, against 7.0 s with the check unpaced. The copy takes 5.9 s at the rate, so the check spent about 617 MB: every page twice, as the review's 150,924 cache misses for that check counted.tests/backup.sx). A writer with busy timeout 0 commits a row every few ms, so any lock the backup held would fail its write at once. Meanwhile the backup copies a seeded database of more than 1,000 pages, 256 a step, at 4 MiB/s. Every write succeeds, at least 20 of them while the backup runs. The backup ends, and the copy passesquick_check. It holds every row committed before the backup began and none committed while it ran, with no gap in the ids. It keeps the source's schema version and the uploaded file's bytes, and it reports the file whose bytes were gone.5c55b92ahas nosak backup.BEGIN). It restarts at every step that sees a write, andtimeout 120killed it still copying (rc 124, after "the database spans many steps of the copy").tests/backup.sx). A pace at 4 MiB/s waits out a stall of at least 1 s, spends the stalled MiB, then 4 MiB must take at least 90% of (1 s − the catch-up).e569ba92(the average rate since the copy began), over the same fields, printedFAIL a copy that stalls makes up no more than a catch-up of it: …,BACKUP FAIL (1 checks): it had banked the whole stall.tests/backup.sx). A pace 200 ms behind its schedule spends a quarter MiB (62.5 ms at the rate) without sleeping.CATCH_UP_NS :: 0) printedFAIL a copy behind its schedule by less than a catch-up makes the time up, …,BACKUP FAIL (1 checks). On s32 a run's 50 ms sleep takes about 191 ms, and with no catch-upsak backup --rate 20copied a 99 MiB database in 19.1 s. With the catch-up it took 4.9 s, against 4.95 s at the rate.tests/backup.sx). A backup whose recorded file cannot be read failsBackupErr.Files, and the directory it made for--tois gone.removewithout deleting that directory printedFAIL a backup that fails leaves nothing of itself, the directory it made included,BACKUP FAIL (1 checks).tests/backup.sx). Another connection holds the write lock for 7 s, and a write on a connection sak's pool opens (db.connect) lands once the lock is let go.src/db.sxwith this test andsrc/backup.sxcopied in printedFAIL its write waits out the other's commit, held 7 s, and landsandBACKUP FAIL (1 checks).tests/deploy_nas.sx): pull, then the backup with the pulled sak, then the start. The script names the backup it takes, and a failed backup starts nothing.The NAS's disk load cannot be reproduced on a runner's SSD, so no test shows a write failing under VACUUM INTO.
Gates
fullpassed atd8b92157, the head, at the af8455c3 pin (gate run 679): the 67tests/*.sxfiles of its loop, release.sh, step9, the deploy contract, the main Playwright config (771 passed) and the runner config (10 passed). This head merges onto master8b9a135fwith no conflict.fullfailed atd8b92157at the old pin (gate run 661), atsx build src/main.sx: sx 49e83601 has noprogressHandlerordbStatus.fast649 andfull651 passed ata01f2351;fast637 andfull638 atdfdfd012,full639 ate569ba92.