Offsite-Backup: Archiv als Datei streamen statt im Speicher halten

.github/workflows/test.yml

@@ -63,8 +63,9 @@ jobs:
           retention-days: 3
           if-no-files-found: ignore
 
-  # Storage targets (docs/storage-targets.md) against real servers: S3
-  # (moto), WebDAV (wsgidav) and SMB (Samba with smbclient, as in the image).
+  # Storage targets (docs/storage-targets.md) and the off-site backup
+  # transports (services/offsite) against real servers: S3 (moto), WebDAV
+  # (wsgidav) and SMB (Samba with smbclient, as in the image).
   storage-targets:
     name: storage targets (S3, WebDAV, SMB)
     runs-on: ubuntu-latest
@@ -108,8 +109,8 @@ jobs:
           done
           curl -sf -X PUT http://127.0.0.1:5055/gc-test-bucket
 
-      - name: Run the live storage target tests
-        run: node --test --test-force-exit tests/storage_targets_live.test.js
+      - name: Run the live storage target and off-site tests
+        run: node --test --test-force-exit tests/storage_targets_live.test.js tests/offsite_live.test.js
         env:
           NODE_ENV: test
           GC_TEST_S3_ENDPOINT: http://127.0.0.1:5055

docs/feature-release-b.md

@@ -177,6 +177,16 @@ Einstellungen `GET/PUT /api/v1/settings/waf`:
   password}`. Antworten enthalten nie Geheimnisse (`has_password: true`).
 - Upload nach jedem geplanten Backup an alle aktiven Ziele, Aufbewahrung `keep`
   (nur eigene Dateien `gatecontrol-*.gcbk` löschen), Ergebnis in `last_*`, SSE `backup`.
+- Das Archiv ist nie als Ganzes im Speicher: `gcbk.writeBackupArchive` schreibt es als
+  Stream in eine Datei im privaten Ordner `<Backup-Ordner>/.incoming` (Name vom Server,
+  zufällig; vorher Prüfung des freien Platzes, sonst `NO_SPACE`/507, Hinweis `no_space`;
+  Löschen im `finally`, Reste eines Absturzes beim ersten Gebrauch). Die Transporte senden
+  aus der Datei (`services/offsite/source.js`): WebDAV ein PUT als Stream mit
+  Content-Length; S3 bis 64 MiB ein PUT, darüber Multipart in 64-MiB-Teilen (jeder Teil mit
+  eigenem SHA-256 signiert, beim Schreiben des Archivs berechnet; Abbruch per
+  AbortMultipartUpload bei einem Fehler); SMB/SFTP `put` direkt aus der Datei (Zeitlimit
+  wächst mit der Größe). Der Wiederherstellungstest lädt per `downloadTo` in eine Datei.
+  Messung: `node scripts/offsite-memory-check.js --mb 400`.
 
 ## 8. Navigation und Schnellsuche (UI B4)
 

docs/guide/02-server-setup.md

@@ -244,6 +244,10 @@ backup. For a NAS at home behind a gateway, choose “Through a VPN device” or
 target; “Take over a storage target from Drive/plugins” fills in its
 settings. See
 [FAQ: Off-site backup to a NAS behind the VPN](07-faq.md#off-site-backup-to-a-nas-behind-the-vpn).
+The encrypted copy is stored briefly on the server's disk next to the
+backups while it is uploaded (it is never held in memory, so a small VPS is
+fine) – plan free space for about one more backup. If it is missing,
+GateControl uploads nothing and says so at the target.
 
 The connection is always encrypted, without any certificate file:
 

docs/handbuch/02-server-einrichten.md

@@ -245,6 +245,10 @@ S3 oder WebDAV hoch. Steht die NAS zu Hause hinter einem Gateway, wähle bei
 beim Speicherziel für Drive; „Speicherziel von Drive/Plugins übernehmen“
 trägt dessen Einstellungen ein. Siehe
 [FAQ: Offsite-Backup auf eine NAS hinter dem VPN](07-faq.md#offsite-backup-auf-eine-nas-hinter-dem-vpn).
+Die verschlüsselte Kopie liegt während des Hochladens kurz auf der Platte
+des Servers neben den Backups (nie als Ganzes im Arbeitsspeicher, ein kleiner
+VPS reicht also) – plane Platz für etwa ein Backup mehr ein. Fehlt er, lädt
+GateControl nichts hoch und sagt es beim Ziel.
 
 Die Verbindung ist immer verschlüsselt, ohne dass du eine Zertifikatsdatei
 brauchst:

public/js/ops-ui.js

@@ -123,6 +123,7 @@
     INVALID_FINGERPRINT: ['offsite.err.generic', 'That did not work.'],
     NOT_GCBK: ['offsite.err.corrupt', 'The file is damaged or not a GateControl backup.'],
     UNSUPPORTED: ['offsite.err.unsupported', 'This version cannot read that archive format.'],
+    NO_SPACE: ['offsite.err.no_space', 'There is not enough free disk space on the GateControl server to store the encrypted archive temporarily.'],
     INVALID_WINDOW: ['autoupdate.err.invalid_window', 'Invalid maintenance window: times as HH:MM, start ≠ end, a valid time zone.'],
     INVALID_MODE: ['autoupdate.err.generic', 'Could not save.'],
     INVALID_NOTIFY_EMAIL: ['autoupdate.err.generic', 'Could not save.'],
@@ -151,6 +152,8 @@
     cert_not_yet_valid: ['offsite.hint.cert_not_yet_valid', 'The NAS certificate is not valid yet. Check the clock of the NAS and of GateControl.'],
     cert_decrypts: ['offsite.hint.cert_decrypts', 'The route decrypts the connection itself. Set the TLS mode of the L4 route to pass-through.'],
     smb_no_encryption: ['offsite.hint.smb_no_encryption', 'The share cannot encrypt (SMB3). On Synology: Control Panel › File Services › SMB › Advanced → SMB3 encryption. Or switch off “Encrypt connection”.'],
+    // the server's own disk: the archive is stored there briefly before the upload
+    no_space: ['offsite.hint.no_space', 'There is too little free disk space on the GateControl server. The encrypted archive is stored there briefly before the upload. Delete old local backups (Settings › Backup) or free up space.'],
   };
   /** The hint of an API answer / a target row, '' when none. */
   function hintText(code) {

scripts/offsite-memory-check.js

@@ -0,0 +1,149 @@
+#!/usr/bin/env node
+'use strict';
+
+// Peak memory of an off-site backup upload (services/offsite) — the archive
+// written as a file and sent from read streams, against holding it as one
+// Buffer as up to 1.258.1 ("--legacy").
+//
+//   node scripts/offsite-memory-check.js [--mb 400] [--legacy] [--part-mb 16]
+//
+// Writes (in a child process) a backup file with <mb> MB of incompressible plugin data (as
+// base64, as in a real backup) into a temp folder, then builds the archive
+// and uploads it to a WebDAV and an S3 server running in this process (both
+// only count and hash what arrives — the S3 one with a multipart upload
+// above --part-mb). RSS is sampled every 10 ms from the start of the build
+// to the end of the uploads. Prints one JSON line:
+//   { mode, plugin_mb, archive_mb, baseline_rss_mb, peak_rss_mb, extra_mb, max_rss_mb, ok }
+// extra_mb = peak − baseline: what the archive transfer added.
+//
+// tests/offsite_memory.test.js runs it with a smaller size and checks the
+// bound; the numbers for the PR come from a run with --mb 400.
+
+process.env.NODE_ENV = process.env.NODE_ENV || 'test';
+process.env.GC_LOG_LEVEL = process.env.GC_LOG_LEVEL || 'silent';
+
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const http = require('node:http');
+const crypto = require('node:crypto');
+
+const args = process.argv.slice(2);
+const opt = (name, dflt) => { const i = args.indexOf(`--${name}`); return i === -1 ? dflt : Number(args[i + 1]); };
+const MB = 1024 * 1024;
+const PLUGIN_MB = opt('mb', 400);
+const PART_MB = opt('part-mb', 16);
+const LEGACY = args.includes('--legacy');
+
+const gcbk = require('../src/services/offsite/gcbk');
+const webdav = require('../src/services/offsite/webdav');
+const s3 = require('../src/services/offsite/s3');
+
+const rssMb = () => process.memoryUsage().rss / MB;
+
+/** The backup file, written piece by piece (its own RSS stays small). */
+async function writeBackup(file) {
+  const out = fs.createWriteStream(file);
+  const write = (s) => new Promise((resolve) => { if (out.write(s)) resolve(); else out.once('drain', resolve); });
+  await write('{\n  "version": 1,\n  "data": {\n    "settings": [],\n    "plugins": [\n      {\n        "id": "big",\n        "db": "');
+  // base64 of random bytes in pieces of 3 MiB (a multiple of 3: the pieces join to one valid base64 text)
+  for (let left = PLUGIN_MB * MB; left > 0; left -= 3 * MB) await write(crypto.randomBytes(Math.min(left, 3 * MB)).toString('base64'));
+  await write('"\n      }\n    ]\n  }\n}\n');
+  await new Promise((resolve, reject) => out.end((err) => (err ? reject(err) : resolve())));
+}
+
+/** A server that takes WebDAV and S3 requests and only counts and hashes the bodies. */
+function sink() {
+  const got = { bytes: 0, objects: new Map() }; // name → { size, sha }
+  const uploads = new Map();
+  const server = http.createServer((req, res) => {
+    const h = crypto.createHash('sha256');
+    let n = 0;
+    req.on('data', (c) => { n += c.length; h.update(c); });
+    req.on('end', () => {
+      got.bytes += n;
+      const u = new URL(req.url, 'http://x');
+      const name = decodeURIComponent(u.pathname.split('/').pop());
+      if (req.method === 'PROPFIND') { res.writeHead(207); res.end('<d:multistatus xmlns:d="DAV:"></d:multistatus>'); return; }
+      if (req.method === 'POST' && u.searchParams.has('uploads')) {
+        const id = crypto.randomBytes(6).toString('hex');
+        uploads.set(id, 0);
+        res.writeHead(200); res.end(`<InitiateMultipartUploadResult><UploadId>${id}</UploadId></InitiateMultipartUploadResult>`); return;
+      }
+      if (req.method === 'PUT' && u.searchParams.has('uploadId')) {
+        uploads.set(u.searchParams.get('uploadId'), uploads.get(u.searchParams.get('uploadId')) + n);
+        res.writeHead(200, { etag: `"${h.digest('hex').slice(0, 32)}"` }); res.end(); return;
+      }
+      if (req.method === 'POST' && u.searchParams.has('uploadId')) {
+        got.objects.set(name, { size: uploads.get(u.searchParams.get('uploadId')), sha: null });
+        res.writeHead(200); res.end('<CompleteMultipartUploadResult></CompleteMultipartUploadResult>'); return;
+      }
+      if (req.method === 'PUT') { got.objects.set(name, { size: n, sha: h.digest('hex') }); res.writeHead(201); res.end(); return; }
+      res.writeHead(204); res.end();
+    });
+  });
+  return new Promise((resolve) => server.listen(0, '127.0.0.1', () => resolve({ server, got, base: `http://127.0.0.1:${server.address().port}` })));
+}
+
+async function main() {
+  const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gc-offsite-mem-'));
+  const { server, got, base } = await sink();
+  try {
+    const json = path.join(dir, 'gatecontrol-20261109-030000.json');
+    // the input in a child process: its memory never shows in this one's RSS
+    require('node:child_process').execFileSync(process.execPath, [__filename, '--mb', String(PLUGIN_MB), '--write-input', json], { stdio: 'inherit' });
+    const davCfg = { url: `${base}/dav/`, username: 'u', password: 'p' };
+    const s3Cfg = { endpoint: base, region: 'us-east-1', bucket: 'bk', prefix: 'gc', access_key_id: 'AK', secret_access_key: 'SK', path_style: true };
+    s3._limits.part = PART_MB * MB;
+    const opts = { passphrase: 'memory check passphrase', kdf: { N: 1024, r: 8, p: 1 }, gcVersion: 'mem' };
+
+    if (global.gc) global.gc();
+    await new Promise((r) => setTimeout(r, 200));
+    const baseline = rssMb();
+    let peak = baseline;
+    const timer = setInterval(() => { peak = Math.max(peak, rssMb()); }, 10);
+    let archiveSize;
+    let sha;
+    try {
+      if (LEGACY) {
+        // up to 1.258.1: the archive as one Buffer, handed to every transport
+        const buf = await gcbk.encryptBackupFile(json, opts);
+        archiveSize = buf.length;
+        sha = crypto.createHash('sha256').update(buf).digest('hex');
+        await webdav.upload(davCfg, 'dav.gcbk', buf);
+        await s3.upload(s3Cfg, 's3.gcbk', buf);
+      } else {
+        const file = path.join(dir, 'archive.gcbk');
+        const r = await gcbk.writeBackupArchive(json, file, { ...opts, partSize: s3._limits.part });
+        archiveSize = r.size;
+        const archive = { path: file, ...r };
+        await webdav.upload(davCfg, 'dav.gcbk', archive);
+        await s3.upload(s3Cfg, 's3.gcbk', archive);
+        sha = crypto.createHash('sha256');
+        for await (const c of fs.createReadStream(file)) sha.update(c);
+        sha = sha.digest('hex');
+      }
+    } finally { clearInterval(timer); }
+    peak = Math.max(peak, rssMb());
+    const dav = got.objects.get('dav.gcbk');
+    const up = got.objects.get('s3.gcbk');
+    const ok = !!dav && !!up && dav.size === archiveSize && up.size === archiveSize && dav.sha === sha;
+    const r1 = (n) => Math.round(n * 10) / 10;
+    process.stdout.write(JSON.stringify({
+      mode: LEGACY ? 'legacy (Buffer)' : 'stream (file)',
+      plugin_mb: PLUGIN_MB, archive_mb: r1(archiveSize / MB), part_mb: PART_MB,
+      baseline_rss_mb: r1(baseline), peak_rss_mb: r1(peak), extra_mb: r1(peak - baseline),
+      max_rss_mb: r1(process.resourceUsage().maxRSS / 1024), ok,
+    }) + '\n');
+    if (!ok) process.exitCode = 1;
+  } finally {
+    server.close();
+    fs.rmSync(dir, { recursive: true, force: true });
+  }
+}
+
+if (args.includes('--write-input')) {
+  writeBackup(args[args.indexOf('--write-input') + 1]).catch((err) => { process.stderr.write(`${err.stack || err}\n`); process.exit(1); });
+} else {
+  main().catch((err) => { process.stderr.write(`${err.stack || err}\n`); process.exit(1); });
+}

src/i18n/de.json

@@ -1830,6 +1830,7 @@
   "offsite.err.cert_mismatch": "Die NAS zeigt inzwischen ein anderes Zertifikat. Bitte noch einmal testen.",
   "offsite.err.not_trustable": "Diesem Zertifikat kann hier nicht vertraut werden. „Testen“ zeigt, warum.",
   "offsite.err.unsupported": "Dieses Archivformat kann diese Version nicht lesen.",
+  "offsite.err.no_space": "Auf dem GateControl-Server ist nicht genug Speicherplatz frei, um das verschlüsselte Archiv zwischenzuspeichern.",
   "offsite.err.generic": "Das hat nicht geklappt.",
   "offsite.f_via": "Verbindung",
   "offsite.via_url": "Direkt zur Adresse",
@@ -1902,6 +1903,7 @@
   "offsite.hint.cert_not_yet_valid": "Das Zertifikat der NAS gilt noch nicht. Prüfe die Uhrzeit von NAS und GateControl.",
   "offsite.hint.cert_decrypts": "Die Weiterleitung entschlüsselt die Verbindung selbst. Stelle bei der L4-Weiterleitung den TLS-Modus auf „Durchleitung“ oder „Keiner“.",
   "offsite.hint.smb_no_encryption": "Die Freigabe kann keine Verschlüsselung (SMB3). Bei Synology: Systemsteuerung › Dateidienste › SMB › Erweitert → SMB3-Verschlüsselung. Oder unter „Bearbeiten“ „Verbindung verschlüsseln (SMB3)“ ausschalten.",
+  "offsite.hint.no_space": "Auf dem GateControl-Server ist zu wenig Speicherplatz frei. Das verschlüsselte Archiv wird vor dem Hochladen kurz dort abgelegt. Lösche alte lokale Backups (Einstellungen › Backup) oder gib Speicher frei.",
   "premig.title": "Sicherungen vor Datenbank-Updates",
   "premig.empty": "Noch keine – die erste entsteht beim nächsten Update mit Datenbank-Änderung.",
   "premig.versions": "Schema {from} → {to}",

src/i18n/en.json

@@ -1830,6 +1830,7 @@
   "offsite.err.cert_mismatch": "The NAS now shows a different certificate. Please test again.",
   "offsite.err.not_trustable": "This certificate cannot be trusted here. “Test” shows why.",
   "offsite.err.unsupported": "This version cannot read that archive format.",
+  "offsite.err.no_space": "There is not enough free disk space on the GateControl server to store the encrypted archive temporarily.",
   "offsite.err.generic": "That did not work.",
   "offsite.f_via": "Connection",
   "offsite.via_url": "Directly to the address",
@@ -1902,6 +1903,7 @@
   "offsite.hint.cert_not_yet_valid": "The NAS certificate is not valid yet. Check the clock of the NAS and of GateControl.",
   "offsite.hint.cert_decrypts": "The route decrypts the connection itself. Set the TLS mode of the L4 route to “Passthrough” or “None”.",
   "offsite.hint.smb_no_encryption": "The share cannot encrypt (SMB3). On Synology: Control Panel › File Services › SMB › Advanced → SMB3 encryption. Or switch off “Encrypt connection (SMB3)” under “Edit”.",
+  "offsite.hint.no_space": "There is too little free disk space on the GateControl server. The encrypted archive is stored there briefly before the upload. Delete old local backups (Settings › Backup) or free up space.",
   "premig.title": "Snapshots before database updates",
   "premig.empty": "None yet — the first one is taken on the next update that changes the database.",
   "premig.versions": "Schema {from} → {to}",

src/services/backupUpload.js

@@ -7,6 +7,9 @@
 // An off-site archive (.gcbk) is decrypted into a second file next to it
 // (gcbk.decryptBackupFileTo). Both files stay until the request is done:
 // the large plugin parts of the parsed backup are still read from them.
+// The off-site backups use the folder too (services/offsite/index.js): the
+// archive of an upload ("<random>.offsite") and the one a restore test
+// downloads ("<random>.verify").
 
 const fs = require('node:fs');
 const path = require('node:path');
@@ -49,7 +52,7 @@ function storage() {
 }
 
 // Names of the files in the folder: always made here (random hex), never taken from a request.
-const NAME_RE = /^[0-9a-f]{24}\.(upload|verify)$/;
+const NAME_RE = /^[0-9a-f]{24}\.(upload|verify|offsite)$/;
 
 /** A file of the incoming folder by its NAME (checked, below the folder) → absolute path; throws otherwise. */
 function incomingPath(name) {