Parallelize filesystem walks across System/Certificates and remove duplicate cert scans (v0.16.25)

README.md

@@ -12,7 +12,7 @@
     <img alt="Docker" src="https://img.shields.io/badge/Docker-ready-2496ED?logo=docker&logoColor=white">
     <img alt="Architectures" src="https://img.shields.io/badge/platform-amd64%20%7C%20arm64-5965F2">
     <img alt="Caddy" src="https://img.shields.io/badge/powered%20by-Caddy-1F88C0">
-    <img alt="Version" src="https://img.shields.io/badge/version-0.16.22-62E6A7">
+    <img alt="Version" src="https://img.shields.io/badge/version-0.16.25-62E6A7">
   </p>
   <p>
     <a href="#why-site-gateway">Why Site Gateway</a> ·

ROADMAP.md

@@ -170,3 +170,9 @@ Roughly in priority order:
 `v0.16.21` gives API Access tokens full parity with every other tile type. Tokens now get a real, persistent custom icon -- a new `icon`/`icon_slug` column pair on the `api_tokens` table (added via an idempotent `ALTER TABLE`, safe on existing installs), matching storage functions, and a `tokens` branch in the shared icon-upload/search/URL routes -- plus the same "•••" card menu every other tile has, with Change icon and Revoke token moved into it. While wiring this up, found and fixed a real pre-existing bug: Groups' own "Change icon" menu item has been broken since it shipped, because the frontend code that actually saves an icon never mapped the `groups` kind to anything and silently fell through to the Hosted Sites endpoint, which always 404'd. Also finished the rest of the API Access fix list: the Full access/Read-only counts in the summary bar now only tally active tokens, so they stay consistent with the Active/Revoked split instead of quietly including tokens that can no longer authenticate; a "Hide revoked" toggle sits at the right of that same summary bar for anyone who's revoked enough tokens over time that the tile grid gets cluttered; and the documentation now explains why a revoked token can't be deleted outright -- the record stays for the same accountability reasons the Audit log is never editable.
 
 `v0.16.22` fixes Docker socket detection for the common case where `/var/run/docker.sock` is correctly bind-mounted but Site Gateway still reports "not detected." Root cause: `detectDockerSocket()` checks that the running process can actually read the socket, but the container drops straight from root to the unprivileged `PUID:PGID` with no supplementary groups, and the socket is typically owned `root:docker` on the host with mode 660 -- so a perfectly correct mount still fails an unprivileged read check with no group membership behind it. `docker-entrypoint.sh` now handles this automatically: while still root, it reads the socket's actual group GID directly off the mount (no hardcoded GID -- it varies by host, Unraid, Debian, Synology, and others all differ), creates a matching local group if one doesn't already exist, adds the app user to it, and hands `su-exec` a username instead of a bare `uid:gid` so supplementary groups actually apply via `initgroups()`. Every step is best-effort and guarded: if anything about the detection or group setup fails, the container starts exactly as it always has, just without Docker integration, the same as if the socket weren't mounted at all. Also documented in the System tab's Environment & Integrations section, including the one thing this can't route around: the check runs once at boot, so a container that already has the mount added still needs an actual restart, not just a reload, to pick it up.
+
+`v0.16.23` fixes the "Restart application" button on the System tab doing nothing at all after you confirm the restart in its popup: the dialog closes, the button text never changes to "Restarting...", no toast appears, and no restart actually happens -- explaining why the earlier restart-not-logged report showed no trace anywhere (no activity entry, no audit entry, no fresh boot sequence in the container's own console log), because the request never reached the server in the first place. Root cause: `event.currentTarget` is only valid while a DOM event is still being dispatched -- the browser resets it to `null` once dispatch finishes. The Restart handler read `event.currentTarget` *after* `await`-ing the confirmation dialog, by which point the click event had long since finished dispatching, so that line threw against a null reference before ever reaching the `/api/system/restart` call, and the error had nowhere to surface since it happened outside the handler's own try/catch. Resync and Reload were never affected because both of those capture their button reference as their very first line, before any `await`. Fixed by capturing the button reference synchronously at the top of the Restart handler too, matching the other two.
+
+`v0.16.24` fixes the "Restart application" button never recovering after a successful restart -- following v0.16.23's fix for the button doing nothing at all, a real restart now goes through correctly (the audit log and activity feed both record it as expected), but the button itself was left stuck on "Restarting..." forever, since nothing in the success path ever reset it or reloaded the page. The handler assumed the toast alone was enough and stopped there, unlike the Danger Zone's Factory Reset flow, which already polls for the server coming back online and reloads automatically. Restart now does the same: once the restart request is accepted, it polls `/api/session` once a second for up to 30 seconds and reloads the page as soon as the dashboard answers again (with a status line explaining what it's waiting on), falling back to a reload regardless if that window elapses -- so the button, and the rest of the UI, recover on their own instead of requiring a manual page refresh.
+
+`v0.16.25` parallelizes every sequential filesystem walk found across the app after noticing the System tab's storage numbers took a while to appear -- the same pattern turned up on the Certificates tab too, and both are fixed the same way. `directorySize()` (the System tab's disk-usage breakdown) and `walkFiles()` (the Certificates tab's search for every issued certificate file) both used to visit one file or subdirectory at a time, `await`-ing each in turn before moving to the next -- on a data directory with any real number of files, that adds up to a lot of small sequential waits. Both now fan out with `Promise.all` and let the filesystem handle everything concurrently, with no change to what they return. The System tab's five-directory breakdown (sites, backups, certificates, logs, database) is now computed in parallel too, instead of one directory at a time. While tracing the Certificates tab's load time, also found and fixed a real duplicate-work bug: the "Run certificate check" button and the downloadable support report were each independently computing the certificate inventory two to three times per request (`dashboardSnapshot()`, the route handler, and `domainReadiness()` each walked and re-parsed every certificate file separately) -- `dashboardSnapshot()` and `domainReadiness()` now both accept an already-computed inventory and reuse it instead of recomputing it, and the two independent halves of a health check (the dashboard snapshot and the domain-readiness check) now run concurrently rather than one after the other. Deliberately left alone: the code paths that read log files (`readAccessLogs`) and start hosted sites/streams on boot, since both are sequential for real reasons -- the log reader stops as soon as it has enough matching entries, so reading files in parallel would do strictly more work for no benefit, and site/stream startup order matters for safe, predictable port binding.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "site-gateway",
-  "version": "0.16.22",
+  "version": "0.16.25",
   "private": true,
   "description": "Site Gateway: simple self-hosted website publishing, reverse proxying, and automatic HTTPS.",
   "type": "module",

src/public/features.js

@@ -604,9 +604,24 @@ function renderSystemPanel() {
       finally { button.disabled = false; button.textContent = original; }
     });
     panel.querySelector("#system-restart").addEventListener("click", async event => {
+      const button = event.currentTarget;
       if (!await themedConfirm("Restart Site Gateway?", "The application will stop and restart. This takes a few seconds and briefly interrupts hosted sites and the dashboard.", "Restart")) return;
-      const button = event.currentTarget; button.disabled = true; button.textContent = "Restarting\u2026";
-      try { await api("/api/system/restart", { method: "POST" }); toast("Restarting \u2014 this dashboard will be unavailable briefly."); }
+      button.disabled = true; button.textContent = "Restarting\u2026";
+      const restartStatus = document.querySelector("#system-restart-status");
+      try {
+        await api("/api/system/restart", { method: "POST" });
+        toast("Restarting \u2014 this dashboard will be unavailable briefly.");
+        button.textContent = "Waiting for Site Gateway\u2026";
+        if (restartStatus) restartStatus.textContent = "Reconnecting once Site Gateway comes back online\u2026";
+        for (let attempt = 0; attempt < 30; attempt++) {
+          await new Promise(resolve => setTimeout(resolve, 1000));
+          try {
+            const response = await fetch("/api/session", { cache: "no-store" });
+            if (response.ok) { location.reload(); return; }
+          } catch { /* Still restarting -- the dashboard is briefly unreachable while the container comes back up. */ }
+        }
+        location.reload();
+      }
       catch (error) { toast(error.message, "error"); button.disabled = false; button.textContent = "Restart application"; }
     });
   }

src/public/index.html

@@ -8,7 +8,7 @@
   <title>Site Gateway</title>
   <meta name="description" content="Host sites, proxy services, and manage HTTPS from one simple dashboard.">
   <link rel="icon" type="image/png" href="/site-gateway-icon-approved.png">
-  <link rel="stylesheet" href="/styles.css?v=0.16.22">
+  <link rel="stylesheet" href="/styles.css?v=0.16.25">
 </head>
 
 <!-- ================================================================
@@ -434,6 +434,6 @@
   <div id="toast" class="toast" role="status"></div>
   <div id="update-banner" class="update-banner hidden" role="status"><span>A new version of Site Gateway is available.</span><div class="update-banner-actions"><button id="update-banner-refresh" class="button primary">Refresh</button><button id="update-banner-dismiss" class="text-button">Dismiss</button></div></div>
   <!-- App scripts: core (app.js) then extended views/admin (features.js) -->
-  <script src="/app.js?v=0.16.22" defer></script><script src="/features.js?v=0.16.22" defer></script><script src="/select-enhance.js?v=0.16.22" defer></script>
+  <script src="/app.js?v=0.16.25" defer></script><script src="/features.js?v=0.16.25" defer></script><script src="/select-enhance.js?v=0.16.25" defer></script>
 </body>
 </html>

src/server.js

@@ -83,14 +83,14 @@ function recordActivity(message, status = "ok") {
 }
 
 async function directorySize(directory) {
-  let total = 0;
   const entries = await fsp.readdir(directory, { withFileTypes: true }).catch(error => error.code === "ENOENT" ? [] : Promise.reject(error));
-  for (const entry of entries) {
+  const sizes = await Promise.all(entries.map(async entry => {
     const itemPath = path.join(directory, entry.name);
-    if (entry.isDirectory()) total += await directorySize(itemPath);
-    else if (entry.isFile()) total += (await fsp.stat(itemPath)).size;
-  }
-  return total;
+    if (entry.isDirectory()) return directorySize(itemPath);
+    if (entry.isFile()) return (await fsp.stat(itemPath)).size;
+    return 0;
+  }));
+  return sizes.reduce((sum, size) => sum + size, 0);
 }
 
 function numberEnv(name, fallback) {
@@ -729,13 +729,14 @@ function publicStream(stream) {
 
 // --- Certificate inventory & domain readiness diagnostics --------------------------------------
 async function walkFiles(directory) {
-  const output = [];
-  for (const entry of await fsp.readdir(directory, { withFileTypes: true }).catch(error => error.code === "ENOENT" ? [] : Promise.reject(error))) {
+  const entries = await fsp.readdir(directory, { withFileTypes: true }).catch(error => error.code === "ENOENT" ? [] : Promise.reject(error));
+  const results = await Promise.all(entries.map(entry => {
     const fullPath = path.join(directory, entry.name);
-    if (entry.isDirectory()) output.push(...await walkFiles(fullPath));
-    else if (entry.isFile()) output.push(fullPath);
-  }
-  return output;
+    if (entry.isDirectory()) return walkFiles(fullPath);
+    if (entry.isFile()) return [fullPath];
+    return [];
+  }));
+  return results.flat();
 }
 
 function certificateNames(certificate) {
@@ -748,15 +749,15 @@ async function certificateInventory() {
   const configured = [...sites.map(item => ({ ...item, kind: "Hosted site" })), ...proxies.map(item => ({ ...item, kind: "Proxy host" })), ...redirects.map(item => ({ ...item, kind: "Redirect host" }))]
     .filter(item => item.enabled && item.domain && item.tls !== "http");
   const configuredDomains = configured.flatMap(item => normalizeDomains(item.domain, item.domains).map(domain => ({ ...item, domain })));
-  const parsed = [];
-  const certificateFiles = [...await walkFiles(certificateDir), ...await walkFiles(customCertificatesDir)];
-  for (const filename of certificateFiles.filter(file => /\.(?:crt|pem)$/i.test(file))) {
+  const [managedCertificateFiles, customCertificateFiles] = await Promise.all([walkFiles(certificateDir), walkFiles(customCertificatesDir)]);
+  const certificateFiles = [...managedCertificateFiles, ...customCertificateFiles];
+  const parsed = (await Promise.all(certificateFiles.filter(file => /\.(?:crt|pem)$/i.test(file)).map(async filename => {
     try {
-      const certificate = new crypto.X509Certificate(await fsp.readFile(filename));
-      const stat = await fsp.stat(filename);
-      parsed.push({ certificate, names: certificateNames(certificate), updatedAt: stat.mtime.toISOString(), filename, source: filename.startsWith(customCertificatesDir) ? "Custom upload" : "Caddy / ACME" });
-    } catch { /* Ignore non-certificate PEM files and unreadable entries. */ }
-  }
+      const [contents, stat] = await Promise.all([fsp.readFile(filename), fsp.stat(filename)]);
+      const certificate = new crypto.X509Certificate(contents);
+      return { certificate, names: certificateNames(certificate), updatedAt: stat.mtime.toISOString(), filename, source: filename.startsWith(customCertificatesDir) ? "Custom upload" : "Caddy / ACME" };
+    } catch { return null; /* Ignore non-certificate PEM files and unreadable entries. */ }
+  }))).filter(Boolean);
   const certificates = configuredDomains.map(item => {
     const found = parsed.find(entry => entry.names.some(name => name === item.domain || (name.startsWith("*.") && item.domain.endsWith(name.slice(1)))));
     if (!found) {
@@ -793,10 +794,9 @@ async function pruneOrphanedCertificates(candidateDomains) {
 }
 
 
-async function domainReadiness() {
+async function domainReadiness(precomputedCertificates) {
   const routes = [...sites.map(item => ({ ...item, kind: "Hosted site" })), ...proxies.map(item => ({ ...item, kind: "Proxy host" })), ...redirects.map(item => ({ ...item, kind: "Redirect host" }))].filter(item => item.enabled && item.domain).flatMap(item => normalizeDomains(item.domain, item.domains).map(domain => ({ ...item, domain })));
-  const certs = await certificateInventory();
-  const [httpResponding, httpsResponding] = await Promise.all([tcpProbe(80), tcpProbe(443)]);
+  const [certs, httpResponding, httpsResponding] = await Promise.all([precomputedCertificates ? Promise.resolve(precomputedCertificates) : certificateInventory(), tcpProbe(80), tcpProbe(443)]);
   return Promise.all(routes.map(async item => {
     let addresses = [], dnsError = null;
     try { addresses = [...new Set((await dns.lookup(item.domain, { all: true })).map(value => value.address))]; } catch (error) { dnsError = error.code || error.message; }
@@ -954,12 +954,12 @@ async function cacheIcon(slug) {
 
 // --- Dashboard snapshot: aggregates health/status across every subsystem for the
 //     Overview page and the /api/dashboard endpoint --------------------------------------------
-async function dashboardSnapshot() {
+async function dashboardSnapshot(precomputedCertificates) {
   const hosted = sites.map(publicSite);
   const proxyHosts = proxies.map(publicProxy);
   const enabledStreams = streams.filter(item => item.enabled !== false);
   const streamingPorts = { total: enabledStreams.length, listening: enabledStreams.filter(item => activeStreams.has(item.id)).length };
-  const certificates = await certificateInventory();
+  const certificates = precomputedCertificates || await certificateInventory();
   const tlsDomains = [...sites, ...proxies].filter(item => item.enabled && item.domain && item.tls !== "http").length;
   const [storageWritable, gatewayResponding, httpResponding, httpsResponding] = await Promise.all([
     fsp.access(dataDir, fs.constants.R_OK | fs.constants.W_OK).then(() => true).catch(() => false),
@@ -1584,10 +1584,8 @@ app.get("/api/config", (req, res) => res.json({ version: appVersion, minPort, ma
 app.get("/api/system/storage", async (req, res, next) => {
   if (req.user.role !== "administrator") return res.status(403).json({ error: "Administrator access is required." });
   try {
-    const breakdown = {};
-    for (const [key, dir] of Object.entries({ sites: sitesDir, backups: backupsDir, certificates: certificatesRoot, logs: logsDir, database: path.join(dataDir, "database") })) {
-      breakdown[key] = await directorySize(dir);
-    }
+    const breakdownDirs = { sites: sitesDir, backups: backupsDir, certificates: certificatesRoot, logs: logsDir, database: path.join(dataDir, "database") };
+    const breakdown = Object.fromEntries(await Promise.all(Object.entries(breakdownDirs).map(async ([key, dir]) => [key, await directorySize(dir)])));
     let capacity = null;
     try {
       const stats = await fsp.statfs(dataDir);
@@ -1783,7 +1781,12 @@ app.get("/api/certificates", async (req, res, next) => {
   catch (error) { next(error); }
 });
 app.post("/api/health/check", async (req, res, next) => {
-  try { await checkAllProxies(); res.json({ dashboard: await dashboardSnapshot(), certificates: await certificateInventory(), readiness: await domainReadiness() }); }
+  try {
+    await checkAllProxies();
+    const certificates = await certificateInventory();
+    const [dashboard, readiness] = await Promise.all([dashboardSnapshot(certificates), domainReadiness(certificates)]);
+    res.json({ dashboard, certificates, readiness });
+  }
   catch (error) { next(error); }
 });
 app.get("/api/readiness", async (req, res, next) => { try { res.json({ checkedAt: new Date().toISOString(), routes: await domainReadiness() }); } catch (error) { next(error); } });
@@ -1795,8 +1798,9 @@ app.get("/api/support-report", async (req, res, next) => {
   try {
     if (req.user.role !== "administrator") return res.status(403).json({ error: "Administrator access is required." });
     const certificateReport = await certificateInventory();
+    const readiness = await domainReadiness(certificateReport);
     certificateReport.latestError = certificateReport.latestError ? { present:true, at:certificateReport.latestError.at } : null;
-    const report = { product: "Site Gateway", generatedAt: new Date().toISOString(), version: appVersion, caddyVersion, nodeVersion: process.version, storage: { engine: "SQLite", integrity: storage.integrity() }, gateway: { healthy: !gatewayError, lastReload: lastGatewayReload }, routes: { hosted: sites.map(({ id,name,domain,tls,enabled,port }) => ({ id,name,domain,tls,enabled,port })), proxies: proxies.map(({ id,name,domain,tls,enabled,target,healthEnabled,healthExpected }) => ({ id,name,domain,tls,enabled,target,healthEnabled,healthExpected })), redirects: redirects.map(({ id,name,domain,tls,enabled,code }) => ({ id,name,domain,tls,enabled,code })) }, certificates: certificateReport, readiness: await domainReadiness(), recentEvents: recentActivity.slice(0,20).map(item => ({ at:item.at, status:item.status, message:item.status === "error" ? "Operational error recorded; review the protected in-app event log for details." : item.message })) };
+    const report = { product: "Site Gateway", generatedAt: new Date().toISOString(), version: appVersion, caddyVersion, nodeVersion: process.version, storage: { engine: "SQLite", integrity: storage.integrity() }, gateway: { healthy: !gatewayError, lastReload: lastGatewayReload }, routes: { hosted: sites.map(({ id,name,domain,tls,enabled,port }) => ({ id,name,domain,tls,enabled,port })), proxies: proxies.map(({ id,name,domain,tls,enabled,target,healthEnabled,healthExpected }) => ({ id,name,domain,tls,enabled,target,healthEnabled,healthExpected })), redirects: redirects.map(({ id,name,domain,tls,enabled,code }) => ({ id,name,domain,tls,enabled,code })) }, certificates: certificateReport, readiness, recentEvents: recentActivity.slice(0,20).map(item => ({ at:item.at, status:item.status, message:item.status === "error" ? "Operational error recorded; review the protected in-app event log for details." : item.message })) };
     res.setHeader("Content-Disposition", `attachment; filename="site-gateway-support-${new Date().toISOString().slice(0,10)}.json"`); res.type("json").send(JSON.stringify(report, null, 2));
   } catch (error) { next(error); }
 });