Parallelize filesystem walks across System/Certificates and remove duplicate cert scans (v0.16.25)

README.md

@@ -12,7 +12,7 @@
     <img alt="Docker" src="https://img.shields.io/badge/Docker-ready-2496ED?logo=docker&logoColor=white">
     <img alt="Architectures" src="https://img.shields.io/badge/platform-amd64%20%7C%20arm64-5965F2">
     <img alt="Caddy" src="https://img.shields.io/badge/powered%20by-Caddy-1F88C0">
-    <img alt="Version" src="https://img.shields.io/badge/version-0.16.24-62E6A7">
+    <img alt="Version" src="https://img.shields.io/badge/version-0.16.25-62E6A7">
   </p>
   <p>
     <a href="#why-site-gateway">Why Site Gateway</a> ·

ROADMAP.md

@@ -174,3 +174,5 @@ Roughly in priority order:
 `v0.16.23` fixes the "Restart application" button on the System tab doing nothing at all after you confirm the restart in its popup: the dialog closes, the button text never changes to "Restarting...", no toast appears, and no restart actually happens -- explaining why the earlier restart-not-logged report showed no trace anywhere (no activity entry, no audit entry, no fresh boot sequence in the container's own console log), because the request never reached the server in the first place. Root cause: `event.currentTarget` is only valid while a DOM event is still being dispatched -- the browser resets it to `null` once dispatch finishes. The Restart handler read `event.currentTarget` *after* `await`-ing the confirmation dialog, by which point the click event had long since finished dispatching, so that line threw against a null reference before ever reaching the `/api/system/restart` call, and the error had nowhere to surface since it happened outside the handler's own try/catch. Resync and Reload were never affected because both of those capture their button reference as their very first line, before any `await`. Fixed by capturing the button reference synchronously at the top of the Restart handler too, matching the other two.
 
 `v0.16.24` fixes the "Restart application" button never recovering after a successful restart -- following v0.16.23's fix for the button doing nothing at all, a real restart now goes through correctly (the audit log and activity feed both record it as expected), but the button itself was left stuck on "Restarting..." forever, since nothing in the success path ever reset it or reloaded the page. The handler assumed the toast alone was enough and stopped there, unlike the Danger Zone's Factory Reset flow, which already polls for the server coming back online and reloads automatically. Restart now does the same: once the restart request is accepted, it polls `/api/session` once a second for up to 30 seconds and reloads the page as soon as the dashboard answers again (with a status line explaining what it's waiting on), falling back to a reload regardless if that window elapses -- so the button, and the rest of the UI, recover on their own instead of requiring a manual page refresh.
+
+`v0.16.25` parallelizes every sequential filesystem walk found across the app after noticing the System tab's storage numbers took a while to appear -- the same pattern turned up on the Certificates tab too, and both are fixed the same way. `directorySize()` (the System tab's disk-usage breakdown) and `walkFiles()` (the Certificates tab's search for every issued certificate file) both used to visit one file or subdirectory at a time, `await`-ing each in turn before moving to the next -- on a data directory with any real number of files, that adds up to a lot of small sequential waits. Both now fan out with `Promise.all` and let the filesystem handle everything concurrently, with no change to what they return. The System tab's five-directory breakdown (sites, backups, certificates, logs, database) is now computed in parallel too, instead of one directory at a time. While tracing the Certificates tab's load time, also found and fixed a real duplicate-work bug: the "Run certificate check" button and the downloadable support report were each independently computing the certificate inventory two to three times per request (`dashboardSnapshot()`, the route handler, and `domainReadiness()` each walked and re-parsed every certificate file separately) -- `dashboardSnapshot()` and `domainReadiness()` now both accept an already-computed inventory and reuse it instead of recomputing it, and the two independent halves of a health check (the dashboard snapshot and the domain-readiness check) now run concurrently rather than one after the other. Deliberately left alone: the code paths that read log files (`readAccessLogs`) and start hosted sites/streams on boot, since both are sequential for real reasons -- the log reader stops as soon as it has enough matching entries, so reading files in parallel would do strictly more work for no benefit, and site/stream startup order matters for safe, predictable port binding.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "site-gateway",
-  "version": "0.16.24",
+  "version": "0.16.25",
   "private": true,
   "description": "Site Gateway: simple self-hosted website publishing, reverse proxying, and automatic HTTPS.",
   "type": "module",

src/public/index.html

@@ -8,7 +8,7 @@
   <title>Site Gateway</title>
   <meta name="description" content="Host sites, proxy services, and manage HTTPS from one simple dashboard.">
   <link rel="icon" type="image/png" href="/site-gateway-icon-approved.png">
-  <link rel="stylesheet" href="/styles.css?v=0.16.24">
+  <link rel="stylesheet" href="/styles.css?v=0.16.25">
 </head>
 
 <!-- ================================================================
@@ -434,6 +434,6 @@
   <div id="toast" class="toast" role="status"></div>
   <div id="update-banner" class="update-banner hidden" role="status"><span>A new version of Site Gateway is available.</span><div class="update-banner-actions"><button id="update-banner-refresh" class="button primary">Refresh</button><button id="update-banner-dismiss" class="text-button">Dismiss</button></div></div>
   <!-- App scripts: core (app.js) then extended views/admin (features.js) -->
-  <script src="/app.js?v=0.16.24" defer></script><script src="/features.js?v=0.16.24" defer></script><script src="/select-enhance.js?v=0.16.24" defer></script>
+  <script src="/app.js?v=0.16.25" defer></script><script src="/features.js?v=0.16.25" defer></script><script src="/select-enhance.js?v=0.16.25" defer></script>
 </body>
 </html>

src/server.js

@@ -83,14 +83,14 @@ function recordActivity(message, status = "ok") {
 }
 
 async function directorySize(directory) {
-  let total = 0;
   const entries = await fsp.readdir(directory, { withFileTypes: true }).catch(error => error.code === "ENOENT" ? [] : Promise.reject(error));
-  for (const entry of entries) {
+  const sizes = await Promise.all(entries.map(async entry => {
     const itemPath = path.join(directory, entry.name);
-    if (entry.isDirectory()) total += await directorySize(itemPath);
+    if (entry.isDirectory()) return directorySize(itemPath);
-    else if (entry.isFile()) total += (await fsp.stat(itemPath)).size;
+    if (entry.isFile()) return (await fsp.stat(itemPath)).size;
-  }
+    return 0;
-  return total;
+  }));
+  return sizes.reduce((sum, size) => sum + size, 0);
 }
 
 function numberEnv(name, fallback) {
@@ -729,13 +729,14 @@ function publicStream(stream) {
 
 // --- Certificate inventory & domain readiness diagnostics --------------------------------------
 async function walkFiles(directory) {
-  const output = [];
+  const entries = await fsp.readdir(directory, { withFileTypes: true }).catch(error => error.code === "ENOENT" ? [] : Promise.reject(error));
-  for (const entry of await fsp.readdir(directory, { withFileTypes: true }).catch(error => error.code === "ENOENT" ? [] : Promise.reject(error))) {
+  const results = await Promise.all(entries.map(entry => {
     const fullPath = path.join(directory, entry.name);
-    if (entry.isDirectory()) output.push(...await walkFiles(fullPath));
+    if (entry.isDirectory()) return walkFiles(fullPath);
-    else if (entry.isFile()) output.push(fullPath);
+    if (entry.isFile()) return [fullPath];
-  }
+    return [];
-  return output;
+  }));
+  return results.flat();
 }
 
 function certificateNames(certificate) {
@@ -748,15 +749,15 @@ async function certificateInventory() {
   const configured = [...sites.map(item => ({ ...item, kind: "Hosted site" })), ...proxies.map(item => ({ ...item, kind: "Proxy host" })), ...redirects.map(item => ({ ...item, kind: "Redirect host" }))]
     .filter(item => item.enabled && item.domain && item.tls !== "http");
   const configuredDomains = configured.flatMap(item => normalizeDomains(item.domain, item.domains).map(domain => ({ ...item, domain })));
-  const parsed = [];
+  const [managedCertificateFiles, customCertificateFiles] = await Promise.all([walkFiles(certificateDir), walkFiles(customCertificatesDir)]);
-  const certificateFiles = [...await walkFiles(certificateDir), ...await walkFiles(customCertificatesDir)];
+  const certificateFiles = [...managedCertificateFiles, ...customCertificateFiles];
-  for (const filename of certificateFiles.filter(file => /\.(?:crt|pem)$/i.test(file))) {
+  const parsed = (await Promise.all(certificateFiles.filter(file => /\.(?:crt|pem)$/i.test(file)).map(async filename => {
     try {
-      const certificate = new crypto.X509Certificate(await fsp.readFile(filename));
+      const [contents, stat] = await Promise.all([fsp.readFile(filename), fsp.stat(filename)]);
-      const stat = await fsp.stat(filename);
+      const certificate = new crypto.X509Certificate(contents);
-      parsed.push({ certificate, names: certificateNames(certificate), updatedAt: stat.mtime.toISOString(), filename, source: filename.startsWith(customCertificatesDir) ? "Custom upload" : "Caddy / ACME" });
+      return { certificate, names: certificateNames(certificate), updatedAt: stat.mtime.toISOString(), filename, source: filename.startsWith(customCertificatesDir) ? "Custom upload" : "Caddy / ACME" };
-    } catch { /* Ignore non-certificate PEM files and unreadable entries. */ }
+    } catch { return null; /* Ignore non-certificate PEM files and unreadable entries. */ }
-  }
+  }))).filter(Boolean);
   const certificates = configuredDomains.map(item => {
     const found = parsed.find(entry => entry.names.some(name => name === item.domain || (name.startsWith("*.") && item.domain.endsWith(name.slice(1)))));
     if (!found) {
@@ -793,10 +794,9 @@ async function pruneOrphanedCertificates(candidateDomains) {
 }
 
 
-async function domainReadiness() {
+async function domainReadiness(precomputedCertificates) {
   const routes = [...sites.map(item => ({ ...item, kind: "Hosted site" })), ...proxies.map(item => ({ ...item, kind: "Proxy host" })), ...redirects.map(item => ({ ...item, kind: "Redirect host" }))].filter(item => item.enabled && item.domain).flatMap(item => normalizeDomains(item.domain, item.domains).map(domain => ({ ...item, domain })));
-  const certs = await certificateInventory();
+  const [certs, httpResponding, httpsResponding] = await Promise.all([precomputedCertificates ? Promise.resolve(precomputedCertificates) : certificateInventory(), tcpProbe(80), tcpProbe(443)]);
-  const [httpResponding, httpsResponding] = await Promise.all([tcpProbe(80), tcpProbe(443)]);
   return Promise.all(routes.map(async item => {
     let addresses = [], dnsError = null;
     try { addresses = [...new Set((await dns.lookup(item.domain, { all: true })).map(value => value.address))]; } catch (error) { dnsError = error.code || error.message; }
@@ -954,12 +954,12 @@ async function cacheIcon(slug) {
 
 // --- Dashboard snapshot: aggregates health/status across every subsystem for the
 //     Overview page and the /api/dashboard endpoint --------------------------------------------
-async function dashboardSnapshot() {
+async function dashboardSnapshot(precomputedCertificates) {
   const hosted = sites.map(publicSite);
   const proxyHosts = proxies.map(publicProxy);
   const enabledStreams = streams.filter(item => item.enabled !== false);
   const streamingPorts = { total: enabledStreams.length, listening: enabledStreams.filter(item => activeStreams.has(item.id)).length };
-  const certificates = await certificateInventory();
+  const certificates = precomputedCertificates || await certificateInventory();
   const tlsDomains = [...sites, ...proxies].filter(item => item.enabled && item.domain && item.tls !== "http").length;
   const [storageWritable, gatewayResponding, httpResponding, httpsResponding] = await Promise.all([
     fsp.access(dataDir, fs.constants.R_OK | fs.constants.W_OK).then(() => true).catch(() => false),
@@ -1584,10 +1584,8 @@ app.get("/api/config", (req, res) => res.json({ version: appVersion, minPort, ma
 app.get("/api/system/storage", async (req, res, next) => {
   if (req.user.role !== "administrator") return res.status(403).json({ error: "Administrator access is required." });
   try {
-    const breakdown = {};
+    const breakdownDirs = { sites: sitesDir, backups: backupsDir, certificates: certificatesRoot, logs: logsDir, database: path.join(dataDir, "database") };
-    for (const [key, dir] of Object.entries({ sites: sitesDir, backups: backupsDir, certificates: certificatesRoot, logs: logsDir, database: path.join(dataDir, "database") })) {
+    const breakdown = Object.fromEntries(await Promise.all(Object.entries(breakdownDirs).map(async ([key, dir]) => [key, await directorySize(dir)])));
-      breakdown[key] = await directorySize(dir);
-    }
     let capacity = null;
     try {
       const stats = await fsp.statfs(dataDir);
@@ -1783,7 +1781,12 @@ app.get("/api/certificates", async (req, res, next) => {
   catch (error) { next(error); }
 });
 app.post("/api/health/check", async (req, res, next) => {
-  try { await checkAllProxies(); res.json({ dashboard: await dashboardSnapshot(), certificates: await certificateInventory(), readiness: await domainReadiness() }); }
+  try {
+    await checkAllProxies();
+    const certificates = await certificateInventory();
+    const [dashboard, readiness] = await Promise.all([dashboardSnapshot(certificates), domainReadiness(certificates)]);
+    res.json({ dashboard, certificates, readiness });
+  }
   catch (error) { next(error); }
 });
 app.get("/api/readiness", async (req, res, next) => { try { res.json({ checkedAt: new Date().toISOString(), routes: await domainReadiness() }); } catch (error) { next(error); } });
@@ -1795,8 +1798,9 @@ app.get("/api/support-report", async (req, res, next) => {
   try {
     if (req.user.role !== "administrator") return res.status(403).json({ error: "Administrator access is required." });
     const certificateReport = await certificateInventory();
+    const readiness = await domainReadiness(certificateReport);
     certificateReport.latestError = certificateReport.latestError ? { present:true, at:certificateReport.latestError.at } : null;
-    const report = { product: "Site Gateway", generatedAt: new Date().toISOString(), version: appVersion, caddyVersion, nodeVersion: process.version, storage: { engine: "SQLite", integrity: storage.integrity() }, gateway: { healthy: !gatewayError, lastReload: lastGatewayReload }, routes: { hosted: sites.map(({ id,name,domain,tls,enabled,port }) => ({ id,name,domain,tls,enabled,port })), proxies: proxies.map(({ id,name,domain,tls,enabled,target,healthEnabled,healthExpected }) => ({ id,name,domain,tls,enabled,target,healthEnabled,healthExpected })), redirects: redirects.map(({ id,name,domain,tls,enabled,code }) => ({ id,name,domain,tls,enabled,code })) }, certificates: certificateReport, readiness: await domainReadiness(), recentEvents: recentActivity.slice(0,20).map(item => ({ at:item.at, status:item.status, message:item.status === "error" ? "Operational error recorded; review the protected in-app event log for details." : item.message })) };
+    const report = { product: "Site Gateway", generatedAt: new Date().toISOString(), version: appVersion, caddyVersion, nodeVersion: process.version, storage: { engine: "SQLite", integrity: storage.integrity() }, gateway: { healthy: !gatewayError, lastReload: lastGatewayReload }, routes: { hosted: sites.map(({ id,name,domain,tls,enabled,port }) => ({ id,name,domain,tls,enabled,port })), proxies: proxies.map(({ id,name,domain,tls,enabled,target,healthEnabled,healthExpected }) => ({ id,name,domain,tls,enabled,target,healthEnabled,healthExpected })), redirects: redirects.map(({ id,name,domain,tls,enabled,code }) => ({ id,name,domain,tls,enabled,code })) }, certificates: certificateReport, readiness, recentEvents: recentActivity.slice(0,20).map(item => ({ at:item.at, status:item.status, message:item.status === "error" ? "Operational error recorded; review the protected in-app event log for details." : item.message })) };
     res.setHeader("Content-Disposition", `attachment; filename="site-gateway-support-${new Date().toISOString().slice(0,10)}.json"`); res.type("json").send(JSON.stringify(report, null, 2));
   } catch (error) { next(error); }
 });