Cache the database integrity check instead of running it on every dashboard fetch

README.md

@@ -12,7 +12,7 @@
     <img alt="Docker" src="https://img.shields.io/badge/Docker-ready-2496ED?logo=docker&logoColor=white">
     <img alt="Architectures" src="https://img.shields.io/badge/platform-amd64%20%7C%20arm64-5965F2">
     <img alt="Caddy" src="https://img.shields.io/badge/powered%20by-Caddy-1F88C0">
-    <img alt="Version" src="https://img.shields.io/badge/version-0.16.44-62E6A7">
+    <img alt="Version" src="https://img.shields.io/badge/version-0.16.45-62E6A7">
   </p>
   <p>
     <a href="#why-site-gateway">Why Site Gateway</a> ·

ROADMAP.md

@@ -216,3 +216,5 @@ Roughly in priority order:
 `v0.16.43` scopes page refreshes to the page actually being viewed, instead of every refresh across the entire app unconditionally re-fetching everything -- Hosted Sites, Proxy Hosts, Redirects, Streams, Access Lists, Groups, the full Dashboard snapshot, and Certificates -- regardless of which single page triggered it. This was confirmed directly from the user's own account of the behavior ("if I'm on Hosted Sites and click refresh, it appears the whole entire site refreshes") and traced to a single shared `refresh()` function that every action in the app called: creating or editing a hosted site or proxy, toggling one on or off, deleting an entry, saving gateway settings, and re-syncing the gateway all ran the identical 8-endpoint fetch no matter which page initiated it. `refresh()` and its endpoints are now built from one shared map (`REFRESH_ENDPOINTS`), and a new `refreshCurrentView()` fetches only the state keys a `VIEW_REFRESH_KEYS` table says the active view actually renders -- Hosted Sites now refetches just `sites`, Proxy Hosts just `proxies`, Streaming just `streams`, Redirects just `redirects`, Access Lists just `accessLists` and `groups`. Every action listed above that's only ever reachable from one specific view (creating/editing/toggling/deleting a hosted site or proxy) now calls `refreshCurrentView()` instead of the full `refresh()`. Overview keeps the full, unscoped fetch deliberately: its attention list and the sidebar's per-section counts summarize the whole gateway, not one section of it, so scoping it would defeat the page's purpose; the initial page load (`boot()`) and the gateway re-sync button (only reachable from Overview) are unchanged for the same reason. A new generic refresh button (the same "↻" icon `refresh-health` already used) now appears on every page except Logs (which keeps its own dedicated "Refresh logs" button) so every view has an explicit, page-scoped way to pull fresh data without a full browser reload -- previously several views (Hosted, Proxy Hosts, Streaming, Redirects, Access Lists) had no refresh control of their own at all and only ever picked up new data from the page's initial load or the next full-page reload. One deliberate trade-off: sidebar badge counts for sections other than the one currently being viewed are not part of a scoped refresh and can go briefly stale until the next full refresh (a fresh page load, or a visit to Overview) -- intentional, since fetching data a page doesn't display was the entire problem being fixed here.
 
 `v0.16.44` is a temporary, diagnostic-only release -- no behavior changes, just logging -- added after v0.16.43 (which fixed the app from over-fetching per page) didn't resolve the user's reported 6-14 second page loads. A Network-tab Timing capture the user sent for a single `GET /api/sites` request showed DNS and TCP connection at 0-7ms but "Waiting" (time to first byte) at 7485ms -- almost the entire delay happened server-side, before the app sent back a single byte of what should be a near-instant, in-memory list. Since this codebase's database and JS execution is single-threaded, that pattern (a trivially cheap request taking seconds) points to something else blocking the whole process at that moment, not a cost specific to any one endpoint. The leading suspect: `importAccessLogsToSqlite()`, a job that runs every 30 seconds, reads Caddy's access-log files, JSON-parses and hashes up to 5000 lines, and batch-inserts them -- all synchronous work with nothing to yield the event loop partway through. Rather than ship a fourth guess-based fix, this release adds two pieces of logging visible in the container's own logs: a warning whenever that import job takes over 500ms (broken down into read/hash/insert time), and a warning whenever any request takes over 1 second to answer. The next slow page load should show, in the logs, either the import job's duration lining up with the slow request's timestamp (confirming the suspect) or a different pattern entirely (pointing somewhere else). Both log lines are marked as temporary instrumentation, intended to be removed once the real cause is confirmed and fixed.
+
+`v0.16.45` fixes the confirmed root cause behind the multi-second page loads reported after v0.16.41-v0.16.43: the user's own container logs, captured with v0.16.44's temporary diagnostics, showed completely unrelated endpoints -- `/api/dashboard`, `/api/system/security`, `/api/logs/prune/preview` -- all finishing within moments of each other at nearly identical ~8.5-9 second durations, right after the container started. That pattern only happens when several requests are queued behind one shared blocking operation, not when each is independently slow. The culprit: `dashboardSnapshot()` (which every `/api/dashboard` fetch runs) called `storage.integrity()` -- a full `PRAGMA integrity_check`, a complete scan of the entire SQLite database file for corruption, one of the most expensive operations SQLite can run -- on every single call, purely to compute one cosmetic "Healthy"/"Needs attention" label. Because this app's SQLite queries run synchronously, that scan didn't just make its own request slow, it froze the entire single-threaded server for its whole duration, on every dashboard fetch, for every user. The fix moves that check off the request path entirely: a new `refreshDatabaseIntegrityCache()` runs the real scan once shortly after startup and then every 30 minutes in the background, caching just the resulting status string, and `dashboardSnapshot()` now reads that cached value instantly instead of re-scanning the whole database on every poll. The (rarely-used, explicitly manual) downloadable support report still runs a live, real-time integrity check, since that's an appropriate place for a slow, thorough scan. v0.16.44's temporary `[perf]` logging stays in place for this release so the fix's effect is directly visible in the container's own logs -- expect no more `[perf] GET ... took` warnings tied to `/api/dashboard` going forward.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "site-gateway",
-  "version": "0.16.44",
+  "version": "0.16.45",
   "private": true,
   "description": "Site Gateway: simple self-hosted website publishing, reverse proxying, and automatic HTTPS.",
   "type": "module",

src/public/index.html

@@ -8,7 +8,7 @@
   <title>Site Gateway</title>
   <meta name="description" content="Host sites, proxy services, and manage HTTPS from one simple dashboard.">
   <link rel="icon" type="image/png" href="/site-gateway-icon-approved.png">
-  <link rel="stylesheet" href="/styles.css?v=0.16.44">
+  <link rel="stylesheet" href="/styles.css?v=0.16.45">
 </head>
 
 <!-- ================================================================
@@ -432,6 +432,6 @@
   <div id="toast" class="toast" role="status"></div>
   <div id="update-banner" class="update-banner hidden" role="status"><span>A new version of Site Gateway is available.</span><div class="update-banner-actions"><button id="update-banner-refresh" class="button primary">Refresh</button><button id="update-banner-dismiss" class="text-button">Dismiss</button></div></div>
   <!-- App scripts: core (app.js) then extended views/admin (features.js) -->
-  <script src="/app.js?v=0.16.44" defer></script><script src="/features.js?v=0.16.44" defer></script><script src="/select-enhance.js?v=0.16.44" defer></script>
+  <script src="/app.js?v=0.16.45" defer></script><script src="/features.js?v=0.16.45" defer></script><script src="/select-enhance.js?v=0.16.45" defer></script>
 </body>
 </html>

src/server.js

@@ -807,6 +807,25 @@ async function syncCaddy() {
 
 
 let configDrift = { checkedAt: null, drift: false, detail: null };
+// storage.integrity() runs a full PRAGMA integrity_check -- a complete scan of the entire SQLite
+// database file for corruption. It's one of the most expensive operations SQLite can run, its
+// cost scales with total database size, and because this app's SQLite queries run synchronously,
+// it blocks the whole single-threaded server for its full duration while it runs -- not just the
+// request that triggered it. dashboardSnapshot() used to call it on EVERY /api/dashboard fetch
+// just to compute one cosmetic "Healthy"/"Needs attention" label, which is why unrelated requests
+// (confirmed via container logs: /api/system/security, /api/logs/prune/preview) were getting
+// stuck behind it in lockstep, all finishing at nearly the same multi-second mark regardless of
+// what they actually needed to do. A dashboard status badge doesn't need a fresh, exhaustive
+// integrity scan on every single poll -- checking it periodically in the background and caching
+// the result is more than sufficient, since real corruption doesn't appear and disappear between
+// one 7-second poll and the next.
+let databaseIntegrityCache = { checkedAt: null, status: "Healthy" };
+function refreshDatabaseIntegrityCache() {
+  try {
+    const result = storage.integrity();
+    databaseIntegrityCache = { checkedAt: new Date().toISOString(), status: result.length === 1 && result[0] === "ok" ? "Healthy" : "Needs attention" };
+  } catch (error) { console.warn("Database integrity check failed:", error.message); }
+}
 let lastUpstreamCheckAt = null;
 let lastAccessLogImportAt = null;
 let lastKnownGoodCaddyConfig = null;
@@ -1148,7 +1167,7 @@ async function dashboardSnapshot(precomputedCertificates) {
   for (const certificate of certificates.certificates.filter(item => ["warning", "critical", "expired", "mismatch"].includes(item.status))) attention.push({ kind: "certificate", target: "certificates", name: certificate.domain, message: certificate.status === "expired" ? "Certificate has expired." : certificate.status === "mismatch" ? "The uploaded certificate does not cover this domain." : `Certificate expires in ${certificate.daysRemaining} day${certificate.daysRemaining === 1 ? "" : "s"}.` });
   if (configDrift.drift) attention.push({ kind: "drift", name: "Configuration drift", message: "Caddy\u2019s live configuration no longer matches the saved configuration.", target: "administration/defaults" });
   const disk = await fsp.statfs(dataDir).catch(() => null);
-  const databaseIntegrity = storage.integrity();
+  // See refreshDatabaseIntegrityCache() above -- this used to be a live storage.integrity() call on every fetch.
   return {
     checkedAt: new Date().toISOString(),
     gateway: { ...gatewayProbe, lastReload: lastGatewayReload },
@@ -1175,7 +1194,7 @@ async function dashboardSnapshot(precomputedCertificates) {
       caddyVersion,
       nodeVersion: process.version,
       databaseEngine: "SQLite",
-      databaseStatus: databaseIntegrity.length === 1 && databaseIntegrity[0] === "ok" ? "Healthy" : "Needs attention",
+      databaseStatus: databaseIntegrityCache.status,
       databaseBytes: (await fsp.stat(storage.databasePath).catch(() => null))?.size || 0,
       publicIp: publicIpState.address,
       publicIpCheckedAt: publicIpState.checkedAt,
@@ -2611,6 +2630,10 @@ setTimeout(() => cleanupOldPruneSnapshots().catch(error => console.warn("Startup
 setInterval(() => checkAllProxies().then(() => { lastUpstreamCheckAt = new Date().toISOString(); }).catch(error => console.warn("Upstream checks failed:", error.message)), 60000).unref();
 setTimeout(() => checkConfigDrift().catch(error => console.warn("Config drift check failed:", error.message)), 10000).unref();
 setInterval(() => checkConfigDrift().catch(error => console.warn("Config drift check failed:", error.message)), 10 * 60000).unref();
+// Runs the (expensive, synchronous, whole-server-blocking) database integrity scan once shortly
+// after boot and then every 30 minutes in the background, rather than on every dashboard fetch.
+setTimeout(refreshDatabaseIntegrityCache, 5000).unref();
+setInterval(refreshDatabaseIntegrityCache, 30 * 60000).unref();
 
 
 // --- Scheduled jobs: automatic backups, log pruning, public IP checks, graceful shutdown ---------------------------------