Add temporary timing diagnostics for slow requests and the access-log import job

README.md

@@ -12,7 +12,7 @@
     <img alt="Docker" src="https://img.shields.io/badge/Docker-ready-2496ED?logo=docker&logoColor=white">
     <img alt="Architectures" src="https://img.shields.io/badge/platform-amd64%20%7C%20arm64-5965F2">
     <img alt="Caddy" src="https://img.shields.io/badge/powered%20by-Caddy-1F88C0">
-    <img alt="Version" src="https://img.shields.io/badge/version-0.16.39-62E6A7">
+    <img alt="Version" src="https://img.shields.io/badge/version-0.16.44-62E6A7">
   </p>
   <p>
     <a href="#why-site-gateway">Why Site Gateway</a> ·

ROADMAP.md

@@ -205,3 +205,14 @@ Roughly in priority order:
 
 
 `v0.16.39` retires the Dashboard Uptime tile's separate 1-second client-side ticker, the same mechanism behind the v0.16.38 "resets to 0m on page load" bug, in favor of treating Uptime as just another field on the shared 7-second `/api/system/health` poll that already drives CPU, memory, swap, disk, and network on both the Dashboard and the Administration > System tab's hero panel. Revisiting the ticker after fixing its race condition, it turned out to be solving a precision problem the display doesn't actually have: `formatDuration()` only ever renders minute-level granularity ("2h 59m"), never seconds, so a per-second tick never changed what was on screen between one 7-second poll and the next. `systemHealthSnapshot()` now includes `uptimeSeconds` (from `process.uptime()`, the same source `dashboardSnapshot()` already used), and `renderHeroPanel()`'s former `includeThroughput` boolean became a `sixthSlot` option ("throughput" for the System tab, which has no other requests/min display, or "uptime" for the Dashboard, which already shows Throughput in its own chip) so the sixth hero slot can be either stat without a special case. The client ticker, its anchor state, and the whole race-condition class it enabled are gone: one poll, one code path, six stats, no anchor to get out of sync.
+
+
+`v0.16.40` moves the Administration → System tab's Version panel Uptime figure onto the same 7-second `/api/system/health` poll driving the hero panel above it, instead of only updating when the page's slower, general dashboard snapshot refetches (on load, or every ~30 seconds while the Dashboard view specifically is active). Found while reviewing the v0.16.39 change: the hero's own Uptime slot was now current to the second, but the separate Version-panel line right below it — the same number, shown twice on the same page — could still be stale by up to half a minute or more. `renderSystemStatus()`'s Uptime is now wrapped in its own `#system-version-uptime` span and updated by a small `updateSystemVersionUptime(health)` helper, called both from the initial render and from the same 7-second timer that already refreshes the hero, so the two Uptime figures on that page can no longer drift apart.
+
+`v0.16.41` cuts redundant work out of the app's shared `refresh()` cycle -- the single function that populates nearly every page (Hosted Sites, Proxy Hosts, Redirects, Streams, Access Lists, Dashboard, and Certificates all pull from it) -- after a user reported the whole site feeling slow to refresh, most concretely on a plain reload of Hosted Sites or Proxy Hosts. Two real causes, found by reading the actual request path rather than guessing: first, `certificateInventory()` (which walks the certificate directories and parses every `.crt`/`.pem` file on disk) was being fully recomputed from scratch on every single call, and `refresh()` calls it twice per cycle -- once via `/api/dashboard`, once via `/api/certificates` -- so a normal page load did that walk-and-parse work twice for identical results. It now carries a short (3 second) in-memory cache, well under the 7-second hero-poll interval, so back-to-back calls within a cycle share one real disk walk instead of two, and nothing on screen goes more than one cycle stale. Second, and the bigger one: `refreshPendingProxies()` -- triggered whenever a page loads with any enabled proxy that doesn't have cached upstream-health data yet, which is the common case right after a page load or a new proxy -- was calling the *entire* `refresh()` again at +1s, +2s, and +3s until every proxy's health came back. That meant a single pending proxy could quietly trigger three additional full 8-endpoint refetches (each with its own pair of certificate walks) in the six seconds after a page appeared to have finished loading. It now re-fetches only `/api/proxies` on those retries, since upstream health is all it was ever waiting on. Together these remove the two largest sources of duplicated, unnecessary work from the most-used code path in the app; whether they fully account for the reported slowness or whether a client-side rendering cost remains to find is still open and being evaluated against a real before/after comparison.
+
+`v0.16.42` finds and fixes the real, dominant cause of the site-wide slowness reported after v0.16.41: a live Network-tab capture from the user's own browser showed a flood of requests to `/api/logs/prune/preview`, some queued for over 15 seconds, with unrelated requests (`/api/dashboard`, `/api/system/health`, `/api/system/security`, `/api/system/storage`) stuck at nearly identical multi-second times in the same batch -- the signature of one blocking operation stalling everything behind it, not several independently slow endpoints. Root cause: `renderRetentionPreview()`'s `setInterval(..., 2000)` polls that endpoint every 2 seconds forever, on every page of the app, not just Administration -> Logs & retention, because its "does the panel exist" guard checks a `<section>` that's written into `index.html` from page load and only ever CSS-hidden -- so the guard was always true, everywhere. There was also no protection against a new poll firing while a previous one was still in flight, so once the server answered slower than 2 seconds even once, requests piled up and never caught back up. Compounding it: `previewPruneEvents()` runs five synchronous SQLite COUNT queries, and one of them (`audit_events`) had no index at all -- a full table scan, every call -- and because this app's SQLite queries run synchronously, that scan doesn't just slow its own request, it blocks the entire Node process for every other request being served at that moment. Fixed on both sides: `renderRetentionPreview()` and the sibling `renderRetentionRunStatus()` (previously also running unconditionally every 500ms) now check that the retention panel is actually visible, not just present in the DOM, before doing any work, and an in-flight guard stops a new preview poll from starting until the last one has landed; `audit_events` now has the same `(instance_id, created_at)` index every sibling events table already had. Together these should remove the vast majority of the "8-10 seconds to load a simple page" behavior reported after v0.16.41 -- that fix (deduplicating certificate-inventory work) was real but minor by comparison to this one.
+
+`v0.16.43` scopes page refreshes to the page actually being viewed, instead of every refresh across the entire app unconditionally re-fetching everything -- Hosted Sites, Proxy Hosts, Redirects, Streams, Access Lists, Groups, the full Dashboard snapshot, and Certificates -- regardless of which single page triggered it. This was confirmed directly from the user's own account of the behavior ("if I'm on Hosted Sites and click refresh, it appears the whole entire site refreshes") and traced to a single shared `refresh()` function that every action in the app called: creating or editing a hosted site or proxy, toggling one on or off, deleting an entry, saving gateway settings, and re-syncing the gateway all ran the identical 8-endpoint fetch no matter which page initiated it. `refresh()` and its endpoints are now built from one shared map (`REFRESH_ENDPOINTS`), and a new `refreshCurrentView()` fetches only the state keys a `VIEW_REFRESH_KEYS` table says the active view actually renders -- Hosted Sites now refetches just `sites`, Proxy Hosts just `proxies`, Streaming just `streams`, Redirects just `redirects`, Access Lists just `accessLists` and `groups`. Every action listed above that's only ever reachable from one specific view (creating/editing/toggling/deleting a hosted site or proxy) now calls `refreshCurrentView()` instead of the full `refresh()`. Overview keeps the full, unscoped fetch deliberately: its attention list and the sidebar's per-section counts summarize the whole gateway, not one section of it, so scoping it would defeat the page's purpose; the initial page load (`boot()`) and the gateway re-sync button (only reachable from Overview) are unchanged for the same reason. A new generic refresh button (the same "↻" icon `refresh-health` already used) now appears on every page except Logs (which keeps its own dedicated "Refresh logs" button) so every view has an explicit, page-scoped way to pull fresh data without a full browser reload -- previously several views (Hosted, Proxy Hosts, Streaming, Redirects, Access Lists) had no refresh control of their own at all and only ever picked up new data from the page's initial load or the next full-page reload. One deliberate trade-off: sidebar badge counts for sections other than the one currently being viewed are not part of a scoped refresh and can go briefly stale until the next full refresh (a fresh page load, or a visit to Overview) -- intentional, since fetching data a page doesn't display was the entire problem being fixed here.
+
+`v0.16.44` is a temporary, diagnostic-only release -- no behavior changes, just logging -- added after v0.16.43 (which fixed the app from over-fetching per page) didn't resolve the user's reported 6-14 second page loads. A Network-tab Timing capture the user sent for a single `GET /api/sites` request showed DNS and TCP connection at 0-7ms but "Waiting" (time to first byte) at 7485ms -- almost the entire delay happened server-side, before the app sent back a single byte of what should be a near-instant, in-memory list. Since this codebase's database and JS execution is single-threaded, that pattern (a trivially cheap request taking seconds) points to something else blocking the whole process at that moment, not a cost specific to any one endpoint. The leading suspect: `importAccessLogsToSqlite()`, a job that runs every 30 seconds, reads Caddy's access-log files, JSON-parses and hashes up to 5000 lines, and batch-inserts them -- all synchronous work with nothing to yield the event loop partway through. Rather than ship a fourth guess-based fix, this release adds two pieces of logging visible in the container's own logs: a warning whenever that import job takes over 500ms (broken down into read/hash/insert time), and a warning whenever any request takes over 1 second to answer. The next slow page load should show, in the logs, either the import job's duration lining up with the slow request's timestamp (confirming the suspect) or a different pattern entirely (pointing somewhere else). Both log lines are marked as temporary instrumentation, intended to be removed once the real cause is confirmed and fixed.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "site-gateway",
-  "version": "0.16.39",
+  "version": "0.16.44",
   "private": true,
   "description": "Site Gateway: simple self-hosted website publishing, reverse proxying, and automatic HTTPS.",
   "type": "module",

src/public/app.js

@@ -123,7 +123,7 @@ document.addEventListener("submit", async event => {
   try {
     await api(`/api/${state.editing.kind === "proxy" ? "proxies" : "sites"}/${state.editing.id}`, { method: "PATCH", headers: { "Content-Type": "application/json" }, body: JSON.stringify(body) });
     if (uploadCustom) { const files = new FormData(); files.append("certificate", certificate); files.append("privateKey", privateKey); await api(`/api/proxies/${state.editing.id}/certificate`, { method: "POST", body: files }); }
-    $("#settings-dialog").close(); await refresh(); toast("Gateway settings applied.");
+    $("#settings-dialog").close(); await refreshCurrentView(); toast("Gateway settings applied.");
   }
   catch (error) { $("#settings-error").textContent = error.message; }
   finally { button.disabled = false; }
@@ -493,7 +493,7 @@ function render() {
   $("#streaming-view").classList.toggle("hidden", state.view !== "streaming"); $("#redirects-view").classList.toggle("hidden", state.view !== "redirects"); $("#access-view").classList.toggle("hidden", state.view !== "access"); $("#documentation-view").classList.toggle("hidden", state.view !== "documentation");
   const activeAdminTab = state.view === "administration" ? document.querySelector("[data-admin-tab].tab-active")?.dataset.adminTab : null;
   const adminUsersActive = activeAdminTab === "users", adminGroupsActive = activeAdminTab === "groups", adminApiActive = activeAdminTab === "api";
-  $("#open-create").classList.toggle("hidden", !(management || adminUsersActive || adminGroupsActive || adminApiActive || ["streaming","redirects","access"].includes(state.view)) || !canManage()); $("#check-health").classList.toggle("hidden", state.view !== "certificates" || !canAdmin()); $("#refresh-logs").classList.toggle("hidden", state.view !== "logs");
+  $("#open-create").classList.toggle("hidden", !(management || adminUsersActive || adminGroupsActive || adminApiActive || ["streaming","redirects","access"].includes(state.view)) || !canManage()); $("#check-health").classList.toggle("hidden", state.view !== "certificates" || !canAdmin()); $("#refresh-logs").classList.toggle("hidden", state.view !== "logs"); $("#refresh-view").classList.toggle("hidden", state.view === "logs");
   if (overview) {
     $("#page-title").textContent = "Dashboard";
     $("#page-subtitle").textContent = "Health, activity, and system status at a glance.";
@@ -532,13 +532,76 @@ function render() {
 }
 
 // --- Data refresh helpers ------------------------------------------------------------------
-async function refresh() { const requests = [api("/api/sites"), api("/api/proxies"), api("/api/redirects"), api("/api/streams"), api("/api/access-lists"), canAdmin() ? api("/api/groups") : Promise.resolve([]), api("/api/dashboard"), api("/api/certificates")]; const results = await Promise.allSettled(requests); results.forEach((result, index) => { if (result.status !== "fulfilled") return; const keys = ["sites", "proxies", "redirects", "streams", "accessLists", "groups", "dashboard", "certificates"]; state[keys[index]] = result.value; }); state.loaded = true; render(); window.renderExtendedViews?.(); const pending = state.proxies.filter(proxy => proxy.enabled !== false && !proxy.upstream).map(proxy => proxy.id); if (pending.length && !state.pendingProxyRefresh) { state.pendingProxyRefresh = true; refreshPendingProxies(pending).finally(() => { state.pendingProxyRefresh = false; }); } }
+// Each entry is the state key a call populates and the fetch that populates it. refresh() (the
+// full, unscoped fetch) and refreshCurrentView() (the page-scoped fetch, see below) both build
+// their request list from this single map, so adding a new piece of shared state only ever means
+// adding one line here.
+const REFRESH_ENDPOINTS = {
+  sites: () => api("/api/sites"),
+  proxies: () => api("/api/proxies"),
+  redirects: () => api("/api/redirects"),
+  streams: () => api("/api/streams"),
+  accessLists: () => api("/api/access-lists"),
+  groups: () => canAdmin() ? api("/api/groups") : Promise.resolve([]),
+  dashboard: () => api("/api/dashboard"),
+  certificates: () => api("/api/certificates"),
+};
+// Which of the keys above each view actually renders. A view not listed here (certificates, logs,
+// performance, administration, account, documentation) already loads its own data separately via
+// loadFeatureView() and never called refresh() at all, so it isn't included. Overview intentionally
+// lists everything: its attention list and the sidebar's per-section counts summarize the whole
+// gateway, not one section of it, so a scoped fetch there would defeat the point of the page.
+const VIEW_REFRESH_KEYS = {
+  overview: Object.keys(REFRESH_ENDPOINTS),
+  hosted: ["sites"],
+  proxies: ["proxies"],
+  streaming: ["streams"],
+  redirects: ["redirects"],
+  access: ["accessLists", "groups"],
+};
+async function refreshKeys(keys) {
+  const results = await Promise.allSettled(keys.map(key => REFRESH_ENDPOINTS[key]()));
+  results.forEach((result, index) => { if (result.status === "fulfilled") state[keys[index]] = result.value; });
+}
+function maybeRefreshPendingProxies() {
+  const pending = state.proxies.filter(proxy => proxy.enabled !== false && !proxy.upstream).map(proxy => proxy.id);
+  if (pending.length && !state.pendingProxyRefresh) { state.pendingProxyRefresh = true; refreshPendingProxies(pending).finally(() => { state.pendingProxyRefresh = false; }); }
+}
+// The original, unscoped refresh -- fetches every shared list plus the dashboard and certificate
+// summaries in one pass. Kept for cases that genuinely need everything at once: first page load
+// (boot()) and the Overview page, whose attention list and counts summarize the entire gateway.
+async function refresh() { await refreshKeys(Object.keys(REFRESH_ENDPOINTS)); state.loaded = true; render(); window.renderExtendedViews?.(); maybeRefreshPendingProxies(); }
+// The page-scoped refresh: fetches only the state a given view actually renders, instead of
+// unconditionally re-fetching sites, proxies, redirects, streams, access lists, groups, the full
+// dashboard snapshot, and certificates every single time -- regardless of which one page the user
+// is looking at. This was the original, most direct cause behind "refreshing one page refetches
+// the whole site": every action (create, edit, toggle, delete) and every manual refresh called the
+// same all-8-endpoints refresh() no matter which view triggered it. Sidebar badge counts for
+// sections other than the current view are not re-fetched by this path and can go briefly stale
+// until the next full refresh() (a fresh page load, or a visit to Overview) -- an intentional
+// trade for not fetching data the current page doesn't display.
+async function refreshCurrentView() {
+  const keys = VIEW_REFRESH_KEYS[state.view] || Object.keys(REFRESH_ENDPOINTS);
+  await refreshKeys(keys);
+  state.loaded = true; render(); window.renderExtendedViews?.();
+  if (keys.includes("proxies")) maybeRefreshPendingProxies();
+}
+// Polls just /api/proxies for upstream health that wasn't ready yet on the last refresh() --
+// e.g. right after a page load or a new proxy, before its first health check has completed.
+// This used to call the full refresh() (all 8 endpoints, including two redundant certificate
+// walks via /api/dashboard + /api/certificates), up to 3 times in a row -- meaning a single
+// pending proxy could quietly trigger 3 extra full-app refetches over 6 seconds. Since all it
+// actually needs is fresh upstream status, it now re-fetches only /api/proxies.
 async function refreshPendingProxies(ids = []) {
   const pending = new Set(ids.map(String));
   for (const delay of [1000, 2000, 3000]) {
     if (!pending.size) return;
     await new Promise(resolve => setTimeout(resolve, delay));
-    await refresh();
+    try {
+      state.proxies = await api("/api/proxies");
+      render();
+      window.renderExtendedViews?.();
+    } catch { /* Keep the last-known proxy list if this poll fails; the next delay tries again. */ }
     for (const proxy of state.proxies) if (pending.has(String(proxy.id)) && proxy.upstream) pending.delete(String(proxy.id));
   }
 }
@@ -635,6 +698,21 @@ $("#dashboard-view").addEventListener("click", event => { const target = event.t
 
 // --- Logs & Performance filter controls -----------------------------------------------------
 $("#refresh-logs").addEventListener("click", () => loadFeatureView().catch(error => toast(error.message, "error")));
+// Generic page-scoped refresh button, shown on every view except Logs (which already has its own
+// "Refresh logs" button wired to loadFeatureView()). Uses refreshCurrentView() for the shared-list
+// views (Overview, Hosted, Proxy Hosts, Streaming, Redirects, Access Lists) so it fetches only
+// what that page renders, and falls back to loadFeatureView() for every other view (Certificates,
+// Performance, Administration, Account, Documentation), which already load their own data scoped
+// to themselves.
+$("#refresh-view").addEventListener("click", async () => {
+  const button = $("#refresh-view"); button.disabled = true; button.classList.add("spinning");
+  try {
+    if (state.view in VIEW_REFRESH_KEYS || state.view === "overview") await refreshCurrentView();
+    else await loadFeatureView();
+    toast("Refreshed.");
+  } catch (error) { toast(error.message, "error"); }
+  finally { button.disabled = false; button.classList.remove("spinning"); }
+});
 $("#log-host").addEventListener("change", () => loadFeatureView().catch(error => toast(error.message, "error")));
 $("#performance-host").addEventListener("change", () => loadFeatureView().catch(error => toast(error.message, "error")));
 $("#performance-range").addEventListener("change", () => loadFeatureView().catch(error => toast(error.message, "error")));
@@ -692,8 +770,8 @@ document.querySelectorAll("dialog").forEach(dialog => dialog.addEventListener("c
 
 // --- Hosted Sites & Proxy Hosts: create form submit handlers --------------------------------
 $("#refresh-health").addEventListener("click", () => refreshDashboard().catch(error => toast(error.message, "error")));
-$("#create-form").addEventListener("submit", async event => { event.preventDefault(); const button = resolveSubmitter(event); button.disabled = true; button.textContent = "Publishing…"; $("#create-error").textContent = ""; try { await api("/api/sites", { method: "POST", body: new FormData(event.target) }); $("#create-dialog").close(); await refresh(); toast("Hosted site created and gateway applied."); } catch (error) { $("#create-error").textContent = error.message; } finally { button.disabled = false; button.textContent = "Create & publish"; } });
-$("#proxy-form").addEventListener("submit", async event => { event.preventDefault(); const button = resolveSubmitter(event); button.disabled = true; button.textContent = "Publishing…"; $("#proxy-error").textContent = ""; const form = new FormData(event.target), certificate = form.get("certificateFile"), privateKey = form.get("privateKeyFile"), wantsCustom = form.get("tls") === "custom"; if (wantsCustom && (!certificate?.size || !privateKey?.size)) { $("#proxy-error").textContent = "Choose both the certificate and private key for Custom HTTPS."; button.disabled = false; button.textContent = "Create & publish"; return; } const body = advancedFormBody(form, Object.fromEntries(form)); delete body.certificateFile; delete body.privateKeyFile; if (wantsCustom) body.tls = "http"; try { const created = await api("/api/proxies", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(body) }); if (wantsCustom) { const files = new FormData(); files.append("certificate", certificate); files.append("privateKey", privateKey); await api(`/api/proxies/${created.id}/certificate`, { method:"POST", body:files }); } $("#proxy-dialog").close(); await refresh(); toast(wantsCustom ? "Proxy host created with its custom certificate." : "Proxy host created. Certificate provisioning runs automatically."); } catch (error) { $("#proxy-error").textContent = error.message; } finally { button.disabled = false; button.textContent = "Create & publish"; } });
+$("#create-form").addEventListener("submit", async event => { event.preventDefault(); const button = resolveSubmitter(event); button.disabled = true; button.textContent = "Publishing…"; $("#create-error").textContent = ""; try { await api("/api/sites", { method: "POST", body: new FormData(event.target) }); $("#create-dialog").close(); await refreshCurrentView(); toast("Hosted site created and gateway applied."); } catch (error) { $("#create-error").textContent = error.message; } finally { button.disabled = false; button.textContent = "Create & publish"; } });
+$("#proxy-form").addEventListener("submit", async event => { event.preventDefault(); const button = resolveSubmitter(event); button.disabled = true; button.textContent = "Publishing…"; $("#proxy-error").textContent = ""; const form = new FormData(event.target), certificate = form.get("certificateFile"), privateKey = form.get("privateKeyFile"), wantsCustom = form.get("tls") === "custom"; if (wantsCustom && (!certificate?.size || !privateKey?.size)) { $("#proxy-error").textContent = "Choose both the certificate and private key for Custom HTTPS."; button.disabled = false; button.textContent = "Create & publish"; return; } const body = advancedFormBody(form, Object.fromEntries(form)); delete body.certificateFile; delete body.privateKeyFile; if (wantsCustom) body.tls = "http"; try { const created = await api("/api/proxies", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(body) }); if (wantsCustom) { const files = new FormData(); files.append("certificate", certificate); files.append("privateKey", privateKey); await api(`/api/proxies/${created.id}/certificate`, { method:"POST", body:files }); } $("#proxy-dialog").close(); await refreshCurrentView(); toast(wantsCustom ? "Proxy host created with its custom certificate." : "Proxy host created. Certificate provisioning runs automatically."); } catch (error) { $("#proxy-error").textContent = error.message; } finally { button.disabled = false; button.textContent = "Create & publish"; } });
 
 
 // --- Health-check field visibility polish for the create forms ------------------------------
@@ -727,7 +805,7 @@ $("#site-grid").addEventListener("click", async event => {
   const card = event.target.closest(".site-card"); if (!card) return; const action = event.target.closest("[data-action]")?.dataset.action, kind = card.dataset.kind;
   if (event.target.closest(".menu-button")) { const opening = !card.classList.contains("menu-open"); closeMenus(); card.classList.toggle("menu-open", opening); card.querySelector(".menu-button").setAttribute("aria-expanded", String(opening)); return; } if (!action) return;
   closeMenus();
-  if (action === "toggle") { const toggleButton = event.target.closest(".toggle"), wasOn = toggleButton.classList.contains("on"); toggleButton.classList.toggle("on", !wasOn); toggleButton.disabled = true; const base = kind === "proxy" ? "proxies" : "sites"; try { await api(`/api/${base}/${card.dataset.id}/toggle`, { method: "POST" }); await refresh(); toast("Status and gateway configuration updated."); } catch (error) { toggleButton.classList.toggle("on", wasOn); toggleButton.disabled = false; toast(error.message || "Could not update status.", "error"); } }
+  if (action === "toggle") { const toggleButton = event.target.closest(".toggle"), wasOn = toggleButton.classList.contains("on"); toggleButton.classList.toggle("on", !wasOn); toggleButton.disabled = true; const base = kind === "proxy" ? "proxies" : "sites"; try { await api(`/api/${base}/${card.dataset.id}/toggle`, { method: "POST" }); await refreshCurrentView(); toast("Status and gateway configuration updated."); } catch (error) { toggleButton.classList.toggle("on", wasOn); toggleButton.disabled = false; toast(error.message || "Could not update status.", "error"); } }
   if (action === "settings") openSettings(kind, card.dataset.id);
   if (action === "delete") { state.pendingDelete = { kind, id: card.dataset.id }; $("#confirm-title").textContent = kind === "proxy" ? "Delete this proxy host?" : "Delete this hosted site?"; $("#confirm-copy").textContent = kind === "proxy" ? "Its domain route will be removed from the gateway." : "Its route and uploaded files will be permanently removed."; $("#confirm-dialog").showModal(); }
   if (action === "replace") { state.pendingReplace = card.dataset.id; $("#replace-files").click(); }
@@ -771,7 +849,7 @@ window.openCaddyConfig = openCaddyConfig;
 
 
 // --- Delete confirmation dialog and replace-files handler ------------------------------------
-$("#confirm-dialog").addEventListener("close", async () => { if ($("#confirm-dialog").returnValue === "confirm" && state.pendingDelete) { const base = state.pendingDelete.kind === "proxy" ? "proxies" : "sites"; await api(`/api/${base}/${state.pendingDelete.id}`, { method: "DELETE" }); await refresh(); toast("Entry deleted and gateway updated."); } state.pendingDelete = null; });
+$("#confirm-dialog").addEventListener("close", async () => { if ($("#confirm-dialog").returnValue === "confirm" && state.pendingDelete) { const base = state.pendingDelete.kind === "proxy" ? "proxies" : "sites"; await api(`/api/${base}/${state.pendingDelete.id}`, { method: "DELETE" }); await refreshCurrentView(); toast("Entry deleted and gateway updated."); } state.pendingDelete = null; });
 $("#replace-files").addEventListener("change", async event => { if (!event.target.files[0] || !state.pendingReplace) return; const data = new FormData(); data.append("files", event.target.files[0]); try { await api(`/api/sites/${state.pendingReplace}/files`, { method: "POST", body: data }); toast("Site files updated."); } catch (error) { toast(error.message, "error"); } event.target.value = ""; state.pendingReplace = null; });
 
 

src/public/features.js

@@ -318,8 +318,20 @@ function normalizeRetentionLayout() { const form = document.querySelector('[data
 normalizeRetentionLayout();
 function cleanRetentionLabels() { const form = document.querySelector('[data-admin-panel="retention"] .retention-form'); if (!form) return; const descriptions = { 'Access logs':'High-volume request records.', 'Gateway activity':'Operational and configuration events.', 'Audit logs':'Administrative accountability records.', 'Certificate events':'Certificate issuance and health changes.', 'Security events':'Authentication and security-related events.' }; [...form.querySelectorAll('label:not(.check-control)')].forEach(field => { const text = field.firstChild; const name = text?.textContent?.trim().replace(/ \(days\)$/, ''); if (!text || !descriptions[name]) return; if (!text.textContent.includes('(days)')) text.textContent = `${name} (days)`; let help = field.querySelector('small'); if (!help) { help = document.createElement('small'); field.append(help); } help.textContent = descriptions[name]; }); }
 setTimeout(() => { cleanRetentionLabels(); normalizeRetentionLayout(); }, 0); setInterval(() => { cleanRetentionLabels(); normalizeRetentionLayout(); }, 300);
-function renderRetentionRunStatus() { if (!state.user) return; const panel = document.querySelector('[data-admin-panel="retention"]'); const form = panel?.querySelector('.retention-form'); if (!panel || !form) return; const value = state.settings?.logsRetention?.lastRunAt ? state.settings.logsRetention : null; let status = panel.querySelector('.retention-run-status'); if (!status) { status = document.createElement('div'); status.className = 'retention-run-status muted'; const actions = form.querySelector('.dialog-actions'); if (actions) actions.before(status); else form.append(status); } status.textContent = value ? `Last run: ${value.lastRunMode || 'manual'} · ${new Date(value.lastRunAt).toLocaleString()} · Snapshot: ${value.lastRunSnapshot || 'available'}` : 'No pruning run yet.'; }
-async function renderRetentionPreview() { if (!state.user) return; const panel = document.querySelector('[data-admin-panel="retention"]'); if (!panel) return; let preview = panel.querySelector('.retention-preview'); if (!preview) { preview = document.createElement('div'); preview.className = 'retention-preview muted'; const form = panel.querySelector('.retention-form'); const status = panel.querySelector('.retention-run-status'); (status || form)?.before(preview); } try { const data = await api('/api/logs/prune/preview'); const counts = data.counts || {}; const total = Object.values(counts).reduce((sum, value) => sum + Number(value || 0), 0); preview.textContent = data.enabled ? `Eligible to prune: ${total} records · Access ${counts.access || 0} · Activity ${counts.activity || 0} · Certificates ${counts.certificate || 0} · Security ${counts.security || 0} · Audit ${counts.audit || 0}` : 'Pruning is disabled. Enable automatic pruning to preview eligible records.'; } catch { preview.textContent = 'Prune preview unavailable.'; } }
+function renderRetentionRunStatus() { if (!state.user || !isRetentionPanelVisible()) return; const panel = document.querySelector('[data-admin-panel="retention"]'); const form = panel?.querySelector('.retention-form'); if (!panel || !form) return; const value = state.settings?.logsRetention?.lastRunAt ? state.settings.logsRetention : null; let status = panel.querySelector('.retention-run-status'); if (!status) { status = document.createElement('div'); status.className = 'retention-run-status muted'; const actions = form.querySelector('.dialog-actions'); if (actions) actions.before(status); else form.append(status); } status.textContent = value ? `Last run: ${value.lastRunMode || 'manual'} · ${new Date(value.lastRunAt).toLocaleString()} · Snapshot: ${value.lastRunSnapshot || 'available'}` : 'No pruning run yet.'; }
+// The retention panel's <section> lives in index.html from page load (just CSS-hidden until its
+// admin tab is selected), so "does the panel element exist" was never a real visibility check --
+// it's always true, on every page of the app. That let this run forever, everywhere, not just on
+// Administration > Logs & retention. Combined with no guard against overlapping calls, a single
+// slow response (previewPruneEvents() runs several SQLite COUNT queries, synchronously, blocking
+// the whole server while they run) let requests pile up faster than the server could drain them --
+// confirmed via a live Network-tab capture showing this same request queued for 15+ seconds while
+// unrelated requests (dashboard, health, security, storage) sat stuck at nearly the same time,
+// waiting behind it. isRetentionPanelVisible() below checks the panel is both present AND not
+// hidden, and retentionPreviewInFlight prevents a new poll from starting until the last one lands.
+function isRetentionPanelVisible() { const panel = document.querySelector('[data-admin-panel="retention"]'); return Boolean(panel && !panel.classList.contains("hidden")); }
+let retentionPreviewInFlight = false;
+async function renderRetentionPreview() { if (!state.user || !isRetentionPanelVisible() || retentionPreviewInFlight) return; const panel = document.querySelector('[data-admin-panel="retention"]'); if (!panel) return; let preview = panel.querySelector('.retention-preview'); if (!preview) { preview = document.createElement('div'); preview.className = 'retention-preview muted'; const form = panel.querySelector('.retention-form'); const status = panel.querySelector('.retention-run-status'); (status || form)?.before(preview); } retentionPreviewInFlight = true; try { const data = await api('/api/logs/prune/preview'); const counts = data.counts || {}; const total = Object.values(counts).reduce((sum, value) => sum + Number(value || 0), 0); preview.textContent = data.enabled ? `Eligible to prune: ${total} records · Access ${counts.access || 0} · Activity ${counts.activity || 0} · Certificates ${counts.certificate || 0} · Security ${counts.security || 0} · Audit ${counts.audit || 0}` : 'Pruning is disabled. Enable automatic pruning to preview eligible records.'; } catch { preview.textContent = 'Prune preview unavailable.'; } finally { retentionPreviewInFlight = false; } }
 async function renderRetentionHistory() { if (!state.user) return; const panel = document.querySelector('[data-admin-panel="retention"]'); if (!panel) return; let history = panel.querySelector('.retention-history'); if (!history) { history = document.createElement('div'); history.className = 'retention-history'; (panel.querySelector('.retention-run-status') || panel.querySelector('.retention-form'))?.after(history); } try { const rows = (await api('/api/audit?action=pruning')).filter(item => /pruning/i.test(item.action)).slice(0, 50); history.innerHTML = `<div class="retention-history-heading"><strong>Prune history</strong><span>${rows.length} runs</span></div>` + (rows.length ? `<div class="retention-history-list">${rows.map(item => `<div class="retention-history-row"><span class="status-dot ${item.status === 'error' ? 'disabled' : 'running'}"></span><span><strong>${extendedEscape(item.action)}</strong><small>${extendedEscape(item.actor || 'System')} · ${extendedEscape(item.status === 'error' ? 'Failed' : 'Success')} · ${extendedEscape(formatTime(item.created_at))}</small></span></div>`).join('')}</div>` : '<p class="muted">No pruning runs recorded yet.</p>'); } catch { history.innerHTML = '<p class="muted">Prune history unavailable.</p>'; } }
 function ensureRetentionLoadMore() { const history = document.querySelector('.retention-history'); if (!history || history.querySelector('[data-retention-load-more]')) return; const button = document.createElement('button'); button.className = 'text-button retention-load-more'; button.dataset.retentionLoadMore = 'true'; button.textContent = 'Load more'; history.append(button); }
 document.addEventListener('click', async event => { const button = event.target.closest('[data-retention-load-more]'); if (!button) return; try { const rows = (await api('/api/audit?action=pruning')).filter(item => /pruning/i.test(item.action)).slice(50); const list = button.parentElement.querySelector('.retention-history-list'); rows.forEach(item => { const row = document.createElement('div'); row.className = 'retention-history-row'; row.innerHTML = `<span class="status-dot ${item.status === 'error' ? 'disabled' : 'running'}"></span><span><strong>${extendedEscape(item.action)}</strong><small>${extendedEscape(item.actor || 'System')} · ${extendedEscape(item.status === 'error' ? 'Failed' : 'Success')} · ${extendedEscape(formatTime(item.created_at))}</small></span>`; list?.append(row); }); button.remove(); } catch { button.textContent = 'History unavailable'; } });
@@ -691,11 +703,19 @@ function renderSystemPanel() {
     if (!state.systemHealthTimer) state.systemHealthTimer = setInterval(() => {
       const systemPanel = document.querySelector('[data-admin-panel="system"]');
       if (state.view !== "administration" || !systemPanel || systemPanel.classList.contains("hidden")) return;
-      api("/api/system/health").then(health => renderHeroPanel("system-hero", health)).catch(() => {});
+      api("/api/system/health").then(health => { renderHeroPanel("system-hero", health); updateSystemVersionUptime(health); }).catch(() => {});
     }, 7000);
   }
   renderSystemStatus(panel);
 }
+// Updates just the Version panel's Uptime figure from a fresh /api/system/health payload --
+// kept separate from the rest of renderSystemStatus() so it can be called on the fast 7s hero
+// poll without re-rendering (or re-fetching) everything else in that panel.
+function updateSystemVersionUptime(health) {
+  const el = document.querySelector("#system-version-uptime");
+  if (!el) return;
+  el.textContent = Number.isFinite(health?.uptimeSeconds) ? formatDuration(health.uptimeSeconds) : "Unavailable";
+}
 async function renderSystemStatus(panel) {
   panel = panel || document.querySelector('[data-admin-panel="system"]');
   if (!panel) return;
@@ -715,16 +735,19 @@ async function renderSystemStatus(panel) {
   }
   if (syncStatus) { const drift = (state.dashboard?.attention || []).some(item => item.kind === "drift"); syncStatus.textContent = drift ? "Configuration drift detected \u2014 the running gateway no longer matches the last known-good configuration." : `Gateway configuration is in sync. Last reload: ${state.dashboard?.gateway?.lastReload ? formatTime(state.dashboard.gateway.lastReload) : "unknown"}.`; syncStatus.className = drift ? "muted status-warning" : "muted"; }
   if (version) {
-    // Uptime, Caddy version, Database status, and Public IP used to live on the Dashboard's
-    // Runtime/System panel -- that panel is now the shared hero component (CPU/memory/swap/disk/
-    // network), so this operational metadata moved here instead, reusing the same system.* fields
-    // from the global dashboard snapshot rather than a separate fetch.
+    // Caddy version, Database status, and Public IP used to live on the Dashboard's Runtime/System
+    // panel -- that panel is now the shared hero component (CPU/memory/swap/disk/network), so this
+    // operational metadata moved here instead, reusing the same system.* fields from the global
+    // dashboard snapshot rather than a separate fetch. Uptime is the one exception: it's wrapped in
+    // its own #system-version-uptime span and kept current by updateSystemVersionUptime(), called
+    // from the same 7-second /api/system/health poll that drives the hero panel above, instead of
+    // only refreshing on the slower ~30s dashboard snapshot like the rest of this block.
     const sys = state.dashboard?.system || {};
     const uptime = Number.isFinite(sys.uptimeSeconds) ? formatDuration(sys.uptimeSeconds) : "Unavailable";
     const database = sys.databaseEngine ? `${extendedEscape(sys.databaseEngine)} \u00b7 ${extendedEscape(sys.databaseStatus || "unknown")} \u00b7 ${formatBytes(sys.databaseBytes)}` : "Unavailable";
     const publicIp = sys.publicIp || (sys.publicIpError ? "Unavailable" : "Checking\u2026");
     const publicIpDetail = sys.publicIpError ? `check failed \u00b7 ${extendedEscape(sys.publicIpError)}` : sys.publicIpCheckedAt ? `checked ${extendedEscape(formatTime(sys.publicIpCheckedAt))}` : "not yet checked";
-    version.innerHTML = `Site Gateway v${extendedEscape(state.config?.version || "unknown")} \u00b7 Caddy ${extendedEscape(sys.caddyVersion || "unknown")}<br>Uptime: ${uptime} \u00b7 Database: ${database} \u00b7 Public IP: ${extendedEscape(publicIp)} (${publicIpDetail})<br>Access this dashboard at: <code>${extendedEscape(location.origin)}</code><br>Data directory: <code>${extendedEscape(state.config?.storage?.databasePath ? state.config.storage.databasePath.replace(/\/database\/.*/, "") : "/data")}</code> &middot; Site ports: <code>${extendedEscape(String(state.config?.minPort ?? ""))}\u2013${extendedEscape(String(state.config?.maxPort ?? ""))}</code>`;
+    version.innerHTML = `Site Gateway v${extendedEscape(state.config?.version || "unknown")} \u00b7 Caddy ${extendedEscape(sys.caddyVersion || "unknown")}<br>Uptime: <span id="system-version-uptime">${uptime}</span> \u00b7 Database: ${database} \u00b7 Public IP: ${extendedEscape(publicIp)} (${publicIpDetail})<br>Access this dashboard at: <code>${extendedEscape(location.origin)}</code><br>Data directory: <code>${extendedEscape(state.config?.storage?.databasePath ? state.config.storage.databasePath.replace(/\/database\/.*/, "") : "/data")}</code> &middot; Site ports: <code>${extendedEscape(String(state.config?.minPort ?? ""))}\u2013${extendedEscape(String(state.config?.maxPort ?? ""))}</code>`;
   }
   try {
     const [sec, store, policy, health] = await Promise.all([
@@ -734,6 +757,7 @@ async function renderSystemStatus(panel) {
       api("/api/system/health").catch(() => null),
     ]);
     renderHeroPanel("system-hero", health);
+    if (health) updateSystemVersionUptime(health);
     if (security) security.innerHTML = [
       { ok: !sec.adminPasswordIsDefault, label: "ADMIN_PASSWORD", detail: sec.adminPasswordIsDefault ? "Still using the built-in default \u2014 set this before exposing the dashboard." : "Configured." },
       { ok: !sec.sessionSecretIsDefault, label: "SESSION_SECRET", detail: sec.sessionSecretIsDefault ? "Not set \u2014 sessions are keyed off the admin credentials instead of an independent secret." : "Configured." },

src/server.js

@@ -64,6 +64,14 @@ let caddyVersion = "Unknown";
 const recentActivity = [];
 const upstreamHealth = new Map();
 const certificateStatusCache = new Map();
+// Certificate inventory is expensive (it walks and parses every certificate file on disk) and is
+// recomputed on every call with no memoization. It's called twice per client refresh() cycle --
+// once from /api/dashboard, once from /api/certificates -- and that whole cycle can itself repeat
+// several times in a row (see refreshPendingProxies in app.js), so a short time-based cache here
+// collapses that duplicate work into a single real disk walk every few seconds. The window is kept
+// well under the 7s hero-poll interval so nothing ever appears more than one cycle stale.
+const CERTIFICATE_INVENTORY_CACHE_MS = 3000;
+let certificateInventoryCache = null; // { at: number, value: object }
 const loginAttempts = new Map();
 const rateLimitBuckets = new Map();
 let dockerSocketMounted = false;
@@ -882,6 +890,7 @@ function certificateNames(certificate) {
 }
 
 async function certificateInventory() {
+  if (certificateInventoryCache && Date.now() - certificateInventoryCache.at < CERTIFICATE_INVENTORY_CACHE_MS) return certificateInventoryCache.value;
   const configured = [...sites.map(item => ({ ...item, kind: "Hosted site" })), ...proxies.map(item => ({ ...item, kind: "Proxy host" })), ...redirects.map(item => ({ ...item, kind: "Redirect host" }))]
     .filter(item => item.enabled && item.domain && item.tls !== "http");
   const configuredDomains = configured.flatMap(item => normalizeDomains(item.domain, item.domains).map(domain => ({ ...item, domain })));
@@ -908,7 +917,9 @@ async function certificateInventory() {
   });
   for (const certificate of certificates) { const previous = certificateStatusCache.get(certificate.domain); if (previous && previous !== certificate.status) recordActivity(`Certificate status changed for ${certificate.domain}: ${previous} → ${certificate.status}.`, certificate.status === "healthy" ? "ok" : "error"); certificateStatusCache.set(certificate.domain, certificate.status); }
   const latestError = recentActivity.find(item => item.status === "error" && /cert|tls|acme|caddy|gateway/i.test(item.message)) || null;
-  return { checkedAt: new Date().toISOString(), thresholds: settings.certificateHealth, latestError, summary: { total: certificates.length, healthy: certificates.filter(item => item.status === "healthy").length, within30Days: certificates.filter(item => item.daysRemaining != null && item.daysRemaining <= 30 && item.daysRemaining > 0).length, within7Days: certificates.filter(item => item.daysRemaining != null && item.daysRemaining <= 7 && item.daysRemaining > 0).length, warning: certificates.filter(item => item.status === "warning").length, critical: certificates.filter(item => item.status === "critical").length, expired: certificates.filter(item => item.status === "expired").length, pending: certificates.filter(item => item.status === "pending").length, mismatch: certificates.filter(item => item.status === "mismatch").length }, certificates };
+  const result = { checkedAt: new Date().toISOString(), thresholds: settings.certificateHealth, latestError, summary: { total: certificates.length, healthy: certificates.filter(item => item.status === "healthy").length, within30Days: certificates.filter(item => item.daysRemaining != null && item.daysRemaining <= 30 && item.daysRemaining > 0).length, within7Days: certificates.filter(item => item.daysRemaining != null && item.daysRemaining <= 7 && item.daysRemaining > 0).length, warning: certificates.filter(item => item.status === "warning").length, critical: certificates.filter(item => item.status === "critical").length, expired: certificates.filter(item => item.status === "expired").length, pending: certificates.filter(item => item.status === "pending").length, mismatch: certificates.filter(item => item.status === "mismatch").length }, certificates };
+  certificateInventoryCache = { at: Date.now(), value: result };
+  return result;
 }
 
 async function pruneOrphanedCertificates(candidateDomains) {
@@ -1007,12 +1018,30 @@ async function readAccessLogs(limit = 100, host = "") {
   return entries;
 }
 
+// Diagnostic timing (v0.16.44): this job reads Caddy's access-log files, JSON.parses up to 5000
+// lines, hashes each one, and batch-inserts them -- all synchronous work that runs on Node's single
+// thread and therefore blocks every other request in the app for its full duration, every time it
+// runs (every 30 seconds). A user reported requests as simple as GET /api/sites randomly taking
+// 6-14 seconds with a Network-tab Timing capture showing nearly all of it as server-side "Waiting"
+// (TTFB) rather than connection/DNS time -- consistent with getting stuck behind a job like this
+// one. Logging real durations here (only when a run takes long enough to plausibly explain that)
+// gives proof of whether this is the actual cause before changing how it works, rather than
+// shipping a fourth guess.
 async function importAccessLogsToSqlite() {
   if (!storage?.recordAccessEvents) return;
+  const startedAt = performance.now();
   try {
+    const readStarted = performance.now();
     const entries = await readAccessLogs(5000);
+    const readMs = Math.round(performance.now() - readStarted);
+    const hashStarted = performance.now();
     const events = entries.map(entry => ({ ...entry, source: crypto.createHash("sha1").update(JSON.stringify([entry.at, entry.host, entry.method, entry.uri, entry.status, entry.size, entry.durationMs, entry.remoteIp])).digest("hex") }));
+    const hashMs = Math.round(performance.now() - hashStarted);
+    const insertStarted = performance.now();
     storage.recordAccessEvents(events);
+    const insertMs = Math.round(performance.now() - insertStarted);
+    const totalMs = Math.round(performance.now() - startedAt);
+    if (totalMs > 500) console.warn(`[perf] importAccessLogsToSqlite took ${totalMs}ms for ${entries.length} entries (read ${readMs}ms, hash ${hashMs}ms, insert ${insertMs}ms) -- this blocks every other request while it runs.`);
   } catch (error) { console.warn("Could not import access logs into SQLite:", error.message); }
 }
 
@@ -1483,6 +1512,20 @@ app.disable("x-powered-by");
 // HTTP layer: Express app setup, auth middleware, and every /api/* route.
 // Routes below are grouped by area; see the section comments for each group.
 // ============================================================================================
+// Diagnostic timing (v0.16.44): logs any request that takes noticeably long to answer, alongside
+// the importAccessLogsToSqlite instrumentation above -- together these should show, in the
+// container's own logs, whether a slow page load lines up with a background job's run window or
+// is a slow request in its own right. Placed first so it wraps the full request, including any
+// auth/body-parsing work below it. Remove once the real cause behind reported multi-second page
+// loads is confirmed and fixed; this is a temporary aid, not a permanent feature.
+app.use((req, res, next) => {
+  const startedAt = performance.now();
+  res.on("finish", () => {
+    const durationMs = Math.round(performance.now() - startedAt);
+    if (durationMs > 1000) console.warn(`[perf] ${req.method} ${req.originalUrl} took ${durationMs}ms`);
+  });
+  next();
+});
 app.use(express.json());
 app.use(express.urlencoded({ extended: false }));
 app.get(["/", "/index.html"], (req, res) => {

src/storage.js

@@ -64,6 +64,7 @@ export async function openStorage(dataDir, backupsDir) {
     CREATE TABLE IF NOT EXISTS access_assignments (instance_id TEXT NOT NULL REFERENCES instances(id) ON DELETE CASCADE, route_kind TEXT NOT NULL, route_id TEXT NOT NULL, access_list_id TEXT NOT NULL REFERENCES access_lists(id) ON DELETE RESTRICT, created_at TEXT NOT NULL, PRIMARY KEY(route_kind,route_id));
     CREATE TABLE IF NOT EXISTS settings (instance_id TEXT PRIMARY KEY REFERENCES instances(id) ON DELETE CASCADE, payload TEXT NOT NULL CHECK(json_valid(payload)), updated_at TEXT NOT NULL);
     CREATE TABLE IF NOT EXISTS audit_events (id INTEGER PRIMARY KEY AUTOINCREMENT, instance_id TEXT REFERENCES instances(id), actor_id TEXT, action TEXT NOT NULL, status TEXT NOT NULL, details TEXT, created_at TEXT NOT NULL);
+    CREATE INDEX IF NOT EXISTS audit_events_instance_created ON audit_events(instance_id,created_at DESC);
     CREATE TABLE IF NOT EXISTS activity_events (id INTEGER PRIMARY KEY AUTOINCREMENT, instance_id TEXT REFERENCES instances(id), message TEXT NOT NULL, status TEXT NOT NULL, category TEXT NOT NULL DEFAULT 'activity', created_at TEXT NOT NULL);
     CREATE INDEX IF NOT EXISTS activity_events_instance_created ON activity_events(instance_id,created_at DESC);
     CREATE TABLE IF NOT EXISTS access_events (id INTEGER PRIMARY KEY AUTOINCREMENT, instance_id TEXT REFERENCES instances(id), at TEXT, host TEXT, method TEXT, uri TEXT, status INTEGER, size INTEGER, duration_ms INTEGER, remote_ip TEXT, source TEXT, UNIQUE(instance_id,source));