marvin/site-gateway
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Stop the retention-preview timer from polling every page forever and index audit_events

Browse Source
This commit is contained in:
marvin
2026-09-19 22:42:35 -04:00
parent 578d39a3ad
commit 926296d0d7
6 changed files with 21 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1 -1
View File
@@ -12,7 +12,7 @@
<img alt="Docker" src="https://img.shields.io/badge/Docker-ready-2496ED?logo=docker&logoColor=white">
<img alt="Architectures" src="https://img.shields.io/badge/platform-amd64%20%7C%20arm64-5965F2">
<img alt="Caddy" src="https://img.shields.io/badge/powered%20by-Caddy-1F88C0">
<img alt="Version" src="https://img.shields.io/badge/version-0.16.41-62E6A7">
<img alt="Version" src="https://img.shields.io/badge/version-0.16.42-62E6A7">
</p>
<p>
<a href="#why-site-gateway">Why Site Gateway</a> ·
ROADMAP.md
+2
View File
@@ -210,3 +210,5 @@ Roughly in priority order:
`v0.16.40` moves the Administration → System tab's Version panel Uptime figure onto the same 7-second `/api/system/health` poll driving the hero panel above it, instead of only updating when the page's slower, general dashboard snapshot refetches (on load, or every ~30 seconds while the Dashboard view specifically is active). Found while reviewing the v0.16.39 change: the hero's own Uptime slot was now current to the second, but the separate Version-panel line right below it — the same number, shown twice on the same page — could still be stale by up to half a minute or more. `renderSystemStatus()`'s Uptime is now wrapped in its own `#system-version-uptime` span and updated by a small `updateSystemVersionUptime(health)` helper, called both from the initial render and from the same 7-second timer that already refreshes the hero, so the two Uptime figures on that page can no longer drift apart.
`v0.16.41` cuts redundant work out of the app's shared `refresh()` cycle -- the single function that populates nearly every page (Hosted Sites, Proxy Hosts, Redirects, Streams, Access Lists, Dashboard, and Certificates all pull from it) -- after a user reported the whole site feeling slow to refresh, most concretely on a plain reload of Hosted Sites or Proxy Hosts. Two real causes, found by reading the actual request path rather than guessing: first, `certificateInventory()` (which walks the certificate directories and parses every `.crt`/`.pem` file on disk) was being fully recomputed from scratch on every single call, and `refresh()` calls it twice per cycle -- once via `/api/dashboard`, once via `/api/certificates` -- so a normal page load did that walk-and-parse work twice for identical results. It now carries a short (3 second) in-memory cache, well under the 7-second hero-poll interval, so back-to-back calls within a cycle share one real disk walk instead of two, and nothing on screen goes more than one cycle stale. Second, and the bigger one: `refreshPendingProxies()` -- triggered whenever a page loads with any enabled proxy that doesn't have cached upstream-health data yet, which is the common case right after a page load or a new proxy -- was calling the *entire* `refresh()` again at +1s, +2s, and +3s until every proxy's health came back. That meant a single pending proxy could quietly trigger three additional full 8-endpoint refetches (each with its own pair of certificate walks) in the six seconds after a page appeared to have finished loading. It now re-fetches only `/api/proxies` on those retries, since upstream health is all it was ever waiting on. Together these remove the two largest sources of duplicated, unnecessary work from the most-used code path in the app; whether they fully account for the reported slowness or whether a client-side rendering cost remains to find is still open and being evaluated against a real before/after comparison.
`v0.16.42` finds and fixes the real, dominant cause of the site-wide slowness reported after v0.16.41: a live Network-tab capture from the user's own browser showed a flood of requests to `/api/logs/prune/preview`, some queued for over 15 seconds, with unrelated requests (`/api/dashboard`, `/api/system/health`, `/api/system/security`, `/api/system/storage`) stuck at nearly identical multi-second times in the same batch -- the signature of one blocking operation stalling everything behind it, not several independently slow endpoints. Root cause: `renderRetentionPreview()`'s `setInterval(..., 2000)` polls that endpoint every 2 seconds forever, on every page of the app, not just Administration -> Logs & retention, because its "does the panel exist" guard checks a `<section>` that's written into `index.html` from page load and only ever CSS-hidden -- so the guard was always true, everywhere. There was also no protection against a new poll firing while a previous one was still in flight, so once the server answered slower than 2 seconds even once, requests piled up and never caught back up. Compounding it: `previewPruneEvents()` runs five synchronous SQLite COUNT queries, and one of them (`audit_events`) had no index at all -- a full table scan, every call -- and because this app's SQLite queries run synchronously, that scan doesn't just slow its own request, it blocks the entire Node process for every other request being served at that moment. Fixed on both sides: `renderRetentionPreview()` and the sibling `renderRetentionRunStatus()` (previously also running unconditionally every 500ms) now check that the retention panel is actually visible, not just present in the DOM, before doing any work, and an in-flight guard stops a new preview poll from starting until the last one has landed; `audit_events` now has the same `(instance_id, created_at)` index every sibling events table already had. Together these should remove the vast majority of the "8-10 seconds to load a simple page" behavior reported after v0.16.41 -- that fix (deduplicating certificate-inventory work) was real but minor by comparison to this one.
package.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "site-gateway",
"version": "0.16.41",
"version": "0.16.42",
"private": true,
"description": "Site Gateway: simple self-hosted website publishing, reverse proxying, and automatic HTTPS.",
"type": "module",
src/public/features.js
+14 -2
View File
@@ -318,8 +318,20 @@ function normalizeRetentionLayout() { const form = document.querySelector('[data
normalizeRetentionLayout();
function cleanRetentionLabels() { const form = document.querySelector('[data-admin-panel="retention"] .retention-form'); if (!form) return; const descriptions = { 'Access logs':'High-volume request records.', 'Gateway activity':'Operational and configuration events.', 'Audit logs':'Administrative accountability records.', 'Certificate events':'Certificate issuance and health changes.', 'Security events':'Authentication and security-related events.' }; [...form.querySelectorAll('label:not(.check-control)')].forEach(field => { const text = field.firstChild; const name = text?.textContent?.trim().replace(/ \(days\)$/, ''); if (!text || !descriptions[name]) return; if (!text.textContent.includes('(days)')) text.textContent = `${name} (days)`; let help = field.querySelector('small'); if (!help) { help = document.createElement('small'); field.append(help); } help.textContent = descriptions[name]; }); }
setTimeout(() => { cleanRetentionLabels(); normalizeRetentionLayout(); }, 0); setInterval(() => { cleanRetentionLabels(); normalizeRetentionLayout(); }, 300);
function renderRetentionRunStatus() { if (!state.user) return; const panel = document.querySelector('[data-admin-panel="retention"]'); const form = panel?.querySelector('.retention-form'); if (!panel || !form) return; const value = state.settings?.logsRetention?.lastRunAt ? state.settings.logsRetention : null; let status = panel.querySelector('.retention-run-status'); if (!status) { status = document.createElement('div'); status.className = 'retention-run-status muted'; const actions = form.querySelector('.dialog-actions'); if (actions) actions.before(status); else form.append(status); } status.textContent = value ? `Last run: ${value.lastRunMode || 'manual'} · ${new Date(value.lastRunAt).toLocaleString()} · Snapshot: ${value.lastRunSnapshot || 'available'}` : 'No pruning run yet.'; }
async function renderRetentionPreview() { if (!state.user) return; const panel = document.querySelector('[data-admin-panel="retention"]'); if (!panel) return; let preview = panel.querySelector('.retention-preview'); if (!preview) { preview = document.createElement('div'); preview.className = 'retention-preview muted'; const form = panel.querySelector('.retention-form'); const status = panel.querySelector('.retention-run-status'); (status || form)?.before(preview); } try { const data = await api('/api/logs/prune/preview'); const counts = data.counts || {}; const total = Object.values(counts).reduce((sum, value) => sum + Number(value || 0), 0); preview.textContent = data.enabled ? `Eligible to prune: ${total} records · Access ${counts.access || 0} · Activity ${counts.activity || 0} · Certificates ${counts.certificate || 0} · Security ${counts.security || 0} · Audit ${counts.audit || 0}` : 'Pruning is disabled. Enable automatic pruning to preview eligible records.'; } catch { preview.textContent = 'Prune preview unavailable.'; } }
function renderRetentionRunStatus() { if (!state.user || !isRetentionPanelVisible()) return; const panel = document.querySelector('[data-admin-panel="retention"]'); const form = panel?.querySelector('.retention-form'); if (!panel || !form) return; const value = state.settings?.logsRetention?.lastRunAt ? state.settings.logsRetention : null; let status = panel.querySelector('.retention-run-status'); if (!status) { status = document.createElement('div'); status.className = 'retention-run-status muted'; const actions = form.querySelector('.dialog-actions'); if (actions) actions.before(status); else form.append(status); } status.textContent = value ? `Last run: ${value.lastRunMode || 'manual'} · ${new Date(value.lastRunAt).toLocaleString()} · Snapshot: ${value.lastRunSnapshot || 'available'}` : 'No pruning run yet.'; }
// The retention panel's <section> lives in index.html from page load (just CSS-hidden until its
// admin tab is selected), so "does the panel element exist" was never a real visibility check --
// it's always true, on every page of the app. That let this run forever, everywhere, not just on
// Administration > Logs & retention. Combined with no guard against overlapping calls, a single
// slow response (previewPruneEvents() runs several SQLite COUNT queries, synchronously, blocking
// the whole server while they run) let requests pile up faster than the server could drain them --
// confirmed via a live Network-tab capture showing this same request queued for 15+ seconds while
// unrelated requests (dashboard, health, security, storage) sat stuck at nearly the same time,
// waiting behind it. isRetentionPanelVisible() below checks the panel is both present AND not
// hidden, and retentionPreviewInFlight prevents a new poll from starting until the last one lands.
function isRetentionPanelVisible() { const panel = document.querySelector('[data-admin-panel="retention"]'); return Boolean(panel && !panel.classList.contains("hidden")); }
let retentionPreviewInFlight = false;
async function renderRetentionPreview() { if (!state.user || !isRetentionPanelVisible() || retentionPreviewInFlight) return; const panel = document.querySelector('[data-admin-panel="retention"]'); if (!panel) return; let preview = panel.querySelector('.retention-preview'); if (!preview) { preview = document.createElement('div'); preview.className = 'retention-preview muted'; const form = panel.querySelector('.retention-form'); const status = panel.querySelector('.retention-run-status'); (status || form)?.before(preview); } retentionPreviewInFlight = true; try { const data = await api('/api/logs/prune/preview'); const counts = data.counts || {}; const total = Object.values(counts).reduce((sum, value) => sum + Number(value || 0), 0); preview.textContent = data.enabled ? `Eligible to prune: ${total} records · Access ${counts.access || 0} · Activity ${counts.activity || 0} · Certificates ${counts.certificate || 0} · Security ${counts.security || 0} · Audit ${counts.audit || 0}` : 'Pruning is disabled. Enable automatic pruning to preview eligible records.'; } catch { preview.textContent = 'Prune preview unavailable.'; } finally { retentionPreviewInFlight = false; } }
async function renderRetentionHistory() { if (!state.user) return; const panel = document.querySelector('[data-admin-panel="retention"]'); if (!panel) return; let history = panel.querySelector('.retention-history'); if (!history) { history = document.createElement('div'); history.className = 'retention-history'; (panel.querySelector('.retention-run-status') || panel.querySelector('.retention-form'))?.after(history); } try { const rows = (await api('/api/audit?action=pruning')).filter(item => /pruning/i.test(item.action)).slice(0, 50); history.innerHTML = `<div class="retention-history-heading"><strong>Prune history</strong><span>${rows.length} runs</span></div>` + (rows.length ? `<div class="retention-history-list">${rows.map(item => `<div class="retention-history-row"><span class="status-dot ${item.status === 'error' ? 'disabled' : 'running'}"></span><span><strong>${extendedEscape(item.action)}</strong><small>${extendedEscape(item.actor || 'System')} · ${extendedEscape(item.status === 'error' ? 'Failed' : 'Success')} · ${extendedEscape(formatTime(item.created_at))}</small></span></div>`).join('')}</div>` : '<p class="muted">No pruning runs recorded yet.</p>'); } catch { history.innerHTML = '<p class="muted">Prune history unavailable.</p>'; } }
function ensureRetentionLoadMore() { const history = document.querySelector('.retention-history'); if (!history || history.querySelector('[data-retention-load-more]')) return; const button = document.createElement('button'); button.className = 'text-button retention-load-more'; button.dataset.retentionLoadMore = 'true'; button.textContent = 'Load more'; history.append(button); }
document.addEventListener('click', async event => { const button = event.target.closest('[data-retention-load-more]'); if (!button) return; try { const rows = (await api('/api/audit?action=pruning')).filter(item => /pruning/i.test(item.action)).slice(50); const list = button.parentElement.querySelector('.retention-history-list'); rows.forEach(item => { const row = document.createElement('div'); row.className = 'retention-history-row'; row.innerHTML = `<span class="status-dot ${item.status === 'error' ? 'disabled' : 'running'}"></span><span><strong>${extendedEscape(item.action)}</strong><small>${extendedEscape(item.actor || 'System')} · ${extendedEscape(item.status === 'error' ? 'Failed' : 'Success')} · ${extendedEscape(formatTime(item.created_at))}</small></span>`; list?.append(row); }); button.remove(); } catch { button.textContent = 'History unavailable'; } });
src/public/index.html
+2 -2
View File
@@ -8,7 +8,7 @@
<title>Site Gateway</title>
<meta name="description" content="Host sites, proxy services, and manage HTTPS from one simple dashboard.">
<link rel="icon" type="image/png" href="/site-gateway-icon-approved.png">
<link rel="stylesheet" href="/styles.css?v=0.16.41">
<link rel="stylesheet" href="/styles.css?v=0.16.42">
</head>
<!-- ================================================================
@@ -432,6 +432,6 @@
<div id="toast" class="toast" role="status"></div>
<div id="update-banner" class="update-banner hidden" role="status"><span>A new version of Site Gateway is available.</span><div class="update-banner-actions"><button id="update-banner-refresh" class="button primary">Refresh</button><button id="update-banner-dismiss" class="text-button">Dismiss</button></div></div>
<!-- App scripts: core (app.js) then extended views/admin (features.js) -->
<script src="/app.js?v=0.16.41" defer></script><script src="/features.js?v=0.16.41" defer></script><script src="/select-enhance.js?v=0.16.41" defer></script>
<script src="/app.js?v=0.16.42" defer></script><script src="/features.js?v=0.16.42" defer></script><script src="/select-enhance.js?v=0.16.42" defer></script>
</body>
</html>
src/storage.js
+1
View File
@@ -64,6 +64,7 @@ export async function openStorage(dataDir, backupsDir) {
CREATE TABLE IF NOT EXISTS access_assignments (instance_id TEXT NOT NULL REFERENCES instances(id) ON DELETE CASCADE, route_kind TEXT NOT NULL, route_id TEXT NOT NULL, access_list_id TEXT NOT NULL REFERENCES access_lists(id) ON DELETE RESTRICT, created_at TEXT NOT NULL, PRIMARY KEY(route_kind,route_id));
CREATE TABLE IF NOT EXISTS settings (instance_id TEXT PRIMARY KEY REFERENCES instances(id) ON DELETE CASCADE, payload TEXT NOT NULL CHECK(json_valid(payload)), updated_at TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS audit_events (id INTEGER PRIMARY KEY AUTOINCREMENT, instance_id TEXT REFERENCES instances(id), actor_id TEXT, action TEXT NOT NULL, status TEXT NOT NULL, details TEXT, created_at TEXT NOT NULL);
CREATE INDEX IF NOT EXISTS audit_events_instance_created ON audit_events(instance_id,created_at DESC);
CREATE TABLE IF NOT EXISTS activity_events (id INTEGER PRIMARY KEY AUTOINCREMENT, instance_id TEXT REFERENCES instances(id), message TEXT NOT NULL, status TEXT NOT NULL, category TEXT NOT NULL DEFAULT 'activity', created_at TEXT NOT NULL);
CREATE INDEX IF NOT EXISTS activity_events_instance_created ON activity_events(instance_id,created_at DESC);
CREATE TABLE IF NOT EXISTS access_events (id INTEGER PRIMARY KEY AUTOINCREMENT, instance_id TEXT REFERENCES instances(id), at TEXT, host TEXT, method TEXT, uri TEXT, status INTEGER, size INTEGER, duration_ms INTEGER, remote_ip TEXT, source TEXT, UNIQUE(instance_id,source));