Service Registry

MOC for all OpenClaw services running on the Hetzner VPS (srv1347501). Covers systemd user-scope services (79 enumerated 2026-05-03), PM2 processes (8 active), and key system-scope daemons. The canonical live state is queried via systemctl --user list-units --type=service --all + pm2 list. Read this hub when debugging service restarts, checking process managers, or planning new deployments. Owner: Henry Hill / Claude Opus 4.7.

⚠️ 2 NEW FAILED SERVICES (PF-A, 2026-05-03):

1. investorlift-cookie-refresh.service — Playwright login fails when AWS Mac Ultra is impaired (403 from InvestorLift; VPS IP is CloudFront-blocked). Last failure: 2026-05-03 13:00 UTC. Root cause: per CLAUDE.md InvestorLift Scraping section, cookie refresh MUST run on EC2 Mac Ultra (ec2-user@100.123.248.46), NOT VPS Playwright. Fix: reroute service to SSH → Mac → run cookie refresh. Status: G-FAILED-SERVICE-MTTR clock started 2026-05-03.
2. perplexity-daily-summary.service — TypeError: unsupported operand type(s) for +: 'int' and 'NoneType' in daily_summary.py line 57 (estimated_cost_usd field is None for some records). Last failure: 2026-05-03 16:00 UTC. Fix: add null-coalescing guard in summarize(). Status: G-FAILED-SERVICE-MTTR clock started 2026-05-03.

Quick reference

Field	Value
Canonical live state	systemctl --user list-units --type=service --all + pm2 list
Live state probe	systemctl --user list-units --type=service --all | grep -c .service
Documented count	~90 (source: ARCHITECTURE + CLAUDE.md)
Live count	79 systemd user services (probed: 2026-05-03) + 8 PM2 processes
Drift	−11 from documented (acceptable — some timers-only services counted differently)
Last audit	2026-05-03
Drift alert	G-FAILED-SERVICE-MTTR daily cron (checks --state=failed, Discord ops if >24h)
Governance gate	G-SERVICE-PRE-START-DOC · G-FAILED-SERVICE-MTTR
Owner agent	system / claude-code

Components

• [[workspace/ARCHITECTURE]] — service table (authoritative service descriptions)
• [[workspace/port-registry]] — port-to-service mapping
• [[workspace/FUNNEL-REGISTRY]] — webhook-specific service routing
• [[wiki/_hubs/systems/port-registry]] — port registry hub (depends-on anchor)
• [[wiki/_hubs/systems/webhook-architecture]] — webhook handler services subset
• [[workspace/systemd/]] — unit files for services not yet in /etc/systemd/user/
• [[workspace/scripts/stack-inventory.js]] — daily stack snapshot collector

Active services by tier

Tier A — Critical path (always-running)

Unit	Process Manager	Port	Status	Description
openclaw-gateway.service	systemd user	18789/18791	✅ active	OpenClaw Gateway v2026.4.15 (Go binary)
portkey-proxy.service	systemd user	18900	✅ active	Portkey per-agent LLM proxy
openphone-webhook.service	systemd user	18792	✅ active	OpenPhone/Quo webhook handler
hubspot (PM2 id:3)	PM2	18790	✅ online	HubSpot + DocuSign + Make webhook handler
salesmsg (PM2 id:2)	PM2	18793	✅ online	SalesMsg webhook handler
twilio (PM2 id:1)	PM2	18797	✅ online	Twilio voice + SMS handler
aurora-worker.service	systemd user	—	✅ active	Aurora in-Quo orchestrator (dispatch + confirmations)
quo-watcher-c4.service	systemd user	—	✅ active	Aurora Quo WebSocket mention watcher
quo-mention-receiver.service	systemd user	18798	✅ active	Aurora Quo mention trigger

Tier B — Supporting services (running)

Unit	Process Manager	Port	Status	Description
anthropic-max-router.service	systemd user	18903	✅ active	Max Plan OAuth router (teamsteph)
claude-max-api-proxy@teamsteph.service	systemd user	18910	✅ active	Max API proxy (teamsteph, idle)
reri-dispo-page.service	systemd user	18803	✅ active	RERI Dispo Model landing page
imessage-webhook.service	systemd user	18802	✅ active	BlueBubbles iMessage handler
broadcast-audit.service	systemd user	18812	✅ active	Broadcast Response Audit UI
models-guardian.service	systemd user	—	✅ active	models.json guardian (max-plan agents)
slack-thread-manager.service	systemd user	—	✅ active	Slack thread hygiene monitor
openwebui.service	systemd user	3000/Tailscale	✅ active	Open WebUI (Kimi/DeepSeek/Ollama)
ops-dashboard.service	systemd user	—	✅ active	RERI Operations Dashboard (static serve)
lovable-api (PM2 id:0)	PM2	18799	✅ online	Lovable API server
discord-lovable-bridge (PM2 id:7)	PM2	18794	✅ online	Discord-Lovable bridge
discord-reaction (PM2 id:5)	PM2	—	✅ online	Discord reaction handler
inbound-alert-watcher (PM2 id:6)	PM2	—	✅ online	Inbound alert watcher
pgmq-consumer (PM2 id:4)	PM2	—	✅ online	PGMQ consumer (Supabase queue)

Tier C — Timer-triggered one-shots (inactive between runs)

Key scheduled services (triggered by paired .timer units):

Unit	Schedule	Last Run	Status
security-audit-funnel.service	Mon 06:00 PT	2026-04-27 ⚠️	inactive (timer firing Mon 2026-05-04)
master-env-backup.service	Weekly	2026-05-03	inactive (runs fine)
escrow-deadline-reminder.service	Daily 17:00 UTC	2026-05-03	inactive (runs fine)
openclaw-vault-sync.service	Every 15 min	runs frequently	inactive between runs
openclaw-vault-pull.service	Every 5 min	runs frequently	inactive between runs
memory-worker.service	scheduled	recent	inactive between runs
embed-if-text.service	scheduled	recent	inactive between runs
perplexity-daily-summary.service	Daily	FAILED 2026-05-03 16:00 ⚠️	inactive/dead — NEEDS FIX
investorlift-cookie-refresh.service	scheduled	FAILED 2026-05-03 13:00 ⚠️	inactive/dead — NEEDS FIX

Tier D — Business logic one-shots

Unit	Purpose	Status
acquisitions-perplexity-intel.service	Daily Perplexity intel for Acq agent	inactive (timer-triggered)
atlas-perplexity-intel.service	Daily Perplexity intel for Atlas	inactive (timer-triggered)
dispo-perplexity-intel.service	Daily Perplexity intel for Dispo	inactive (timer-triggered)
research-perplexity-intel.service	Competitive intelligence	inactive (timer-triggered)
email-deal-intake.service	Gmail deal intake → Supabase	inactive
hubspot-showing-sync.service	HubSpot showing requests sync (30 min)	inactive
showing-day-confirmations.service	Showing day SMS confirmations (7:30 AM PT)	inactive
showing-followup-trigger.service	Showing follow-up trigger (every 15 min)	inactive
deal-outreach-auto-approver.service	Yellow-zone auto-approver (4h window)	inactive
gmail-email-harvest.service	Gmail email harvest → Supabase	inactive
gmail-inbox-scanner.service	Gmail inbox auto-label	inactive
gmail-label-worker.service	Gmail label worker	inactive
sm-broadcast-sync.service	SalesMsg broadcast sync	inactive
sm-token-refresh-aurora.service	SalesMsg OAuth token refresh	inactive
webhook-health-monitor.service	Webhook health monitoring	inactive
coordinate-plans-heartbeat.service	Active plan heartbeat	inactive
monitoring-alerts.service	OpenClaw monitoring alerts → Discord	inactive
stack-inventory.service	Daily stack inventory snapshot	inactive
audit-weekly-triage.service	Weekly vendor-audit Discord post	inactive
vendor-audit-url-verify.service	Daily URL verification for audit citations	inactive
external-tool-benchmark.service	Daily latency + cost tracking	inactive
founder-mem-checkpoint.service	Founder memory checkpoint	inactive
omni-conversation-linker.service	Link events to conversations	inactive
orphan-approval-sweep.service	Orphan approval catch (Phase 1 Step 17)	inactive
prediction-daily.service	Prediction market daily analytics	inactive
prediction-expiring.service	Prediction market resolution decay scan	inactive
prediction-history.service	Prediction market history snapshot	inactive
prediction-reconcile.service	Prediction market position reconciliation	inactive
prediction-trader.service	Prediction market paper trading	inactive
arbitrage-monitor.service	BetterTrading cross-venue arbitrage monitor	inactive
portfolio-cache-refresh.service	Portfolio snapshots cache refresh	inactive
cost-monitor.service	LLM cost monitor (Max Plan quota)	inactive
openclaw-logrotate.service	Log and temp file cleanup	inactive
openclaw-gateway-restart.service	Nightly gateway restart (memory leak bounds)	inactive
bluebubbles-memory-sync.service	BlueBubbles → Aurora memory sync	inactive
email-signal-stage-updater.service	Auto-update deal stages from TC email events	inactive

Tier E — System scope (root systemd)

Unit	Status	Notes
nginx.service	active	HTTP redirect :80 + nginx upstreams
redis.service	active	Redis cache (:6379 loopback)
cloudflared.service	active	Cloudflare Tunnel daemon (:20241 control)
netdata.service	active	Observability (:19999 loopback)
tailscaled.service	active	Tailscale node daemon
otel-plugin	active	OpenTelemetry collector (:4317 loopback)
tor.service	active ⚠️	Tor SOCKS — disable candidate (F2 audit M3)
cupsd.service	active ⚠️	Printing — disable on headless server (F2 audit M4)
sshd.service	active	SSH (:22, Tailscale-fronted)

Live state snapshot (2026-05-03)

Metric	Documented	Live (probed)	Drift	Status
Total user services (systemctl)	~90	79	−11	within range
PM2 processes	8	8	0	ok
Active/running services	—	14 (systemd user)	—	ok
Failed services	0	0 (currently inactive/dead, not failed state)	—	monitor
investorlift-cookie-refresh	—	inactive/dead (failed last run)	—	⚠️ needs fix
perplexity-daily-summary	—	inactive/dead (failed last run)	—	⚠️ needs fix
security-audit-funnel timer	Mon 06:00 PT	last fired 2026-04-27	6 days stale	⚠️ flagged

How it’s used

• Service health check: systemctl --user status <unit> + journalctl --user -u <unit> --no-pager --since "1h ago"
• Start/restart: systemctl --user restart <unit> (confirm with Henry per action-gate unless session-level authorization)
• Add new service: create unit file → add to port-registry → add to ARCHITECTURE → daemon-reload → start (G-SERVICE-PRE-START-DOC)
• Failed service triage: G-FAILED-SERVICE-MTTR: fix within 24h, explicitly disable, or archive per feedback_archive_not_delete
• Failure mode: undocumented wildcard-public bind or credential hardcoded in unit file → security finding

Cross-links

Agents that touch this

• _summary — aurora-worker, quo-watcher-c4, quo-mention-receiver
• _summary — acquisitions-perplexity-intel
• _summary — dispo-perplexity-intel
• _summary — atlas-perplexity-intel

Skills that invoke this

• acquisitions-outreach — depends on salesmsg + openphone-webhook services
• acquisitions-followup — depends on salesmsg + openphone-webhook
• dispo-blast — depends on salesmsg service
• il-marketplace-pull — investorlift-cookie-refresh dependency (needs fix)

Plans that govern this

• openclaw-fragmentation-fix-2026-05-01 — G-FAILED-SERVICE-MTTR gate (§A1)
• openclaw-obsidian-vault-2026-05-02 — vault-sync/pull timer services
• nemoclaw-audit-2026-05-03 — reserved port :18901 (pending B1-B3 ratification)

Feedback rules

• feedback_service_pre_start_doc — G-SERVICE-PRE-START-DOC
• feedback_failed_service_mttr — G-FAILED-SERVICE-MTTR: 24h resolution window
• feedback_archive_not_delete — retire vs delete for old services
• feedback_no_plaintext_creds — G-NO-PLAINTEXT-CREDS: check unit files for hardcoded keys
• feedback_models_guardian_pattern — models.json guardian prevents gateway overwrite

KB / source docs

• README — VPS config, systemd user scope
• README — cloudflared tunnel service
• README — salesmsg-gateway ⚠️ P0: hardcoded key in unit file

System maps

• ports-topology — visual port/service topology
• agents-tier-structure — agent-to-service mapping

Related cluster — System catalog

• port-registry — port assignments for all services above
• webhook-architecture — webhook service subset (18790/18792/18793/18797/18801)
• mcp-registry — MCP server processes (not systemd — stdio via .mcp.json)
• hetzner — VPS host (systemd user scope runs here)
• aws — EC2 Mac Ultra (investorlift-cookie-refresh must SSH here)

⚠️ Security finding

salesmsg-gateway.service — unit file at /etc/systemd/system/salesmsg-gateway.service contains hardcoded ANTHROPIC_API_KEY=<redacted> in plaintext. P0 — rotation pending per G-NO-PLAINTEXT-CREDS. Replace with op://Aurora/anthropic/api-key reference.

Open issues / TODOs

• P0 — fix investorlift-cookie-refresh.service: reroute to SSH → EC2 Mac Ultra; VPS Playwright always 403
• P0 — fix perplexity-daily-summary.service: add or 0 null-coalescing in daily_summary.py line 57
• P0 — rotate salesmsg-gateway unit file credential (G-NO-PLAINTEXT-CREDS)
• Phase 2.1: build service registry audit cron (complement to port-registry-audit.timer)
• Identify undocumented processes on ports 3001, 3100, 5174, 8080, 8891 — add to service registry
• Disable cupsd and tor on hardening pass
• security-audit-funnel.timer last fired 2026-04-27 — manual run recommended before next Monday

Recent activity

• 2026-05-03: system hub created (W2-S4, Wave 2), live probe executed (79 user services, 8 PM2)
• 2026-05-03 16:00 UTC: perplexity-daily-summary.service failed (TypeError NoneType)
• 2026-05-03 13:00 UTC: investorlift-cookie-refresh.service failed (Playwright 403 from VPS)
• 2026-05-02: openclaw-vault-sync.timer + openclaw-vault-pull.timer added to CLAUDE.md