UPES-ECS — Production Readiness & Go-Live¶
Last updated: 2026-07-05
This is the authoritative "is it ready for a real deployment?" checklist. It records what is hardened and verified, what is automated into the build, and the few decisions only you (the site owner) can make before go-live.
1. What is production-ready now (verified)¶
| Area | State | Evidence |
|---|---|---|
| Emergency line 111 | Answers, records, queues, escalates, coaches, voicemails | dialplan loads clean; call-flow tested |
| 102 offline panic-coach | Live — deterministic first-aid, zero internet | 15 prompts, all branches resolve |
| Press-1 fast-path + parallel coaching | Live | queue breakout + background responder alert |
| Answer-point staffing | On-shift via *22/*23 + ert-shift.sh, logged |
shift log verified |
| Crash auto-recovery | systemd Restart=always — asterisk restarts in ~3s if it dies |
killed -9, self-healed |
| Auto-start on boot | VM + Console launch at Windows logon (no admin) | Startup-folder launchers installed |
| Nightly backups | 02:00 cron → /var/backups/upes-ecs/*.tgz, keep 14 |
backup ran, archive present |
| Health + retention | 5-min healthcheck, 03:30 retention cleanup | cron.d/upes-ecs |
| Security | fail2ban active (asterisk + sshd jails); no anonymous/guest SIP; key-only SSH | audited |
| Secrets | Real per-user/position SIP secrets; real paging PIN; not web-served | secrets/, 404 verified |
| Accounts | 19 endpoints (7 people + 12 positions) load | pjsip show endpoints |
| Dynamic-across-routers | Console server-IP auto-updates as the van moves | status.json.serverIp |
All of the above is persisted into the build (deploy/qemu/build-vm.ps1 + seed/setup-in-vm.sh),
so a fresh build-vm.ps1 reproduces a hardened server — configs, accounts, scripts, prompts,
crash-recovery, crons, and a freshly-generated paging PIN.
2. Go-live checklist¶
Done (automated/verified):
- [x] Dialplan + accounts load with no errors
- [x] 111 → queue → escalation/alert → offline coach → voicemail path proven
- [x] Crash auto-recovery (systemd) + boot auto-start (VM + Console)
- [x] Nightly backup + retention + health crons
- [x] fail2ban, no guest SIP, real secrets, real paging PIN
- [x] Every team/person has a real credential (secrets/TEAM-CREDENTIALS.md)
Before you flip it on at a real site (your action):
- [ ] Deliver credentials to each person/desk over a secure channel; register their handset
- [ ] Assign people to teams/positions (Medical/Security/Warden/Ops/IT) — table in secrets/TEAM-CREDENTIALS.md
- [ ] Staff the ERT queue for each shift (*22) so 111 is answered (wallboard shows READY)
- [ ] Set the real escalation roster — UPES_LEAD / UPES_BACKUP globals to the actual Lead/backup extensions
- [ ] Rotate the paging PIN and re-deliver (PAGING_PIN_700)
- [ ] Battery/Wi-Fi: disable battery optimisation + background limits on every responder handset (see field guide)
- [ ] Physical drill: run a 199 drill, then a supervised 111 end-to-end, per SOP 03/17
- [ ] Backups off-box: copy /var/backups/upes-ecs/ to external media on a schedule
3. Decisions that change the build (need site input)¶
| Decision | Default today | Production option |
|---|---|---|
| SIP transport | UDP, unencrypted (fine on an isolated LAN) | TLS + SRTP if the LAN isn't trusted — needs certs on PBX + phones |
| Handsets | Android softphones (Linphone) | Dedicated IP desk phones per position (BLF, speed-dial) |
| 101 online AI | Not deployed (design only) | Local-first stack (Ollama/llama.cpp + Whisper + Piper) on a dedicated GPU host — the TCG van VM can't run it in real time (see AI-101/) |
| Paging speakers | None (ALLCAMPUS_SPEAKERS placeholder) |
Real IP speakers wired to 700-series zones |
| Second PBX / HA | Single node | Standby PBX + failover if a single van node isn't enough |
live_dangerously=yes |
On (required for incident-ID SHELL() + System() alerts) |
Accepted, contained: the dialplan is version-controlled and not user-editable |
4. Operations quick-reference¶
Start VM ............ qemu\start-vm.ps1 (auto at logon)
Stop VM ............. qemu\stop-vm.ps1
Console ............. Console\Serve.ps1 (auto at logon; http://localhost:8080)
Set van LAN IP ...... qemu\Set-UpesLanIp.ps1
Auto-start on/off ... deploy\qemu\Register-Autostart.ps1 [-Remove]
On the PBX (ssh -p 2222):
Staff a position .. /opt/upes-ecs/ert-shift.sh on|off|status <ext>
Backup now ........ sudo /opt/upes-ecs/upes-ecs-backup.sh
Restore ........... stop asterisk; tar xzf <backup>.tgz -C / ; start asterisk
Health ............ cat /var/lib/upes-ecs/health.txt
Recover asterisk .. automatic (systemd Restart=always); manual: systemctl restart asterisk
Full call-flow: ../Blueprint/03-Call-Flows.md · Runbook: ../deploy/qemu/README.md · AI layer: ../AI-101/README.md