Roadblocks & Solutions¶
The engineering log for UPES-ECS: every roadblock we actually hit building a LAN-only Asterisk emergency phone system as a QEMU VM on a no-admin Windows laptop, and how we got past it. Grouped by area. Each entry: Symptom → Cause → Fix, plus a Lesson where it earns one.
Key files referenced throughout: config/extensions_custom.conf, deploy/qemu/README.md, deploy/README.md, SOP 09 – Dialplan Design.
1. Virtualization & networking on Windows¶
1.1 Docker Desktop can't carry SIP/RTP media¶
- Symptom: Calls signalled fine but audio (RTP) was unreliable/one-way when the PBX ran under Docker Desktop.
- Cause: Docker Desktop on Windows sits behind a user-mode NAT layer that rewrites and blocks UDP media. SIP/RTP is exactly the traffic that NAT mangles — the SDP advertises addresses the NAT then breaks.
- Fix: Moved the live-audio test off Docker. Docker is kept only for fast dialplan/config validation, not for anything that has to move real media. See deploy/README.md.
- Lesson: Container NAT is fine for signalling smoke-tests, wrong for RTP. Prove audio on a network path you actually control.
1.2 No admin rights on the Windows host¶
- Symptom: Could not install QEMU via
choco/winget, could not add a Windows Firewall rule, could not enable the Windows Hypervisor Platform (WHPX/Hyper-V) — every path demanded elevation we didn't have. - Cause: The van laptop is a managed, non-admin machine.
- Fix: Installed QEMU portably — downloaded the official weilnetz Windows build and extracted it with 7-Zip, no installer, no admin. The one-time firewall rule is documented as an elevated command to run when an admin is available; acceleration falls back to software (see 1.3). See deploy/qemu/README.md.
- Lesson: "No admin" is a design constraint, not a blocker — portable binaries + software emulation get you a working PBX with zero privileges.
1.3 QEMU hardware acceleration (WHPX) unavailable¶
- Symptom: QEMU exited instantly.
-accel whpxfailed:
-accel tcg. Slower first boot, but works
with zero privileges.
- Lesson: TCG is the universal fallback. Accept the slow boot; the VM only boots once
per deployment.
1.4 Debian 12 dropped the asterisk package¶
- Symptom: VM provisioning failed:
asterisk package in the base repos.
- Fix: Switched the VM base image to Ubuntu 22.04, where asterisk is in the
universe repo. Bonus: it matches the WSL distro we test on, so behaviour is
consistent. See deploy/qemu/README.md.
- Lesson: Pin the base image to one that actually packages your core dependency —
verify apt-cache policy before committing to a distro.
1.5 Building a cloud-init seed ISO on Windows with no mkisofs¶
- Symptom: cloud-init needs a seed ISO, but Windows has no
mkisofs/genisoimageand we couldn't install one. - Cause: Standard ISO tooling is Linux-only; installing it needs admin.
- Fix: Built ISO9660+Joliet images (
CIDATAandUPESDATAvolume labels) using the native Windows IMAPI2 COM API from PowerShell, driving it with an inline C#ISOFilewriter. No external tools, no admin. See deploy/qemu/build-vm.ps1 and deploy/qemu/README.md. - Lesson: Windows already ships an ISO burner (IMAPI2) — reach for the OS API before assuming you need to install a Unix utility.
1.6 Hosting a SIP server behind QEMU user-mode NAT (SLIRP)¶
- Symptom: With QEMU's default user-mode networking, the guest PBX advertised its
private
10.0.2.15address in SIP/SDP, so external LAN phones couldn't get media back. - Cause: QEMU SLIRP user-mode NAT hides real client IPs from the guest and the guest has no idea what the laptop's LAN address is.
- Fix: Configured Asterisk to run behind NAT:
external_media_address/external_signaling_address= the laptop's real LAN IPrtp_symmetric = yes,force_rport = yes- a small fixed RTP range 10000–10019, forwarded on all interfaces
Proven: a softphone registering to the laptop's real LAN IP (192.168.1.16)
registered and exchanged live RTP. See
deploy/asterisk/pjsip.conf,
deploy/asterisk/rtp.conf.
- Lesson: A NAT'd PBX must be told its own public-facing address and a fixed,
small RTP range you can forward — never let it advertise the private guest IP.
1.7 "Dynamic across routers" — the LAN IP moves with the laptop¶
- Symptom: Every time the van laptop joins a new network, its LAN IP changes, and the behind-NAT config from 1.6 points at the old address.
- Cause: The advertised LAN IP is the one piece of config that is environment-dependent.
- Fix: deploy/qemu/Set-UpesLanIp.ps1 auto-detects the current LAN IP and rebinds Asterisk on every boot (and on demand). Documented that bridged networking / native Linux is the zero-touch production alternative.
- Lesson: Isolate the one value that changes per-site and automate it; don't hand-edit config on every move.
1.8 Windows Firewall blocks inbound by default¶
- Symptom: External LAN phones couldn't reach the PBX at all until a firewall rule was added — and adding one needs admin.
- Cause: Windows Firewall blocks inbound connections by default; SIP (UDP 5060) and the RTP range are dropped before they reach QEMU.
- Fix: Documented a one-time elevated command to open the ports, to run when an admin is available:
New-NetFirewallRule -DisplayName "UPES-ECS SIP/RTP" -Direction Inbound `
-Protocol UDP -LocalPort 5060,10000-10019 -Action Allow
See deploy/qemu/README.md. - Lesson: Inbound reachability is a hard prerequisite — call it out as a one-time admin step in the runbook so it isn't discovered during a live drill.
2. Asterisk / SIP / audio¶
2.1 WSL2 headless softphone has no working audio source¶
- Symptom: Trying to prove audio with a headless
baresipsoftphone failed two ways:
ausine module only
emits 48 kHz; aufile had no working source. WSL2/QEMU are headless — there is no mic
hardware for a softphone to open.
- Fix: Proved the media path from the server side instead of the softphone's mic:
- pjsip show channelstats RTP counters (~48 packets/s ≈ the G.711 20 ms frame rate)
- the server-side MixMonitor recording as the audio artefact
See deploy/wsl-rtp-proof.sh, deploy/wsl-call-test.sh. - Lesson: On headless VMs, don't fight for a fake mic — measure RTP at the PBX and let the recording be your proof of audio.
2.2 MixMonitor ,b flag silently dropped unanswered recordings (real defect we found)¶
- Symptom: An unanswered 111 call produced a 44-byte empty WAV — a WAV header and no audio.
- Cause: The dialplan recorded with
MixMonitor(${FILE},b). Thebflag records only while the call is BRIDGED, so anything before answer (ringing, queue, hold) and the whole voicemail leg were never captured. This directly contradicts Feature 4, which requires recording hold + voicemail too. - Fix: Changed to
MixMonitor(${FILE})— records the whole call. Verified: the same scenario went from 44 bytes → ~185 KB / 9.98 s of real audio. Fixed in both config/extensions_custom.conf and SOP 09 – Dialplan Design; see also Feature 4.
; before (bug): only records the bridged segment
same => n,MixMonitor(${MIXMONITOR_FILENAME},b)
; after (fix): records the whole call incl. hold + voicemail
same => n,MixMonitor(${MIXMONITOR_FILENAME})
2.3 func_shell SHELL() disabled by default¶
- Symptom: Incident-ID generation returned nothing —
SHELL()produced no output. - Cause: Asterisk gates
SHELL()(and other risky functions) behindlive_dangerously, which is off by default. - Fix: Enabled
live_dangerously = yesinasterisk.conf(applied by the VM setup script, deploy/qemu/seed/setup-in-vm.sh). Incident IDs in the formERT-YYYYMMDD-NNNNthen generated correctly via scripts/incident_id.sh. - Lesson: If a dialplan function silently does nothing, check whether Asterisk has
gated it behind
live_dangerouslybefore debugging your script.
3. PowerShell & tooling gotchas¶
3.1 Auto-mode classifier blocked piping a remote script into Invoke-Expression¶
- Symptom: The plan to bootstrap via
irm get.scoop.sh | iexwas blocked. - Cause: Two problems: piping an unseen remote script straight into execution is arbitrary remote code, and Scoop wasn't the tool the user asked for anyway.
- Fix: Downloaded the QEMU binary directly — the tool actually named — instead of bootstrapping a package manager we didn't want.
- Lesson:
curl | iex/irm | iexis a code-execution smell. Fetch the exact artefact you need and verify it, rather than running someone else's installer blind.
3.2 $pid is a read-only automatic variable¶
- Symptom:
stop-vm.ps1errored on assignment, and the fallback logic then tried to kill the wrong PID — nearly the shell itself. - Cause:
$pidis a read-only automatic variable in PowerShell (the current process ID). Assigning to it fails, leaving the variable holding the shell's own PID. - Fix: Renamed the variable to
$vmpid. See deploy/qemu/stop-vm.ps1. - Lesson: Never reuse PowerShell automatic-variable names (
$pid,$input,$host,$error, …). A shadowed automatic can turn a bug into a process-kill.
3.3 PowerShell 5.1 mangles embedded double-quotes when calling ssh.exe¶
- Symptom: The remote host received
asterisk -rx corewith the inner quotes stripped:
ssh.exe, so asterisk -rx "core show ..."
loses its quoting.
- Fix: Base64-encode the remote script and run it quote-free on the far side:
No quotes have to survive the PowerShell → ssh → shell trip. - Lesson: When PS 5.1 quoting fights you, stop escaping — encode the payload so there's nothing left to mangle.
3.4 Git Bash (MSYS) path conversion mangles Linux paths¶
- Symptom: Linux paths handed to
docker/wslwere rewritten, e.g./opt/...becameC:/Program Files/Git/opt/.... - Cause: MSYS auto-converts anything that looks like a Unix path into a Windows path before passing it to the child process.
- Fix: Prefixed the affected commands with
MSYS_NO_PATHCONV=1. - Lesson: In Git Bash, guard container/WSL commands that carry Linux paths with
MSYS_NO_PATHCONV=1.
3.5 UTF-8-without-BOM .ps1 files break the PS 5.1 parser¶
- Symptom: Scripts failed to parse at lines unrelated to the real problem, e.g.:
.ps1 files were saved UTF-8 without BOM and contained em-dashes.
PowerShell 5.1 assumes ANSI without a BOM, misreads the multi-byte characters, and the
parser then derails on a later, innocent line.
- Fix: Re-saved every .ps1 as UTF-8 with BOM, and added a parse-check step so a
bad encoding is caught before the script is trusted.
- Lesson: For PS 5.1, always save scripts as UTF-8 with BOM — and distrust a parse
error whose location makes no sense; suspect the encoding first.
4. Design corrections¶
4.1 SAP IDs come in two formats — 9-digit student and 8-digit staff¶
- Symptom: A dialplan that only matched the 9-digit student pattern would have silently broken all staff calling.
- Cause: UPES issues two real ID formats: 9-digit student SAP IDs (
5xxxxxxxx, e.g.500120597) and 8-digit staff/employee IDs (4xxxxxxx, e.g.40000001). An early pattern assumed one length. - Fix: The dialplan matches both (
_5XXXXXXXXfor students,_4XXXXXXXfor staff). Confirmed the 4-digit fixed-device range (4xxx) does not collide — it's a different length. See config/extensions_custom.conf and SOP 09 – Dialplan Design. - Lesson: Enumerate the real identifier formats before writing match patterns; one assumed length can exclude an entire user population.
4.2 Generalising ERT-only "positions" into ctx_responder¶
- Symptom: The early model treated only ERT roles as staffed "positions," leaving Medical / Security / Warden / Ops / IT modelled inconsistently.
- Cause: Those departments are also generic positions staffed by shift, not people — but they are dispatch targets, not 111-queue answerers.
- Fix: Introduced a dedicated
ctx_respondercontext for Medical/Security/Warden/ Ops/IT: they receive handoffs and join coordination rooms, but do not answer the ERT queue, page all-campus, or control the queue. See config/extensions_custom.conf, provisioning/responder-positions.csv, and SOP 30 – ERT Roles & Shifts. - Lesson: Distinguish "answers the emergency queue" from "gets dispatched to" — they need different contexts and different permissions.
5. Data integrity¶
5.1 Fabricated placeholder data had crept into CSVs and docs¶
- Symptom: Invented people and a wrong SAP ID (
500123456) had appeared in CSVs and documentation. - Cause: Placeholder/example data written before the real roster was confirmed, never cleaned out.
- Fix: Removed all of it. The system now uses only the confirmed roster in
../Notes/Confirmed Details.md (the real SAP ID example
is
500120597). Secrets ship as__SET_ON_IMPORT__, are generated withopenssl, and are never committed. See provisioning/pilot-users.csv and provisioning/README.md. The stale500123456in the docs was corrected — logged in Doc-Fixes.md. - Lesson: Placeholder identities are a data-integrity risk. Keep a single source of truth for the roster and purge invented values before anything ships.
5.2 Status monitoring reported wrong numbers (disk % and registrations)¶
- Symptom: The health/status output showed an empty disk % and a null registration count.
- Cause: Two distinct parsing traps:
dfon the recordings path was permission-denied to the non-root status user because the parent dir is750, so the query returned nothing.pjsip show contactsprintsNo objects found— notObjects found: 0— when empty, so the count parsed asnull.- Fix: Query the root filesystem for disk usage instead of the locked-down
recordings path, and count
Contact:lines directly rather than trusting the summary string. See scripts/upes-ecs-healthcheck.sh. - Lesson: A monitor that silently reports empty/null is worse than one that errors — validate parsers against the empty and permission-denied cases, not just the happy one.
This log covers the roadblocks that actually occurred during build and test. It is maintained alongside the code and config it references.