Field-Test Issues & Production Mitigations¶
Engineering write-up of the SIP / registration / audio issues hit during a live multi-phone field test of UPES-ECS, and the production-grade fixes that stop each one recurring at scale.
Context of the test
| Item | Value |
|---|---|
| System | UPES-ECS — LAN-only campus emergency phone system on Asterisk (PJSIP) |
| Deployment under test | Asterisk in a QEMU VM on a Windows "van laptop", QEMU user-mode NAT (SLIRP) + port-forward |
| PBX reachable IP (this test) | 192.168.1.16 : UDP 5060, forwarded RTP 10000–10019 |
| Endpoints | Real Android phones running Linphone on campus Wi-Fi |
| Testers (real roster) | Rohan Batra 500120597, Student Example Two 500000002, Student Example Three 500000003, Student Example Four 500000004; staff e.g. Staff Member One 40000001 |
| Server codecs / media | G.711 only (ulaw/alaw), plain RTP (no SRTP configured) |
How to read this doc. Each issue is written as Symptom → Root cause → Immediate fix (what we did in the field) → Production mitigation (how to prevent it entirely at scale). The build-time roadblocks that pre-date this test live in Roadblocks-and-Solutions.md; this doc is the field-test companion and cross-references it where the same root cause recurs.
Related documents: SOP 05 – Student SIP Setup Guide · SOP 14 – Device Provisioning Sheet · SOP 24 – Mobile App Reliability & Battery · QEMU deployment README · config/extensions_custom.conf · deploy/asterisk/pjsip.conf · Roadblocks-and-Solutions.md
Issue 1 — Registration hangs on "Operation in progress" (wrong transport)¶
Symptom. A phone opened Linphone, entered SAP ID + password + domain, and the account sat forever on "Operation in progress…" and never went green/Registered. No error, no timeout, just a hang. On the server there was zero trace of the phone.
Root cause. Linphone defaulted the account Transport to TLS (or TCP). The
UPES-ECS test server is UDP-only (transport-udp, bind=0.0.0.0:5060, protocol
udp — see deploy/asterisk/pjsip.conf). The phone's
REGISTER was sent over a transport the server never listens on, so it got no reply
and the client just kept "in progress." Because nothing UDP left the phone for 5060, not
a single SIP packet reached the PBX — from the server's point of view the phone did not
exist.
Immediate fix (field). Forced the account transport to UDP:
- In Linphone: Account → Transport = UDP, or
- Encode it in the domain/proxy:
192.168.1.16;transport=udp
The phone registered within a second and went green.
Production mitigation.
- Provision, don't ask users to pick. Ship a pre-built Linphone profile / QR /
.rcconfig file (see Section A) so the transport is baked in and no user ever chooses TLS/TCP by hand. This is the single highest-value fix — it removes the entire class of "wrong transport" errors. - Or stand up TLS + SRTP properly and standardize on it end-to-end (server cert,
transport-tls, matching endpoints, phones provisioned for TLS). Do this only if you commit to it fully — a half-configured TLS server is exactly what produced this hang. - Document the exact Linphone settings (table in Section A) and treat "Transport = UDP" as a hard field, not "unless IT tells you otherwise."
- Validate reachability first. Step 1 of onboarding is
ping 192.168.1.16(or the site PBX IP) from the phone's network tools — prove L3 reachability before debugging SIP. See the runbook.
Related: SOP 05 – Student SIP Setup Guide (the Transport line there must read UDP, not "unless IT tells you otherwise").
Issue 2 — +91 country code auto-prepended to SAP IDs¶
Symptom. A tester dialed another student's SAP ID (e.g. 500120597) and the call
failed / hit no such extension, even though both phones were registered. The call log
showed the dialed number as +91500120597.
Root cause. Linphone's international prefix / dial-plan (dial assistant) feature
was on. It treats dialed strings as phone numbers and auto-prepends the country code
+91. UPES-ECS extensions are SAP IDs, not phone numbers — +91500120597 matches no
extension pattern (_5XXXXXXXX expects exactly the 9 digits), so the dialplan rejected it.
Immediate fix (field). Two-sided fix:
- Server — added a strip rule to
ctx_studentso a mis-dialed+91…is normalized back to the bare extension (now committed in config/extensions_custom.conf):
; strip the +91 country code some softphones auto-prepend, then re-enter
exten => _+91X.,1,Goto(${EXTEN:3},1)
exten => _5XXXXXXXX,1,Dial(PJSIP/${EXTEN},30) ; student-to-student
same => n,Hangup()
${EXTEN:3} drops the 3-character +91 prefix and re-enters the dialplan at the real
SAP ID.
- Client — disabled the country / international prefix in the phone's Linphone
settings so it stops mangling the dialed digits in the first place.
Production mitigation.
- Ship the strip rule by default. The
_+91X.normalizer is now in the reference dialplan and belongs in every deployment'sctx_student(and any context students dial from) as a safety net. - Disable the international prefix in the provisioned profile so the client never
prepends
+91— belt and suspenders with rule 1. - Consider an explicit numbering-vs-dialing scheme. SAP IDs are identifiers, not E.164 numbers; the provisioning profile should mark the account as "not a phone number" (no dial assistant, no prefix) so no OS/app "helpfully" reformats it.
Issue 3 — "Connected but no voice" on a phone-to-phone call (NAT media)¶
Symptom. Two registered phones placed a call. Signalling worked — it rang, both sides showed Connected — but neither side heard the other (dead air, both ways).
Root cause — QEMU SLIRP NAT + direct media. The PBX runs behind QEMU user-mode networking (SLIRP), a user-mode NAT. Asterisk, by default, tries direct media: once the call is up it sends a re-INVITE telling the two phones to stream RTP directly to each other, taking the PBX out of the media path. Behind SLIRP that fails:
- Asterisk (and the phones) advertise addresses in SDP that the NAT rewrites or that are simply not routable between the two clients as the PBX described them.
- SLIRP hides real client IPs from the guest, so the addresses handed to each phone for "send your audio here" are wrong. Signalling (which the PBX relays) is fine; media (which it told the phones to send peer-to-peer) goes into a black hole.
This is the same class of problem as build-time roadblock 1.6 (SLIRP hides real client IPs) in Roadblocks-and-Solutions.md and 1.1 (container NAT mangles RTP).
Immediate fix (field). Force the PBX to relay the RTP (stay in the media path) and tell it its real reachable address. On the endpoint template in deploy/asterisk/pjsip.conf:
[endpoint-tpl](!)
type=endpoint
disallow=all
allow=ulaw
allow=alaw
rewrite_contact=yes
rtp_symmetric=yes ; return media to where it actually came from (Wi-Fi/NAT)
force_rport=yes ; reply to the source port, not the SDP-claimed port
direct_media=no ; <-- RELAY RTP through the PBX; do NOT re-INVITE peers together
[transport-udp]
; the PBX's reachable LAN IP so Asterisk advertises the right address to phones
external_media_address=192.168.1.16
external_signaling_address=192.168.1.16
Plus a fixed, forwarded RTP range 10000–10019 (rtp.conf) so the media ports are the
ones the QEMU port-forward + firewall actually open. With direct_media=no the two phones
each talk RTP only to the PBX, and the PBX bridges — no peer-to-peer path through SLIRP
is ever attempted. Audio came up both ways immediately.
Production mitigation.
- Run the production PBX natively / bridged with a real LAN IP. The whole media
problem is an artifact of SLIRP user-mode NAT. Deploy Asterisk on the van's own
Linux box (or the VM on bridged networking with a TAP adapter) so it pulls its
own DHCP LAN address and sits on the LAN as a first-class host. That removes SLIRP
entirely and with it this entire class of NAT-media failures — no
external_*_address, no port-forward, no fixed-range juggling. This is already flagged as the "truly zero-touch" target in the QEMU deployment README. - On native/bridged,
direct_mediacan even be re-enabled (phones on the same flat LAN can legitimately talk peer-to-peer), reducing PBX CPU/bandwidth — but keepdirect_media=nounless you have a specific reason and have tested it, because relaying is the robust default across Wi-Fi client-isolation and NAT. - Standardize
Media encryption = None. The server offers plain RTP. If a phone insists on SRTP/ZRTP and the server does not, media negotiation breaks (see Issue 5). Provision encryption = None everywhere unless SRTP is configured end-to-end.
Issue 4 — One phone "couldn't register" while another worked (per-phone config)¶
Symptom. On the same Wi-Fi, at the same time, one tester's phone registered fine while another's would not register — despite identical network conditions.
Root cause. Because ping to the failing phone worked and other phones on the same subnet registered, the network was ruled out. The fault was per-phone config: a wrong transport (Issue 1), a typo in SAP ID / password / domain, or a stale international-prefix setting (Issue 2) — an individual-device problem, not an infrastructure one.
Immediate fix (field).
- Confirmed L3 reachability:
ping 192.168.1.16from the phone and ping the phone from the laptop — both succeeded, so the LAN was fine. - Verified the on-file password for that SAP ID against the roster.
- Forced Transport = UDP and corrected the typo → registered.
Production mitigation.
- Profile / QR provisioning removes per-phone manual error. If SAP ID, password, domain, and transport come from a scanned config, there is nothing to typo. See Section A and SOP 14 – Device Provisioning Sheet.
- A reachability + registration self-test is the onboarding gate. Before a device is
marked Active, it must pass:
ping PBX→ Registered (green) → dial 198 (echo) → dial 199 (drill) → dial 111 (or a directory SAP ID). No green + no echo = device is not provisioned, regardless of what the user says.
Issue 5 — Media-encryption / codec mismatch risk¶
Symptom (risk class). A call connects (signalling completes, both sides say Connected) but there is no audio or one-way audio — not because of NAT (Issue 3) but because the two ends could not agree on how to encode/encrypt the media.
Root cause. UPES-ECS servers allow only G.711 (disallow=all; allow=ulaw;
allow=alaw) and offer plain RTP (no SRTP). If a phone is configured to offer only:
- a codec the server does not allow (e.g. Opus/G.729-only), or
- mandatory SRTP/ZRTP while the server speaks plain RTP,
then SDP negotiation yields no common media → the call sets up on signalling but the audio stream never establishes (or one leg does and the other doesn't).
Immediate fix (field). Not hit as a hard failure in this test (Linphone offered G.711 + plain RTP by default once transport/NAT were fixed), but called out because it is the next failure mode if a phone's media settings drift.
Production mitigation.
- Standardize codecs in the provisioned profile: G.711 (
ulaw/alaw) only — match the server exactly. Leaving Opus enabled is harmless if G.711 is also offered, but the safe default is to pin G.711. - Standardize
Media encryption = Noneto match the plain-RTP server — or configure SRTP end-to-end and provision SRTP on every phone. Never leave it mixed. - Document both in the settings table (Section A) so a re-imaged phone is rebuilt identically.
Issue 6 — Battery / background suspension silently deregisters the app¶
Symptom. A phone that registered fine stopped ringing for incoming calls after sitting idle / locked for a while. Outbound (open app, dial 111) still worked; incoming ERT callbacks never rang.
Root cause. Android battery optimization / background app suspension froze or killed the SIP app, which dropped its SIP registration. With no live registration the PBX has no contact to ring — so inbound calls (exactly the ERT-callback path) silently fail. This is Risk R2 and the whole reason SOP 24 exists.
Immediate fix (field). Applied the per-OS battery/background exceptions (Unrestricted, allow background, allow auto-start, don't swipe-kill) and re-registered the app.
Production mitigation.
- Follow the per-OS steps in SOP 24 – Mobile App Reliability & Battery — summarized in Section B.
- Dedicated always-on-charger answer points. Critical inbound roles (ERT, control room, medical, security) run on fixed dedicated Android devices on the charger, not personal phones — treat them like appliances (SOP 24 §6, and SOP 14 §3).
- Tune keepalive / register expiry server-side so drops are detected fast and the registration path stays warm:
| PJSIP setting | Recommended | Why |
|---|---|---|
qualify_frequency |
~30–60 s | Keepalive OPTIONS keep the NAT path open + detect a dead endpoint quickly |
| Registration expiry | ~120–300 s | Short enough to notice a drop, long enough to avoid re-REGISTER thrash |
rewrite_contact |
yes |
Replies reach the phone's actual source address on Wi-Fi/NAT |
rtp_symmetric / force_rport |
yes |
Media/replies return to where packets really came from |
A. Production phone provisioning (do it once, right)¶
Every config issue above (1, 2, 4, 5 — and the setting half of 6) collapses to the same root cause: a human typed or picked a setting by hand. The production answer is to provision the phone from a controlled artifact so the user never touches transport, prefix, codec, or encryption.
A.1 Ship a pre-built account artifact¶
Deliver one of these instead of a written setup sheet:
| Method | What it is | Best for |
|---|---|---|
| QR / config-URI | Linphone opens linphone-config://… (or scanned QR) that imports the account |
Fast per-user onboarding at a desk |
.rc / provisioning file |
A Linphone remote-provisioning .rc fetched at first launch (account + transport + codecs + encryption baked in) |
Fleet / bulk rollout |
| Pre-configured device image | Dedicated fixed-role Androids imaged identically | ERT / control-room appliances |
The artifact must set: Domain = site PBX IP (192.168.1.16 in this test),
Transport = UDP, Media encryption = None, codecs = G.711 (ulaw/alaw),
international/dial-assistant prefix OFF. Credentials (SAP ID + one-time secret) come
from SOP 14; secrets are per-account, never a
shared password.
A.2 Onboarding self-test (the gate to "Active")¶
A device is not Active until it passes, in order:
1. ping <PBX IP> -> L3 reachable (e.g. ping 192.168.1.16)
2. app shows Registered -> green / Connected
3. dial 198 (echo) -> hear your own voice => mic + speaker + media path OK
4. dial 199 (drill) -> drill prompt => dialplan reachable, NO real dispatch
5. dial 111 or a SAP ID -> reaches ERT / peer => two-way audio confirmed
Record the result per device model (some Android brands need the SOP 24 battery steps before step 2 will stay green). This mirrors SOP 05 Step 4.
A.3 The exact correct Linphone settings¶
| Setting | Correct value | Why (which issue it prevents) |
|---|---|---|
| Username | your SAP ID (e.g. 500120597) |
Identity = extension (SOP 14) |
| Password | your one-time UPES-ECS secret | Per-account, never shared |
| Domain / SIP proxy | site PBX IP — 192.168.1.16 in this test (pbx.upes.lan where DNS exists) |
Reachability |
| Transport | UDP | Issue 1 — server is UDP-only |
| Media encryption | None | Issue 3 / 5 — server is plain RTP |
| Audio codecs | G.711 only — ulaw (PCMU) + alaw (PCMA) |
Issue 5 — server allows only G.711 |
| International / country prefix (dial assistant) | OFF / disabled | Issue 2 — no +91 prepend |
| Register expiration | ~120–300 s | Issue 6 — drop detection vs. thrash |
| Background / battery | app Unrestricted, auto-start on | Issue 6 / SOP 24 |
One-line domain form that pins the transport regardless of the UI:
500120597@192.168.1.16;transport=udp
B. Keep the phone registered (battery & background)¶
Full detail lives in SOP 24 – Mobile App Reliability & Battery (Risk R2). This is a pointer + the essentials — do not treat this as the source of truth; SOP 24 is.
The 7 essential rules (SOP 24 §1):
- Allow background running for the SIP app.
- Turn OFF battery optimization — set the app to Unrestricted.
- Allow auto-start / start-on-boot.
- Don't swipe-kill the app from recents.
- Allow microphone + notifications permissions.
- Stay on campus Wi-Fi (LAN-only by design).
- Keep the phone charged — dead phone = no emergency line.
Per-brand pointers (SOP 24 §2 has the full steps):
| Brand | Key setting |
|---|---|
| Samsung (One UI) | Add Linphone to Never sleeping apps; ensure it is not in Sleeping/Deep-sleeping apps |
| Xiaomi / Redmi / POCO | Autostart ON + battery No restrictions; lock the app in recents |
| Oppo / Realme / OnePlus / Vivo / iQOO | Allow auto-launch / auto-start + allow background activity; disable sleep-standby optimization |
Inbound is never 100% guaranteed on a personal phone. Critical answer points (ERT, control room, medical, security) run on dedicated always-on-charger devices — SOP 24 §6 and SOP 14 §3. See SOP 24 for iOS guidance and the per-phone pilot checklist.
Field troubleshooting runbook¶
Fast triage for the live-test symptoms above. Work top-to-bottom; most faults are transport, NAT-media, or battery.
| Symptom | Quick check | Fix |
|---|---|---|
| Won't register ("Operation in progress" / not green) | ping <PBX IP> from the phone; is Transport = UDP?; SAP ID + password + domain correct? |
Force Transport = UDP (…@192.168.1.16;transport=udp); correct typos; confirm campus Wi-Fi. Issue 1 / 4 |
| Registers then can't place calls (no such extension) | Look at the dialed number in the log — is it +91…? |
Disable Linphone international prefix; server strip rule _+91X. → ${EXTEN:3} (already in dialplan). Issue 2 |
| Connected but no voice (both ways dead) | Is the PBX behind QEMU SLIRP NAT? Is direct_media off? |
Set direct_media=no, external_media_address=<PBX IP>, rtp_symmetric=yes, force_rport=yes, fixed RTP 10000–10019. Long-term: native/bridged LAN IP. Issue 3 |
| One-way audio | Codec/encryption offer vs. server (G.711 + plain RTP)? Wi-Fi client-isolation? | Provision encryption = None, G.711 only; confirm rtp_symmetric=yes; check AP client-isolation. Issue 5 / 3 |
| Drops / won't ring when phone locked | App still Registered after idle? Battery optimization on? | Apply SOP 24 battery steps (Unrestricted, auto-start, don't swipe-kill); use a fixed charger device for critical roles. Issue 6 |
+91 prepended to a SAP ID |
Dialed string shows +91 in front |
Turn OFF the country / dial-assistant prefix in Linphone; server _+91X. strip is the safety net. Issue 2 |
This log covers the SIP/registration/audio issues that actually occurred during the live multi-phone field test, with production mitigations. It is the field-test companion to Roadblocks-and-Solutions.md and is maintained alongside the config it references — deploy/asterisk/pjsip.conf, config/extensions_custom.conf.