// problem

Users on a private VPN service reported that Telegram was noticeably laggy on phones over LTE, while desktop and Wi-Fi felt fine. Symptoms didn't match a typical "bad node" story — exit nodes were healthy, Telegram-owned domains resolved and responded in 100–300ms from every exit.

// diagnosis

Captured packets on a real phone, not a synthetic test. The failure was not in the VPN path — it was Happy Eyeballs. Carriers handed dual-stack, the client opened the IPv6 socket through the tunnel, 6 of 8 exits had no IPv6, and the kernel reject + xray buffering added 200–500ms before fallback. Telegram opens dozens of concurrent sockets; the delay surfaced as "laggy UI".

// solution

Two-layer fix. Server-side: instant ip6tables reject so clients fall back immediately. Proper fix in Xray routing.rules, applied across all 6 exit profiles and both RU bridges:

Rollout strictly one node at a time with verify-after-each (sub-3s xray reload, client auto-reconnect).

// result

Mobile Telegram latency restored to feel parity with desktop on every tested LTE carrier. ~15 minutes per node including verification. No client-side changes required.

// problem

Live VPN service had 6 subscription templates (MIHOMO, CLASH, XRAY_JSON Default/Balance, SINGBOX, STASH) drifting from current best practices. RU traffic was getting routed throughVPN in some templates (defeating local-bypass). One template had a 2-line bug. Couldn't break active subscriptions — paying customers were on these templates right now.

// diagnosis

Pulled all 6 from /api/subscription-templates and snapshot-saved them. Diffed each against the upstream community config to identify minimum-diff changes. Flagged the SINGBOX bug (RU bundle pointing to proxy instead of direct). Noticed the MIHOMO 🏠 RU group was select[DIRECT, VPN] — one tap from breaking RU-bypass.

// solution

Per-template minimum-diff:

· MIHOMO/CLASH — sync from upstream + lock 🏠 RU to [DIRECT] only
· XRAY_JSON Default — sync from Lite, keep inbounds/outbounds
· XRAY_JSON Balance — sync + auto_balancer + injectHosts + burstObservatory
· SINGBOX — fix 2-line bug, no other changes
· STASH — untouched (risk > reward)

Each applied via panel API with a per-template rollback command saved to README.

// result

All 6 migrated in one session, zero subscription disruption. Per-template rollbacks documented and tested. RU-bypass UX hardened against accidental misconfiguration.

// problem

A multi-node VPN fleet had no visibility into what users were doing through it. No way to investigate abuse complaints against specific destinations. No automated detection of port-scan patterns. Bridge architecture made it worse — hundreds of real users hid behind one synthetic identity at the exit node.

// diagnosis

Reviewed off-the-shelf solutions — none handled bridge-correlation, none had Telegram-friendly ergonomics. Sized the data: ~50–500MB/day per node. Needed ≥7d active + ≥90d historical retention.

// solution

Two-tier system in Go. Per-node agent tails access.log, batches entries, ships over WebSocket to central. Server normalizes & stores in Postgres 17, exposes a Next.js dashboard. Key subsystems:

· bridge correlation — time-based fan-out (±15s) writes one row per candidate to bridged_flows; most-frequent over a window = most likely originator
· anomaly detectors — port_scan, abuse_port_flood, burst_scan, blacklist via threat_matches
· anti-hang patch— configurable pongWait / pingPeriod / writeWait + TCP keepalive, replaced "drop entries → OOM → restart" with auto-reconnect
· SQLite→Postgres 17 migration — hybrid copy, 6s cutover, rollback warm. 65+ storage tests on testcontainers-go.

// result

All 8 fleet nodes reporting within a week of agent rollout (one node at a time, strict verification). Single SQL query attributes abuse to a real user IP within seconds. Survives WebSocket proxy flakiness — no manual restarts since the anti-hang patch. Open-sourced — 20+ operators run it now.

github.com/qwertyhq/xray-analyzer ↗

// problem

Standard MTProto distributions (mtg) had dropped ad-tag support. The Erlang seriyps fork kept ad-tag + FakeTLS + Secure-dd — but no Prometheus, no GeoIP, no way to see unique real Telegram sessions vs raw TCP connection counts.

// diagnosis

Read the source. mtp_metric exposed counter callbacks but no backend was wired up. No GeoIP module. The auth_key_id (first 8 bytes of the first decoded MTProto packet) was already parsed and then discarded — free signal for unique-session counts.

// solution

9 patches — 2 new modules + 7 modifications:

· mtp_prometheus_metric.erl — metric backend, registers active/passive, runs prometheus_httpd on :9091
· mtp_unique_users.erl — gen_server + 2 ETS tables (IP→country, auth_key_id→last_seen), 5-min sliding window, GeoIP via locus
· mtp_handler.erl — captures auth_key_id from the first decoded packet
· mtp_full.erl / mtp_down_conn.erl — counters on CRC/seq mismatch, handshake timeout, keepalive

Hard lessons baked in: iptables must ACCEPT lo first or curl 127.0.0.1 times out. Em-dash in an Erlang docstring → 500 from prometheus_http_impl. Per-IP labels = cardinality blow-up. ulimits.nofile=65536 or Ranch chokes around 1k conns.

// result

33/38 dashboard panels render live data (87%) — the 5 [no data] panels are conceptually impossible without rewriting state machines. "Real TG sessions" via unique_auth_keys_5mcorrelates within expected variance to MTProxybot's own daily count. Per-country breakdown (RU/PK/UA/DE/US) via GeoLite2.

github.com/qwertyhq/telemt-install ↗

// problem

Multi-VM setup on a mixed bag of small VPSes was painful to operate: backups inconsistent, no private mesh between hosts, monitoring fragmented, deployments depended on hosting-provider quirks. Needed to consolidate onto owned hardware without losing small-VM ergonomics.

// diagnosis

Sized the workload: 3–4 production VMs, ~64GB RAM, snapshot-friendly FS, growth headroom. Chose OVH dedicated (Ryzen 9700X, 64GB, ZFS mirror) over hyperscaler — better $/RAM, full control over kernel/networking.

// solution

End-to-end build:

· Proxmox VE 9 on ZFS mirror, per-VM zvols, 15-minute ZFS snapshots
· PBS as an LXC on the same host, daily 4:00 backups across all VMs
· R2 offsite sync (WIP) for 3-2-1 backup policy
· NetBird mesh as data plane — VMs + Mac client all peers on 10.10.10.0/24 overlay; no public ports on app VMs
· External edge node terminates TLS (Caddy) and socats into the mesh — clean separation between exposed surface and private infra
· Prometheus + Grafana + Loki, alerts piped to Telegram

// result

One dedicated host runs SHM billing VM, VPN panel VM, frontend VM, and PBS — with capacity headroom. Backups verified end-to-end (PBS restore tested, ZFS rollback tested). Documented in a dedicated infra/ repo with runbooks.md + known-issues.md.

// problem

A scanning incident upstream made it clear that public :22 and Proxmox UI :8006 were liabilities. Closing them without breaking access was the actual challenge — the same firewall change that hides you from scanners can lock the operator out if the NetBird daemon hiccups.

// diagnosis

Inventoried external surface: :51820/udp NetBird mesh (the new control plane, keep), and :22 / :8006 / :25 / :111 / :8080 / :9100 / :9115 / :9874 / :3128 — all DROP candidates. Mapped existing iptables — NetBird daemon installs dynamic ACL chains that get flushed by iptables-restore. That's the foot-gun.

// solution

A safety-net pattern I now use for every iptables change on prod:

Then the lockdown itself: custom NETBIRD-SAFETY chain DROP-ing everything on vmbr0 except :51820/udp + ICMP. SSH alias switched to NetBird IP with ProxyJump via edge node as fallback.

// result

Public surface reduced to :51820/udp + ICMP. Scanners see nothing else. Two independent access paths (direct NetBird, ProxyJump fallback) — no single point of failure. Rollback pattern is reusable, documented, exercised.

// problem

The Telegram Mini App needed realtime push for billing/service events. Initial WebSocket implementation worked locally but died on production: connections closed after ~1 second under the production reverse proxy (a Caddy build with a security plugin that quietly broke the WebSocket tunnel).

// diagnosis

Reproduced with Node's raw HTTP/1.1 client — WebSocket handshake completed, then the connection got cut mid-stream. Browser WebSocket API (HTTP/2 Extended CONNECT) failed even harder. The proxy was the problem, not the server or the client. Two options on the table: replace the proxy stack (high-risk for a single feature), or switch transport — drop to Server-Sent Events, which is just a long-lived HTTP response and works through every HTTP proxy in the world.

// solution

Picked SSE and built it so WebSocket remained a fallback:

· Single realtime-server.pl handles both protocols on the same port
· SSE became the primary transport, WebSocket kept for future bidirectional needs
· Backend wired into Redis pub/sub (shm:events) — same notify path, different transport
· Frontend: shared event handler handleRealtimeEvent; separate transport hooks useSSEUpdates / useWebSocketUpdates; transport selectable via VITE_REALTIME_TRANSPORT
· nginx sse-location.conf disables buffering on /sse

// result

SSE works reliably through the production proxy — no patches to the proxy itself, no version pin. Transport is swappable by env var, so future-proof if the proxy stack changes. One realtime server, two protocols, single code path for events.

// problem

Subscribers were spread across Clash, Mihomo, Stash and SingBox clients. None of them are branded for the service, RU-bypass ergonomics are fragile (one tap from sending RU traffic through the tunnel), and support requests need to start with "which client are you on, what version, what mode". A single native client owned by us removes that whole class of friction.

// diagnosis

Mihomo (the Clash-fork proxy core) has a clean REST API and ships statically-linked binaries for Mac/Win/Linux — perfect to bundle. Electron + Vite gives one codebase for Mac and Windows, native window chrome, and access to OS-level signals (SSID, HWID, OS version) we need for support and auto-pause.

// solution

Electron app shape:

· Main process — owns the bundled mihomo binary, IPC handlers, system-proxy toggling, elevation for first-run admin tasks
· Renderer — React + Radix UI; pages for connections, proxies, rules, DNS, TUN, sniffer, logs, system proxy, resources, settings
· Profile sync — pulls subscription from the Remnawave URL, runs buildControlledOverlay + cleanProfile to merge app config and strip user-mutable bits (DNS / TUN / sniffer / external-controller / LAN IPs)
· Floating window — borderless traffic widget with context menu, lives outside the main window for menubar-style quick toggle
· SSID-aware auto-pause — reads current Wi-Fi SSID, pauses the tunnel on configured trusted networks (so home Wi-Fi traffic skips the VPN automatically)
· Device info — HWID / OS / version / model exposed for support correlation
· i18n + custom theme system (CSS custom properties, swappable at runtime)

// result

One codebase ships to electron-builder --mac and --win. Subscribers get a branded client with the bypass model locked correctly, support gets device fingerprints in one click, and the proxy core (Mihomo) is the same as the community standard — so debugging tracks back to upstream behaviour, not a custom fork.

// problem

A maritime client monitoring an exclusive economic zone (EEZ) ran their watch across half a dozen single-purpose tools: vessel registry in one place, AIS alerts in another, asset dispatch by phone, UAV telemetry in a separate viewer, SIGINT in a closed CLI. Investigations spanned hours of context-switch before someone could say this boat, this incident, this action.

// diagnosis

The fix wasn't more tools — it was one operator console that put the map first. Side panels for situational data (alerts, vessels, asset status), modals for time-critical actions (dispatch, contact SOP, case open), and a case surface that ties an alert all the way through to closure.

// solution

Built as a single-page operator workstation, deployed to the client's ops team:

· Map-first layout — Mapbox GL via react-map-gl, AIS-status colored vessel markers, EEZ polygon overlay, zoom-aware clustering
· Alert workflow — severity-graded panel, acknowledge / escalate / open-case actions, audit trail
· Asset dispatch — navy / coast guard / UAV inventory, dispatch dialog with VHF / SatComm contact SOP wired in
· UAV control — live feed slot, telemetry panel, remote command hand-off
· SIGINT collection — IMSI/IMEI intercept log, satcomm session metadata, attribution to vessel-of-interest
· Case management — full lifecycle: alert → investigation → dispatch → contact → closure with evidence pack

Built on Next.js 15 App Router, TypeScript 5.7, shadcn/ui primitives, Zustand state, Tailwind 4. Playwright for end-to-end coverage of the alert→case workflows.

// result

One operator surface replaces five. Alert acknowledgment to asset dispatch is two clicks. SIGINT and case history live in the same audit thread as the alert that triggered the investigation. Stack is plain web — no Java client, no installer, no platform lock.

// problem

I already had an Erlang MTProto stack (see Case 04) tuned for observability — but observability and absolute scale are different problems. At provider-level concurrency (millions of live TCP sessions), Erlang runtime overhead becomes the bottleneck. I wanted a clean C implementation with kernel-level acceleration and no language-runtime baggage.

// diagnosis

The pieces needed: kernel-side filtering of bad TCP attempts before they hit userspace (TCP_DEFER_ACCEPT), a memory allocator that survives high churn without fragmentation death (jemalloc with background-thread GC), shared-memory IP stats so multi-worker counters don't need a coordinator, and a hardened build so a hostile peer can't buffer-overflow the proxy into something unpleasant.

// solution

C99 from scratch. The build-time decisions matter as much as the runtime ones:

· TCP_DEFER_ACCEPT — kernel drops zero-byte connections before they cost a syscall in userspace
· jemalloc with MALLOC_CONF=background_thread:true,dirty_decay_ms:5000,muzzy_decay_ms:30000 — survives the allocation pattern of millions of short-lived sessions
· Multi-worker, shared-memory IP table — workers count uniques without coordinator overhead; dynamic table scales to 10M+ entries
· Native Prometheus metrics — per-country unique IPs, Russian region breakdown, online/total — fed by libmaxminddb + GeoLite2-City
· Runtime config — TOML file + --cli-flag overrides; no recompile to retune workers, GeoIP path, stats endpoint
· Build hardening — stack-protector, FORTIFY_SOURCE=2, PIE, full RELRO, BIND_NOW
· systemd — LimitNOFILE=10485760, TasksMax=infinity, BBR congestion control on the host

// result

Production-grade proxy with native observability and a hardened binary. Open-sourced under GPLv2 — the goal is for other operators running provider-scale Telegram traffic to have a sane baseline to start from.

github.com/dpibreaker/mtbolt ↗