// problem

Users on a private VPN service reported that Telegram was noticeably laggy on phones over LTE, while desktop and Wi-Fi felt fine. Symptoms didn't match a typical "bad node" story — exit nodes were healthy, Telegram-owned domains resolved and responded in 100–300ms from every exit.

// diagnosis

Captured packets on a real phone, not a synthetic test. The failure was not in the VPN path — it was Happy Eyeballs. Carriers handed dual-stack, the client opened the IPv6 socket through the tunnel, 6 of 8 exits had no IPv6, and the kernel reject + xray buffering added 200–500ms before fallback. Telegram opens dozens of concurrent sockets; the delay surfaced as "laggy UI".

// solution

Two-layer fix. Server-side: instant ip6tables reject so clients fall back immediately. Proper fix in Xray routing.rules, applied across all 6 exit profiles and both RU bridges:

Rollout strictly one node at a time with verify-after-each (sub-3s xray reload, client auto-reconnect).

// result

Mobile Telegram latency restored to feel parity with desktop on every tested LTE carrier. ~15 minutes per node including verification. No client-side changes required.

// problem

Live VPN service had 6 subscription templates (MIHOMO, CLASH, XRAY_JSON Default/Balance, SINGBOX, STASH) drifting from current best practices. RU traffic was getting routed throughVPN in some templates (defeating local-bypass). One template had a 2-line bug. Couldn't break active subscriptions — paying customers were on these templates right now.

// diagnosis

Pulled all 6 from /api/subscription-templates and snapshot-saved them. Diffed each against the upstream community config to identify minimum-diff changes. Flagged the SINGBOX bug (RU bundle pointing to proxy instead of direct). Noticed the MIHOMO 🏠 RU group was select[DIRECT, VPN] — one tap from breaking RU-bypass.

// solution

Per-template minimum-diff:

· MIHOMO/CLASH — sync from upstream + lock 🏠 RU to [DIRECT] only
· XRAY_JSON Default — sync from Lite, keep inbounds/outbounds
· XRAY_JSON Balance — sync + auto_balancer + injectHosts + burstObservatory
· SINGBOX — fix 2-line bug, no other changes
· STASH — untouched (risk > reward)

Each applied via panel API with a per-template rollback command saved to README.

// result

All 6 migrated in one session, zero subscription disruption. Per-template rollbacks documented and tested. RU-bypass UX hardened against accidental misconfiguration.

// problem

A multi-node VPN fleet had no visibility into what users were doing through it. No way to investigate abuse complaints against specific destinations. No automated detection of port-scan patterns. Bridge architecture made it worse — hundreds of real users hid behind one synthetic identity at the exit node.

// diagnosis

Reviewed off-the-shelf solutions — none handled bridge-correlation, none had Telegram-friendly ergonomics. Sized the data: ~50–500MB/day per node. Needed ≥7d active + ≥90d historical retention.

// solution

Two-tier system in Go. Per-node agent tails access.log, batches entries, ships over WebSocket to central. Server normalizes & stores in Postgres 17, exposes a Next.js dashboard. Key subsystems:

· bridge correlation — time-based fan-out (±15s) writes one row per candidate to bridged_flows; most-frequent over a window = most likely originator
· anomaly detectors — port_scan, abuse_port_flood, burst_scan, blacklist via threat_matches
· anti-hang patch— configurable pongWait / pingPeriod / writeWait + TCP keepalive, replaced "drop entries → OOM → restart" with auto-reconnect
· SQLite→Postgres 17 migration — hybrid copy, 6s cutover, rollback warm. 65+ storage tests on testcontainers-go.

// result

All 8 fleet nodes reporting within a week of agent rollout (one node at a time, strict verification). Single SQL query attributes abuse to a real user IP within seconds. Survives WebSocket proxy flakiness — no manual restarts since the anti-hang patch. Open-sourced — 20+ operators run it now.

github.com/qwertyhq/xray-analyzer ↗

// problem

Standard MTProto distributions (mtg) had dropped ad-tag support. The Erlang seriyps fork kept ad-tag + FakeTLS + Secure-dd — but no Prometheus, no GeoIP, no way to see unique real Telegram sessions vs raw TCP connection counts.

// diagnosis

Read the source. mtp_metric exposed counter callbacks but no backend was wired up. No GeoIP module. The auth_key_id (first 8 bytes of the first decoded MTProto packet) was already parsed and then discarded — free signal for unique-session counts.

// solution

9 patches — 2 new modules + 7 modifications:

· mtp_prometheus_metric.erl — metric backend, registers active/passive, runs prometheus_httpd on :9091
· mtp_unique_users.erl — gen_server + 2 ETS tables (IP→country, auth_key_id→last_seen), 5-min sliding window, GeoIP via locus
· mtp_handler.erl — captures auth_key_id from the first decoded packet
· mtp_full.erl / mtp_down_conn.erl — counters on CRC/seq mismatch, handshake timeout, keepalive

Hard lessons baked in: iptables must ACCEPT lo first or curl 127.0.0.1 times out. Em-dash in an Erlang docstring → 500 from prometheus_http_impl. Per-IP labels = cardinality blow-up. ulimits.nofile=65536 or Ranch chokes around 1k conns.

// result

33/38 dashboard panels render live data (87%) — the 5 [no data] panels are conceptually impossible without rewriting state machines. "Real TG sessions" via unique_auth_keys_5mcorrelates within expected variance to MTProxybot's own daily count. Per-country breakdown (RU/PK/UA/DE/US) via GeoLite2.

github.com/qwertyhq/telemt-install ↗

// problem

Multi-VM setup on a mixed bag of small VPSes was painful to operate: backups inconsistent, no private mesh between hosts, monitoring fragmented, deployments depended on hosting-provider quirks. Needed to consolidate onto owned hardware without losing small-VM ergonomics.

// diagnosis

Sized the workload: 3–4 production VMs, ~64GB RAM, snapshot-friendly FS, growth headroom. Chose OVH dedicated (Ryzen 9700X, 64GB, ZFS mirror) over hyperscaler — better $/RAM, full control over kernel/networking.

// solution

End-to-end build:

· Proxmox VE 9 on ZFS mirror, per-VM zvols, 15-minute ZFS snapshots
· PBS as an LXC on the same host, daily 4:00 backups across all VMs
· R2 offsite sync (WIP) for 3-2-1 backup policy
· NetBird mesh as data plane — VMs + Mac client all peers on 10.0.0.0/24 overlay; no public ports on app VMs
· External edge node terminates TLS (Caddy) and socats into the mesh — clean separation between exposed surface and private infra
· Prometheus + Grafana + Loki, alerts piped to Telegram

// result

One dedicated host runs SHM billing VM, VPN panel VM, frontend VM, and PBS — with capacity headroom. Backups verified end-to-end (PBS restore tested, ZFS rollback tested). Documented in a dedicated infra/ repo with runbooks.md + known-issues.md.

// problem

A scanning incident upstream made it clear that public :22 and Proxmox UI :8006 were liabilities. Closing them without breaking access was the actual challenge — the same firewall change that hides you from scanners can lock the operator out if the NetBird daemon hiccups.

// diagnosis

Inventoried external surface: :51820/udp NetBird mesh (the new control plane, keep), and :22 / :8006 / :25 / :111 / :8080 / :9100 / :9115 / :9874 / :3128 — all DROP candidates. Mapped existing iptables — NetBird daemon installs dynamic ACL chains that get flushed by iptables-restore. That's the foot-gun.

// solution

A safety-net pattern I now use for every iptables change on prod:

Then the lockdown itself: custom NETBIRD-SAFETY chain DROP-ing everything on vmbr0 except :51820/udp + ICMP. SSH alias switched to NetBird IP with ProxyJump via edge node as fallback.

// result

Public surface reduced to :51820/udp + ICMP. Scanners see nothing else. Two independent access paths (direct NetBird, ProxyJump fallback) — no single point of failure. Rollback pattern is reusable, documented, exercised.

// problem

The Telegram Mini App needed realtime push for billing/service events. Initial WebSocket implementation worked locally but died on production: connections closed after ~1 second under the production reverse proxy (a Caddy build with a security plugin that quietly broke the WebSocket tunnel).

// diagnosis

Reproduced with Node's raw HTTP/1.1 client — WebSocket handshake completed, then the connection got cut mid-stream. Browser WebSocket API (HTTP/2 Extended CONNECT) failed even harder. The proxy was the problem, not the server or the client. Two options on the table: replace the proxy stack (high-risk for a single feature), or switch transport — drop to Server-Sent Events, which is just a long-lived HTTP response and works through every HTTP proxy in the world.

// solution

Picked SSE and built it so WebSocket remained a fallback:

· Single realtime-server.pl handles both protocols on the same port
· SSE became the primary transport, WebSocket kept for future bidirectional needs
· Backend wired into Redis pub/sub (shm:events) — same notify path, different transport
· Frontend: shared event handler handleRealtimeEvent; separate transport hooks useSSEUpdates / useWebSocketUpdates; transport selectable via VITE_REALTIME_TRANSPORT
· nginx sse-location.conf disables buffering on /sse

// result

SSE works reliably through the production proxy — no patches to the proxy itself, no version pin. Transport is swappable by env var, so future-proof if the proxy stack changes. One realtime server, two protocols, single code path for events.

// problem

Subscribers were spread across Clash, Mihomo, Stash and SingBox clients. None of them are branded for the service, RU-bypass ergonomics are fragile (one tap from sending RU traffic through the tunnel), and support requests need to start with "which client are you on, what version, what mode". A single native client owned by us removes that whole class of friction.

// diagnosis

Mihomo (the Clash-fork proxy core) has a clean REST API and ships statically-linked binaries for Mac/Win/Linux — perfect to bundle. Electron + Vite gives one codebase for Mac and Windows, native window chrome, and access to OS-level signals (SSID, HWID, OS version) we need for support and auto-pause.

// solution

Electron app shape:

· Main process — owns the bundled mihomo binary, IPC handlers, system-proxy toggling, elevation for first-run admin tasks
· Renderer — React + Radix UI; pages for connections, proxies, rules, DNS, TUN, sniffer, logs, system proxy, resources, settings
· Profile sync — pulls subscription from the Remnawave URL, runs buildControlledOverlay + cleanProfile to merge app config and strip user-mutable bits (DNS / TUN / sniffer / external-controller / LAN IPs)
· Floating window — borderless traffic widget with context menu, lives outside the main window for menubar-style quick toggle
· SSID-aware auto-pause — reads current Wi-Fi SSID, pauses the tunnel on configured trusted networks (so home Wi-Fi traffic skips the VPN automatically)
· Device info — HWID / OS / version / model exposed for support correlation
· i18n + custom theme system (CSS custom properties, swappable at runtime)

// result

One codebase ships to electron-builder --mac and --win. Subscribers get a branded client with the bypass model locked correctly, support gets device fingerprints in one click, and the proxy core (Mihomo) is the same as the community standard — so debugging tracks back to upstream behaviour, not a custom fork.