Configuration-Induced Delivery Failures in NATS JetStream: Detection and Remediation

Biplab Kumar Das

Home/Archive/arXiv

arXiv · 2609.35310

Configuration-Induced Delivery Failures in NATS JetStream: Detection and Remediation

Biplab Kumar Das

View original source ↗Read source document ↗Export metadata ↓

Abstract

NATS JetStream's at-least-once delivery guarantee is conditional: five common configuration mistakes silently violate it, causing duplicate message processing, data loss, or redelivery storms with no error logged anywhere. The standard Prometheus NATS exporter exposes only server-level throughput metrics and cannot detect any of these failures. We present nats-lens, a standalone monitor that reads from the JetStream management API and detects all five violation classes without requiring changes to monitored applications or client code. We formally characterize each class with a precise condition, prove that standard Prometheus NATS metrics are structurally incapable of detecting any of them, and implement five targeted detectors. In a controlled evaluation of 30 rounds per scenario on both single-node and 3-node JetStream clusters, nats-lens achieves 100% detection coverage across all five classes---versus 0% for the baseline---with zero false positives over 30 minutes of healthy operation. Detection latency ranges from 2,003 ms to 8,013 ms (within three poll cycles). We confirm language-agnostic detection empirically using consumers in Rust, Go, and Python. The tool is open source and exposes findings through four output channels: web dashboard, Prometheus metrics, REST API, and NATS health events.

Explore related subjects

cs.NI

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Biplab Kumar Das. 2026-09-28. Configuration-Induced Delivery Failures in NATS JetStream: Detection and Remediation. https://arxiv.org/abs/2609.35310

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

SHORTCUT: In-Collective Topology Reconfiguration for Low-Latency AllReduce

Distributed ML training relies on efficient AllReduce communication to aggregate data across nodes. In this setting, reconfigurable optical interconnects offer high-bandwidth, energy-efficient direct links between accelerators but often produce ring-based topologies that remain static during a collective. The Ring AllReduce algorithm naturally matches these topologies but its cumulative latency grows linearly with node count. Low-latency algorithms such as Recursive Doubling (RD) instead achieve logarithmic cumulative per-step latency, but their long-distance exchanges incur dilation and congestion costs on a static ring. In-collective topology reconfiguration can eliminate these penalties but each topology change adds reconfiguration delay. The key question to improve AllReduce completion time is therefore not only how to reconfigure RD efficiently, but when selectively reconfigured RD becomes faster than the topology-matched Ring algorithm. We present Shortcut: an effective strategy for topology reconfiguration that enables RD to shortcut costly multi-hop communication only when it pays off - beyond the performance of the Ring algorithm. Across small and medium messages on 32 nodes, Shortcut achieves $4.4\times$-$6.0\times$ speedups over Ring. At 128 nodes, it remains up to $7\times$ faster with a $10\,μs$ reconfiguration delay, showing that selective reconfiguration is especially effective in latency-sensitive, large-scale settings.

cs.NI↗

arXiv2026

Enhancing BGP Security by Understanding BGP's Language with LLMs

The trust-based nature of Border Gateway Protocol (BGP) makes it vulnerable to prefix hijacking and misconfigurations. Traditional BGP anomaly detection relies on manual inspection with poor scalability, while Machine/Deep Learning (M/DL)-based approaches suffer from suboptimal precision, limited generalizability, and high retraining cost. This is because existing M/DL methods focus on topological structures rather than semantic characteristics of Autonomous Systems (ASes), assigning dissimilar embeddings to functionally similar but topologically distant ASes. To address this, we propose BGPShield, a novel anomaly detection framework built on an Adaptive LLM BGP Encoder that captures each AS's Behavior Portrait and Routing Policy Rationale beyond topology. Inspired by multimodal LLMs, the encoder generates embeddings representing both routing behaviors and semantics of ASes via contrastive learning. We further introduce SAM-ED to quantify BGP-specific semantic deviations between historical and updated paths, rather than naively accumulating distances without awareness of BGP-specific structures. Evaluated on 16 real-world datasets, BGPShield detects 100% of verified anomalies with an average false discovery rate below 5%. The open-source LLMs used by BGPShield were released prior to several evaluation events, verifying generalizability on unseen events. Furthermore, BGPShield can construct the representation for a previously unseen AS within one second, significantly outperforming BEAM which demands thorough retraining (averagely 65 hours).

cs.NI↗

arXiv2026

GATE: GPU-Accelerated Traffic Engineering for the WAN

Traffic engineering (TE) has become a crucial tool for enforcing routing policy and maintaining operational efficiency in large networks. Existing TE solutions pick an objective function to optimize, aiming to balance (i) allocating traffic optimally with (ii) reacting quickly to demand changes and disruption events. However, as the scale of networks grows, the runtime of the existing optimal solution becomes infeasibly large. The alternative - approximate solvers - result in costly inefficiencies. We present GPU-Accelerated Traffic Engineering (GATE), which achieves the best of both worlds: enabling fast TE runtimes through a highly-parallelizable GPU-compatible decomposition, while iteratively converging to the provably optimal solution. GATE unlocks a unique set of desirable properties: it becomes increasingly parallelizable with network size, supports a wide spectrum of fairness objectives, and offers theoretically guaranteed convergence to the optimal solution and near-optimal convergence within a bounded time. We evaluate GATE on production traces from two large cloud WANs, and show that GATE achieves near-optimal solutions 4-10x faster than state-of-the-art.

cs.NI↗