Architectural Case Study
Distributed SystemsLLM InferencePythonPyTorch

SwarmInfer: Decentralized Heterogeneous Swarm LLM Inference Engine

Decentralized, heterogeneous swarm inference engine pooling consumer devices (Metal MPS, CUDA, CPU) to run 70B+ LLMs without port forwarding or VPNs.

SwarmInfer - Decentralized Heterogeneous Swarm LLM Inference Engine
70B+ Frontier LLM Swarm Pooling
< 300ms WAN P2P Activation Latency
0 MB PyTorch Streaming Host RAM
380+ Adversarial Tests Passed (Zero Leak)

SwarmInfer: Decentralized Heterogeneous Swarm LLM Inference Engine

SwarmInfer is a decentralized, heterogeneous swarm inference engine designed to run frontier Large Language Models (14B, 32B, 70B+) by aggregating the memory and compute of everyday consumer hardware across the internet without requiring VPNs, port forwarding, or centralized relay servers.

Problem Solved

Serving modern frontier open-weight models introduces steep infrastructural and financial barriers:

  • The VRAM & Hardware Wall: Running 32B to 70B+ models demands 32GB to 160GB+ of VRAM, pricing developers out with $5,000–$20,000 workstation requirements or expensive cloud GPU subscriptions.
  • Untapped Consumer Hardware: Many developers own capable machines—such as an Apple Silicon Mac (Metal MPS), a gaming PC with an NVIDIA RTX GPU (CUDA), and multi-core work laptops—with no seamless mechanism to pool their collective VRAM and compute power.
  • Datacenter Assumptions in Existing Frameworks: Frameworks like vLLM, DeepSpeed, and Ray assume high-speed, flat datacenter networks (InfiniBand/RoCE, homogeneous GPUs, direct IP routing). They fail immediately across consumer Wi-Fi, cellular 5G hotspots, and asymmetric network conditions.

Impact & Engineering Benchmarks

Rather than theoretical figures, SwarmInfer was engineered with verified systems-level benchmarks:

  • Zero-RAM Disk Shard Streaming: Pre-sliced .safetensors stream chunk-by-chunk directly from disk over network sockets—guaranteeing 0 MB PyTorch tensor RAM allocation on streaming hosts.
  • Ultra-Low Coordinator Footprint: The master coordinator requires $<50\text{ MB}$ of RAM, utilizing lazy layer materialization only after topology partitioning is resolved.
  • Sub-300ms WAN Activation Transport: Custom pipelined sliding-window UDP ($W=64$) streams activation tensors across cellular 5G hotspots with in-order sequence reassembly.
  • Zero-GC Latency Elimination: Eradicated all gc.collect() calls during forward passes and introduced an $O(1)$ fast path in ensure_shards_loaded(), guaranteeing stable tensor residency and zero latency spikes.
  • NAT Traversal Without VPNs: Combined RFC 5389 STUN and RFC 5128 Port-Window Sweeping ($\pm 64$ ports) to punch through symmetric and cone NATs without port forwarding or Tailscale.
  • Adversarial Verification: Validated across 380+ tests in 30+ test modules, including empirical stress suites verifying adversarial packet corruption, simulated packet drops, multi-process race conditions, and zero resource leaks (-W error::ResourceWarning).

Solution

SwarmInfer splits Transformer execution into a resilient, decentralized peer-to-peer pipeline:

  • Heterogeneous Fractional Sharding: Partitions Transformer layers dynamically using the Largest Remainder Method across asymmetric hardware (Metal MPS on macOS, NVIDIA CUDA on Linux/Windows, and multi-core CPU) proportional to available device memory and compute.
  • Decentralized Rendezvous & Topology: Coordinates discovery via BitTorrent BEP 15 UDP Trackers (Magnet hashes) for global peer rendezvous without centralized servers. Includes 60s join grace periods, 30s heartbeats, and auto-healing topology rebalancing.
  • P2P Ring Forwarding: Directly streams activation tensors between peers ($Node_1 \to Node_2 \to Node_3$) to maximize network bandwidth, with automatic fallback to star topology if an intermediate peer disconnects.
  • 100% Cache Bypass: Workers with pre-cached layer files transmit SKIP tokens and initialize directly from local storage, eliminating redundant network transfers on cluster restart.
  • Robust Daemon Lifecycle: Built a comprehensive CLI (serve, join, generate, status, firewall, stun) with atomic cluster state persistence (active_cluster.json) and clean SIGINT/SIGTERM process handling.

Tech Stack

  • Compute & ML: Python, PyTorch, Apple Metal (MPS), NVIDIA CUDA, HuggingFace Transformers, Safetensors
  • Networking & Protocols: Custom Sliding-Window UDP ($W=64$), RFC 5389 STUN, RFC 5128, BitTorrent BEP 15 UDP Trackers
  • Architecture: Asynchronous I/O (asyncio), P2P Ring Pipelining, Fractional Sharding (Largest Remainder)
  • Quality & Testing: Python unittest, Stress/Adversarial Network Testing Suites, Zero Resource Leak Enforcement