More Podcasts by InfoQ episodes

Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM thumbnail

Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM

Published 9 Nov 2026

Duration: 00:43:37

WebAssembly and Tornado VM enable efficient local inference, while LLMD optimizes distributed GPU usage and semantic routing for on-premises AI deployment.

Episode Description

In this episode, Edoardo Vacchi discusses the evolving landscape of WebAssembly on the backend, running Java workloads on GPUs using TornadoVM, and op...

Overview

WebAssembly has evolved from a browser technology into a backend sandboxing platform, with the component model driving much of current development. Its relevance has grown alongside the agentic workload trend, and it enables cross-language distribution: a library written in Rust can be compiled to WebAssembly and consumed from multiple languages. Running small language models in a browser via Wasm is feasible, though larger models generally require GPU acceleration; experiments exist to translate Wasm into GPU kernels.

Tornado VM is a JVM plugin that compiles Java code into GPU-specific kernels such as PTX for NVIDIA or SPIRV. It allows writing GPU kernels in Java, with an example of running Llama 3 entirely in Java, though it cannot run arbitrary Java applications like Spring on the GPU.

LLMD, a CNCF project that recently reached 1.0, orchestrates distributed inference on Kubernetes by solving routing challenges unique to LLM workloads. Because LLM conversations are stateful through KV and prefix caches, traditional round-robin load balancing degrades performance; LLMD balances sticky sessions against cluster saturation. It also provides batch inference for idle hardware, semantic classification to route prompts to right-sized models, and observability metrics. Vache notes that even single-instance or bare-metal deployments benefit from LLMD's flow control and quality-of-service features, and that smaller open models - such as 27-billion-parameter models - can run on consumer hardware, making on-premises inference increasingly practical.

What If

  • What if you ran small language model inference directly in browsers using WebAssembly for niche applications?

    • Move: Experiment with compiling lightweight Rust-based libraries to WebAssembly and serving them through browser-based UIs to handle specific, low-latency inference tasks without relying on remote APIs.
    • Why Now?: WebAssembly's sandboxing capabilities are increasingly relevant, and examples already exist - such as Hugging Face running small models in browsers - and browser-to-device live migration of Wasm workloads has been demonstrated.
    • Expected Upside: You could offer privacy-preserving, offline-capable AI features for narrow use cases (e.g., form-filling bots or local code assistants), differentiating your product with zero data leakage.
  • What if you deployed a Java-based LLM directly on GPUs using Tornado VM to reduce dependency on Python tooling?

    • Move: Prototype a Java-written inference kernel using Tornado VM to run on GPU backends, leveraging existing JVM expertise instead of adopting Python-centric AI stacks.
    • Why Now?: A fully Java-written implementation of Llama 3 running on GPU via Tornado VM has already been demonstrated, showing feasibility for developers embedded in Java ecosystems.
    • Expected Upside: This could streamline deployment in Java-heavy enterprises, reduce context switching, and open opportunities to build GPU-accelerated services without learning new languages or frameworks.
  • What if you used LLMD to optimize inference efficiency on affordable, on-premise hardware instead of relying on cloud APIs?

    • Move: Deploy LLMD with VLLM on a single high-RAM machine (e.g., a 48GB Mac or similar server) to serve models like the 27-billion-parameter Quen 3.8 27b locally, avoiding third-party inference costs.
    • Why Now?: Eduardo notes that models like Quen 3.8 27b run on laptops and are comparable to Opus 4.6 from just one year prior - indicating rapid capability democratization. LLMD is now production-ready and supports bare metal.
    • Expected Upside: You could offer compliant, low-latency AI services to clients without data leaving their infrastructure, using hardware that is now relatively affordable and sufficient for many real-world tasks.

Takeaway

  • Evaluate LLMD even for a single VLLM instance or bare-metal setup; it adds automatic flow control and HTTP QoS prioritization that VLLM alone does not provide.
  • Schedule batch inference during off-peak hours using LLMD's batch processing to keep expensive GPU hardware utilized at 100% and reduce per-unit inference cost.
  • If your team works in Java, prototype GPU kernels with Tornado VM (PTX/SPIRV output) before committing to a C++/CUDA stack; it supports writing Llama 3 inference entirely in Java.
  • Deploy a smaller open model (e.g., 27B parameters) on consumer-grade hardware for niche tasks rather than defaulting to frontier models, since "the old models are still useful" and can run on a single server.
  • Implement semantic routing (LLMD Semantic Classifier or VLLM Semantic Router) to match prompt complexity to model size, so routine queries hit a cheaper fine-tuned model and only complex prompts escalate to a larger one.

Recent Episodes of Podcasts by InfoQ

More Podcasts by InfoQ episodes