cloud

ZenteiQ scales batch inference 4x with GKE Inference Gateway powered by llm-d

ZenteiQ uses GKE Inference Gateway and llm-d to boost multi-model inference throughput 4x and keep foundation model ...
NextBigFuture

Defeating Nondeterminism in LLM Inference by Thinking Machines

A research article by Horace He and the Thinking Machines Lab (X-OpenAI CTO Mira Murati founded) addresses a long-standing issue in large language models (LLMs). Even with greedy decoding bu setting ...
BIOENGINEER

New Co-Design Framework Speeds Up LLM Inference on Mixed Hardware by Taming Memory Bottlenecks

CoOpt, an algorithm-hardware co-design framework that boosts large language model inference throughput by up to 13.43 percent ...
Martin Cid Magazine

llm-d 1.0: Red Hat, IBM, Google and Nvidia bet that AI’s real bill is inference, not training

For three years the AI conversation has been about who can build the biggest model. The less glamorous question — what it ...
MarkTechPost

Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets

Google team have released llm-d 1.0, the first stable version of its distributed inference stack. This open-source, ...