DEV Community

#llminference

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
FreeToken: Run Frontier MoE Models on Your Own GPU, and the RAM It Takes
itxtayab profile

FreeToken: Run Frontier MoE Models on Your Own GPU, and the RAM It Takes

#opensource #localllm #mixtureofexperts #llminference
Comments
9 min read
What optimizations would you employ to achieve high throughput and low latency?
technaom profile

What optimizations would you employ to achieve high throughput and low latency?

#aiinterview #llminference #genai #aiarchitect
Comments
2 min read
LLM cold starts on HyperPod: from 27 minutes to seconds with model caching
fernando_azevedo_6844e930 profile

LLM cold starts on HyperPod: from 27 minutes to seconds with model caching

#aiagents #sagemakerhyperpod #llminference #coldstart
Comments
9 min read
vLLM v0.28.0: the breaking change small GPU users must read
induwara_ashinsana_9e4d5b profile

vLLM v0.28.0: the breaking change small GPU users must read

#vllm #llminference #opensource
Comments
5 min read
VKAE: VIDRAFT's Inference Engine Hits 23 GPU Speedup and ~10K Tokens/sec on a Single Nvidia B200
ai_openfree_b23025ef075cf profile

VKAE: VIDRAFT's Inference Engine Hits 23 GPU Speedup and ~10K Tokens/sec on a Single Nvidia B200

#llminference #gpuacceleration #vkae #nvidiab200
5
Comments
4 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.