DEV Community

#llamacpp

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Do Drafters Work on Small Models? Gemma 4 E2B and E4B on Two Small GPUs
xbill profile

Do Drafters Work on Small Models? Gemma 4 E2B and E4B on Two Small GPUs

#gemma #llamacpp #cuda #benchmarking
7
Comments
19 min read
Running MoE LLMs Larger Than Your RAM: A Practical Guide to mmap, Expert Offloading and SSD Streaming with llama.cpp
eme_gug_0821b41b948be6516 profile

Running MoE LLMs Larger Than Your RAM: A Practical Guide to mmap, Expert Offloading and SSD Streaming with llama.cpp

#llm #llamacpp #moe #performance
Comments
8 min read
Same Local Model, 3 Coding Agents: pi 12/12, Codex 12/12, Claude Code 7/12 on a 6GB GPU
kunfieldnotes profile

Same Local Model, 3 Coding Agents: pi 12/12, Codex 12/12, Claude Code 7/12 on a 6GB GPU

#localllm #llamacpp #ai #claudecode
2
Comments 4
3 min read
Choosing a Local LLM Through Fate (2026 Edition)
akari_iku profile

Choosing a Local LLM Through Fate (2026 Edition)

#ai #llm #localhackday #llamacpp
2
Comments
28 min read
Gemma 4 on a Used 8 GB RTX 3050: 116 Tokens/s for One User, 618 for 64

Uses a 2014 desktop and custom MCP tools
xbill profile

Gemma 4 on a Used 8 GB RTX 3050: 116 Tokens/s for One User, 618 for 64

#gemma #llamacpp #cuda #localllm
10
Comments 6
13 min read
llama-server's prompt cache reuses KV computed under a different LoRA scale
homelabpm profile

llama-server's prompt cache reuses KV computed under a different LoRA scale

#llamacpp #llm #selfhosted #lora
Comments
6 min read
Fine-tuning Qwen2.5-1.5B on a Mac to fix a parroting chatbot (and halve the prompt)
kharlampiiev profile

Fine-tuning Qwen2.5-1.5B on a Mac to fix a parroting chatbot (and halve the prompt)

#ai #llm #machinelearning #llamacpp
Comments
6 min read
Running 100B+ MoE Models on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp
eme_gug_0821b41b948be6516 profile

Running 100B+ MoE Models on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp

#llm #llamacpp #gpu #machinelearning
Comments
8 min read
Running 100B+ MoE LLMs on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp
eme_gug_0821b41b948be6516 profile

Running 100B+ MoE LLMs on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp

#llm #llamacpp #gpu #selfhosted
Comments
7 min read
Gemma 4's MTP Drafter, Explained: Speculative Decoding and a 4-Bit Drafter at 60% of the Size
xbill profile

Gemma 4's MTP Drafter, Explained: Speculative Decoding and a 4-Bit Drafter at 60% of the Size

#gemma #llamacpp #huggingface #ai
2
Comments 2
9 min read
When Chat Templates Go Wrong
pielounw profile

When Chat Templates Go Wrong

#llm #opensource #ai #llamacpp
Comments
7 min read
Deploying LLMs Locally: Ollama, llama.cpp, vLLM & Text Generation Inference
ipsphil_digital profile

Deploying LLMs Locally: Ollama, llama.cpp, vLLM & Text Generation Inference

#ollama #llamacpp #vllm #tutorial
Comments
12 min read
Deploying LLMs Locally: Complete Guide to Ollama, llama.cpp, vLLM & TGI
ipsphil_digital profile

Deploying LLMs Locally: Complete Guide to Ollama, llama.cpp, vLLM & TGI

#ollama #llamacpp #vllm #tutorial
Comments
9 min read
Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Comparison
ipsphil_digital profile

Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Comparison

#llm #ollama #llamacpp #tutorial
Comments
5 min read
llama.cpp vs Ollama in 2026: Which Runtime Should You Run?
rosgluk profile
Rost

llama.cpp vs Ollama in 2026: Which Runtime Should You Run?

#llamacpp #ollama #llm #selfhosting
Comments
18 min read
👋 Sign in for the ability to sort posts by relevant, latest, or top.