Quantization

👋 Sign in for the ability to sort posts by relevant, latest, or top.

amrit

Oct 12

How One Startup Cut Diffusion‑LLM Inference Costs by 45 % – No Calibration Needed

#ai #quantization #diffusionllm

6 min read

Muhammad Tahir

Oct 12

Model Distillation & Quantization: Running 8B Parameter Models at a Fraction of Cloud Cost

#modeldistillation #quantization #vllm #awq

10 min read

Jon Handler

Oct 8

From 1.3 TB to 66 GB: On-Disk Vector Search in OpenSearch

#vectorsearch #airetrieval #opensearch #quantization

8 min read

Jon Handler

Oct 6

Native Quantization: Let OpenSearch Service Compress Your Vectors

#vectorsearch #airetrieval #opensearch #quantization

6 min read

GINIGEN AI

Oct 6

4-bit GGUF Quality for MoE Models: Why Only 3B of 180B Params Fire, and How to Prove Parity

#ai #llm #quantization #edgeai

7 min read

GINIGEN AI

Oct 6

Running a 180B Model on a Laptop With No GPU: How 4-bit GGUF Keeps Full Accuracy

#ai #llm #edgeai #quantization

6 min read

Reno Lu

Oct 2

Qwen3.8-27B on one RTX 3090, and the setting its README says chat clients should leave at default

#vllm #localllm #speculativedecoding #quantization

4 min read

Jon Handler

Oct 2

1.3 TB of RAM for 100M Vectors? Shrink Them Before They Hit OpenSearch

#vectorsearch #airetrieval #opensearch #quantization

8 min read

Nainik Mehta

Sep 23

Hot Take: Deploy Small Language Models to the Edge

#edge #machinelearning #mlops #quantization

4 min read

Syed Anzar

Sep 18

What Model Quantization Actually Does: From Float16 to 4-Bit Weights

#ai #llm #quantization #machinelearning

6 min read

Qasim Parray

Sep 30

Testing the 9x Smaller Local LLM Claim on a GPU-less VPS

#localllm #quantization #selfhosting #gguf

6 min read

Anaz S. Aji for Codecora Dev

Sep 2

A Small Transformer Trained in 1.5 Hours Beat Many LLMs on ARC

#machinelearning #vectorsearch #quantization #benchmark

5 min read

Seth Wheeler

Aug 25

A Better FP4 Gradient Quantizer That Training Couldn't Notice

#llm #measurement #quantization #training

7 min read

Eryk Kubiak

Sep 25

Why Does Your Local Model Crash at 32k Tokens?

#llm #gpu #quantization #machinelearning

12 min read

Seth Wheeler

Aug 21

Error Feedback, Gradient Compression, and Why Adam Breaks It

#llm #measurement #quantization #training

8 min read

👋 Sign in for the ability to sort posts by relevant, latest, or top.