DEV Community

Cover image for Semantic cache for Bedrock with ElastiCache and Lambda
Sascha Mirow-Stanitzok
Sascha Mirow-Stanitzok

Posted on Edited on Originally published at sascha-ms.de

Semantic cache for Bedrock with ElastiCache and Lambda

A semantic cache answers a question from memory when a question with the same meaning has been asked before. That sounds like a small trick, yet in AI applications it can save a large share of model calls. The topic keeps coming back to me because many chatbot and FAQ projects show the same picture: hundreds of people ask the same twenty questions, just with different words each time. Even so, each time the language model computes the answer from scratch.

In this article, I build a semantic cache in front of Amazon Bedrock, using ElastiCache Serverless for Valkey as the vector store, a Lambda function and the Serverless Framework. For example, I use an FAQ assistant for a fictional bike rental. Just as important as the code, though, is the question of when such a cache becomes dangerous. After all, a false hit delivers a wrong answer with full confidence.

In short

A semantic cache stores a language model’s answers together with the embedding of the question and returns them again when a new question is close enough. It pays off most for recurring, general questions, and an exact cache level in front of it even saves the embedding. The biggest pitfall is false hits: questions that sound almost the same but need a different answer. Hard filters before the vector search and a carefully calibrated threshold protect you from them.

What you will learn in this article

  • How a semantic cache works and how it differs from prompt caching
  • When the cache pays off and which questions never belong in it
  • Preventing false hits with hard filters and a calibrated threshold
  • Building the cache with Lambda, ElastiCache Serverless and the Serverless Framework
  • Keeping an eye on hit rate and cost

What is a semantic cache?

A classic cache works with exact keys. If someone asks “How long can I rent a bike?”, the answer is stored under exactly that text. When the next person writes “What is the maximum rental period?”, the cache finds nothing, even though both want to know the same thing.

A semantic cache therefore compares meaning instead of wording. To do so, an embedding model turns every question into a vector, a long list of numbers. As a result, questions with a similar meaning end up close to each other. Then the cache looks up the nearest stored vector for the new question. If the distance is small enough, it returns the stored answer. Otherwise it asks the language model and stores that answer for next time.

In practice, the distance is usually measured as cosine distance. For normalized vectors it ranges from 0 to 2, where 0 means “same direction”. Many articles talk about similarity instead, which is simply 1 minus the distance. So a distance of 0.1 corresponds to a similarity of 0.9.

How it differs from prompt caching

The name is easy to confuse, so here is a quick distinction. Prompt caching in Bedrock remembers a long, always identical beginning of a prompt, for example an extensive system prompt. However, the model is still called for every request; only the repeated part is processed more cheaply and quickly. A semantic cache, by contrast, skips the model call entirely. The two don’t exclude each other: prompt caching helps with every miss of the semantic cache.

When a semantic cache pays off

At the end of 2025, AWS published an evaluation on its Database Blog with roughly 63,000 real chatbot queries, placing a semantic cache on ElastiCache in front of Bedrock. With a similarity threshold of 0.75, daily costs in that experiment dropped by about 86 percent, and average response time fell from 4.35 to 0.51 seconds. Still, these are vendor figures from a single dataset with many rephrasings. That is why I find the table behind them more interesting: the more generous the threshold, the higher the hit rate, but the lower the accuracy of the answers served.

This leads to a simple rule of thumb. A semantic cache pays off when many people ask similar questions and the right answer does not depend on who is asking. Typical candidates are:

  • FAQ assistants for opening hours, rules, processes and terms
  • internal knowledge bots, for example about vacation requests or travel expenses
  • explanations of products, features and technical terms
  • classifications where the same text should always get the same category

Which questions never belong in the cache

The opposite list matters just as much. Personal questions such as “Where is my booking 4711815?” must never end up in the cache, otherwise the next person might see another customer’s data. Similarly, time-sensitive questions such as “Is a cargo bike still free today?” go stale within minutes. Multi-turn dialogs are tricky as well, because the meaning of a question depends on the conversation so far. For such cases, the example contains a rule that bypasses the cache on purpose.

The architecture with two cache levels

The solution consists of three levels that are asked one after the other. Level 1 is a perfectly normal exact cache: the question is normalized, which means lowercased and stripped of punctuation, and looked up as a hash key. This costs only a single read and no embedding. Only when nothing is found there does level 2 create an embedding and look for the nearest neighbor in the vector index. Finally, level 3 is the language model itself. Its answer is then stored in both cache levels, each with a time to live.

🎬 How a request flows through the semantic cache – this animated diagram only runs on my site: watch it on sascha-ms.de

All three levels run in a single Lambda function behind an HTTP API. As in the article on ElastiCache vector search, I run the cache with a public endpoint. As a result, the function needs no VPC and no NAT gateway, while the connection is still encrypted with TLS 1.3 and secured with IAM.

Avoiding false hits in a semantic cache

Now to the part that many tutorials skip. Take the questions “Do I need a helmet for the e-bike?” and “Do I need a helmet for the cargo bike?”. For an embedding model, these are almost identical sentences: same structure, same topic, just one word different. The answers, however, can differ considerably. A generous threshold would happily serve the wrong answer here.

Semantic cache: cosine distance scale with threshold 0.1 and a false hit, e-bike instead of cargo bike

Schematic: a threshold alone does not reliably separate similar questions with different answers.

The threshold alone does not solve this problem, because it is always a compromise for all questions at once. That is why I also rely on hard filters before the vector search. Valkey supports this directly: before the arrow => comes a filter on TAG fields, followed by the KNN search. In other words, only entries that match the filter exactly are searched. In the example, there are three such fields:

  • Topic such as “rental” or “prices”, which the user interface sends along
  • Knowledge version (kbv), a version label for your content, so that old answers no longer match after a price change
  • Bike type, which a small list of key terms and synonyms detects in the question

The last point in particular carries over to other domains. Almost every domain has a handful of terms that flip an answer: plan names, product variants, countries, contract types. If you treat these terms as an exact filter, the vector search can only find similar entries within the matching group. Admittedly, that costs a few hits, but it prevents exactly the embarrassing mix-ups.

Building a semantic cache step by step

The following nine steps take you from an empty account to the first answer from the cache. You will also find the complete code in the example repository, linked at the end of the article.

Prerequisites

  • An AWS account with permissions for ElastiCache, Lambda, API Gateway, IAM and Bedrock
  • AWS CLI version 2, configured for eu-central-1
  • Node.js 24 and npm 10 or newer
  • The Serverless Framework v4 (npm i -g serverless)
  • Access to Amazon Titan Text Embeddings V2 and a text model in Bedrock, Amazon Nova Lite in this example

Step 1: Create the ElastiCache Serverless cache

You create the cache once via the CLI. What matters are Valkey version 9 or later, the connection type public and the system-managed user group for IAM authentication. With --cache-usage-limits you cap storage and compute, which is a good safeguard against surprises in test accounts.

aws elasticache create-serverless-cache \
  --serverless-cache-name bike-station-cache \
  --engine valkey \
  --major-engine-version 9 \
  --connection-type public \
  --user-group-id default.iam-user-group \
  --cache-usage-limits "DataStorage={Maximum=1,Unit=GB},ECPUPerSecond={Maximum=5000}" \
  --region eu-central-1

# Query status and address (repeat until "available" appears)
aws elasticache describe-serverless-caches \
  --serverless-cache-name bike-station-cache \
  --query "ServerlessCaches[0].[Status,Endpoint.Address]" \
  --output text --region eu-central-1
Enter fullscreen mode Exit fullscreen mode

Note the address as CACHE_HOST. Also check which cross-region inference profiles your account offers for Nova Lite. You will need the profile ID as the model ID in a moment:

aws bedrock list-inference-profiles --region eu-central-1 \
  --type-equals SYSTEM_DEFINED \
  --query "inferenceProfileSummaries[?contains(inferenceProfileId, 'nova-lite')].inferenceProfileId" \
  --output text
Enter fullscreen mode Exit fullscreen mode

Step 2: Set up the project

As the client, I use Valkey GLIDE again. It comes with IAM authentication including automatic token renewal and has its own module for search commands. In addition, you need the Bedrock SDK for embeddings and answers, plus Vitest for the tests.

{
  "name": "bike-station-faq",
  "version": "1.0.0",
  "private": true,
  "type": "commonjs",
  "scripts": {
    "build": "tsc",
    "test": "vitest run",
    "setup-index": "tsx scripts/setup-index.ts",
    "calibrate": "tsx scripts/calibrate.ts",
    "deploy": "npm run build && serverless deploy --param=\"cacheHost=$CACHE_HOST\""
  },
  "dependencies": {
    "@aws-sdk/client-bedrock-runtime": "^3.1169.0",
    "@valkey/valkey-glide": "^2.6.0"
  },
  "devDependencies": {
    "@types/aws-lambda": "^8.10.164",
    "@types/node": "^24.19.2",
    "tsx": "^4.23.15",
    "typescript": "^7.0.2",
    "vitest": "^5.1.0"
  }
}
Enter fullscreen mode Exit fullscreen mode
{
  "compilerOptions": {
    "target": "ES2023",
    "module": "nodenext",
    "moduleResolution": "nodenext",
    "types": ["node"],
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "outDir": "dist",
    "rootDir": "."
  },
  "include": ["src", "scripts", "test"]
}
Enter fullscreen mode Exit fullscreen mode

I take the cache connection and the embedding almost unchanged from the vector search article. The only difference: the request timeout is set to 500 milliseconds. After all, a cache that takes longer than that no longer saves anything.

// src/valkey.ts
import { GlideClusterClient, ServiceType } from "@valkey/valkey-glide";

let pending: Promise<GlideClusterClient> | undefined;

/** One client per Lambda instance; GLIDE refreshes the IAM token itself. */
export function getClient(): Promise<GlideClusterClient> {
  pending ??= GlideClusterClient.createClient({
    addresses: [{ host: required("CACHE_HOST"), port: 6379 }],
    useTLS: true,
    requestTimeout: 500,
    credentials: {
      username: "default.iam-user",
      iamConfig: {
        clusterName: required("CACHE_NAME").toLowerCase(),
        service: ServiceType.Elasticache,
        region: required("AWS_REGION"),
      },
    },
  }).catch((error: unknown) => {
    pending = undefined; // reconnect on the next call
    throw error;
  });
  return pending;
}

export function required(name: string): string {
  const value = process.env[name];
  if (!value) throw new Error(`Environment variable ${name} is missing`);
  return value;
}
Enter fullscreen mode Exit fullscreen mode
// src/embed.ts
import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";

export const EMBEDDING_DIM = 512;
const bedrock = new BedrockRuntimeClient({});

/** Creates a normalized embedding with Titan Text Embeddings V2. */
export async function embed(text: string): Promise<number[]> {
  const response = await bedrock.send(
    new InvokeModelCommand({
      modelId: "amazon.titan-embed-text-v2:0",
      contentType: "application/json",
      accept: "application/json",
      body: JSON.stringify({ inputText: text, dimensions: EMBEDDING_DIM, normalize: true }),
    }),
  );
  const parsed = JSON.parse(new TextDecoder().decode(response.body)) as { embedding: number[] };
  return parsed.embedding;
}

/** Valkey expects FLOAT32 values as little-endian bytes. */
export function toVectorBuffer(vector: number[]): Buffer {
  const bytes = Buffer.alloc(vector.length * 4);
  vector.forEach((value, i) => bytes.writeFloatLE(value, i * 4));
  return bytes;
}
Enter fullscreen mode Exit fullscreen mode

Step 3: Create the vector index

The index watches all hashes with the prefix answer:. Furthermore, it contains the three filter fields as TAG and the embedding as a vector with 512 dimensions and cosine distance. Question and answer themselves don’t need to be indexed; they are only returned with the result.

// scripts/setup-index.ts
import { GlideFt } from "@valkey/valkey-glide";
import { getClient } from "../src/valkey";
import { EMBEDDING_DIM } from "../src/embed";
import { INDEX_NAME, KEY_PREFIX } from "../src/semantic-cache";

async function main(): Promise<void> {
  const client = await getClient();
  const existing = (await GlideFt.list(client)).map(String);
  if (existing.includes(INDEX_NAME)) {
    console.log(`Index ${INDEX_NAME} already exists`);
  } else {
    await GlideFt.create(
      client,
      INDEX_NAME,
      [
        { type: "TAG", name: "topic" },
        { type: "TAG", name: "kbv" },
        { type: "TAG", name: "bike" },
        {
          type: "VECTOR",
          name: "embedding",
          attributes: { algorithm: "HNSW", dimensions: EMBEDDING_DIM, distanceMetric: "COSINE", type: "FLOAT32" },
        },
      ],
      { dataType: "HASH", prefixes: [KEY_PREFIX] },
    );
    console.log(`Index ${INDEX_NAME} created`);
  }
  client.close();
}

main().catch((error: unknown) => {
  console.error(error);
  process.exit(1);
});
Enter fullscreen mode Exit fullscreen mode
npm install
export AWS_REGION=eu-central-1
export CACHE_NAME=bike-station-cache
export CACHE_HOST=bike-station-cache-abc123.public.serverless.euc1.cache.amazonaws.com
npm run setup-index
Enter fullscreen mode Exit fullscreen mode

You should see the line “Index idx:answers created”. On a second run, the script reports that the index already exists.

Step 4: The rules as pure functions

Every decision that can go wrong lives in one file without dependencies on AWS. This includes normalization, the rule for personal questions, bike type detection, the key for the exact level and the query for the vector search. Because tag values end up in the query language, assertSafeTag only lets lowercase letters and digits through. Therefore, nobody can inject their own filters with crafted input. One English-specific detail: compounds like “cargo bike” are often written as two words, so the function joins them before the lookup.

// src/policy.ts
import { createHash } from "node:crypto";

export const TOPICS = ["rental", "prices", "repair", "general"] as const;
export type Topic = (typeof TOPICS)[number];

/** Key terms that change an answer. They must match exactly. */
const BIKE_TYPES: Record<string, string> = {
  "e-bike": "ebike",
  ebike: "ebike",
  pedelec: "ebike",
  "cargo-bike": "cargo",
  cargobike: "cargo",
  "kids-bike": "kids",
  citybike: "city",
  "city-bike": "city",
  tandem: "tandem",
};

export function isTopic(value: unknown): value is Topic {
  return typeof value === "string" && (TOPICS as readonly string[]).includes(value);
}

/** Lowercase, drop punctuation, collapse whitespace. */
export function normalizeQuestion(question: string): string {
  return question
    .toLowerCase()
    .replace(/[^\p{L}\p{N}\s-]/gu, " ")
    .replace(/\s+/g, " ")
    .trim();
}

/**
 * Personal or time-sensitive questions never go into the cache:
 * booking numbers, email addresses, “my booking”, “today”, “right now” etc.
 */
export function isCacheable(question: string): boolean {
  const q = question.toLowerCase();
  if (/\d{5,}/.test(q)) return false; // booking or customer number
  if (/[\w.+-]+@[\w-]+\.[\w.]+/.test(q)) return false; // email address
  if (/\b(my|mine)\b/.test(q)) return false;
  if (/\b(today|tomorrow|right now|now|still free|available)\b/.test(q)) return false;
  return true;
}

/** Detects bike types; the result is stored as a TAG and must match exactly. */
export function detectBikeType(question: string): string {
  // English often writes compounds as two words: "cargo bike" becomes "cargo-bike".
  const words = normalizeQuestion(question)
    .replace(/\b(cargo|kids|city) bike\b/g, "$1-bike")
    .split(" ");
  const found = new Set<string>();
  for (const word of words) {
    const type = BIKE_TYPES[word];
    if (type) found.add(type);
  }
  return found.size === 0 ? "none" : [...found].sort().join("and");
}

/** Key for the exact level: same question in the same context. */
export function exactKey(scope: CacheScope, question: string): string {
  const hash = createHash("sha256").update(normalizeQuestion(question)).digest("hex");
  return `exact:${scope.topic}:${scope.kbVersion}:${hash}`;
}

export interface CacheScope {
  topic: Topic;
  kbVersion: string;
  bikeType: string;
}

/** Lowercase letters and digits only: these values end up in the query language. */
export function assertSafeTag(value: string): string {
  if (!/^[a-z0-9]{1,40}$/.test(value)) throw new Error(`Invalid tag value: ${value}`);
  return value;
}

/** Pre-filter on topic, knowledge version and bike type, then the nearest neighbor. */
export function buildLookupQuery(scope: CacheScope): string {
  const filter = [
    `@topic:{${assertSafeTag(scope.topic)}}`,
    `@kbv:{${assertSafeTag(scope.kbVersion)}}`,
    `@bike:{${assertSafeTag(scope.bikeType)}}`,
  ].join(" ");
  return `(${filter})=>[KNN 1 @embedding $vec AS distance]`;
}

/** Hit only if the cosine distance is small enough (0 = identical). */
export function isHit(distance: number | undefined, maxDistance: number): boolean {
  return distance !== undefined && Number.isFinite(distance) && distance <= maxDistance;
}

/** TTL with a random share so that not all entries expire at the same time. */
export function ttlWithJitter(baseSeconds: number, jitterRatio = 0.1, random = Math.random): number {
  const jitter = Math.floor(baseSeconds * jitterRatio * random());
  return baseSeconds - jitter;
}
Enter fullscreen mode Exit fullscreen mode

The rule isCacheable is deliberately strict and simple. Put differently, it would rather ask the model once too often than pass on personal data once too often. In a real project, I would not send such questions to a language model anyway, but to the booking system.

Step 5: Look up and store

The file for cache access has three functions. lookupExact reads the exact level, lookupSemantic runs the filtered KNN search with k = 1 and returns the distance as well. store writes both levels and sets the time to live. When a key expires, it also disappears from the index, so you don’t need to clean up anything yourself.

// src/semantic-cache.ts
import { randomUUID } from "node:crypto";
import { GlideFt, TimeUnit, type GlideClusterClient } from "@valkey/valkey-glide";
import { toVectorBuffer } from "./embed";
import { buildLookupQuery, exactKey, type CacheScope } from "./policy";

export const INDEX_NAME = "idx:answers";
export const KEY_PREFIX = "answer:";

export interface SemanticMatch {
  answer: string;
  question: string;
  distance: number;
}

/** Level 1: exactly the same question, no embedding at all. */
export async function lookupExact(client: GlideClusterClient, scope: CacheScope, question: string) {
  const value = await client.get(exactKey(scope, question));
  return value === null ? undefined : String(value);
}

/** Level 2: most similar stored question in the same context. */
export async function lookupSemantic(
  client: GlideClusterClient,
  scope: CacheScope,
  vector: number[],
): Promise<SemanticMatch | undefined> {
  const [, docs] = await GlideFt.search(client, INDEX_NAME, buildLookupQuery(scope), {
    params: [{ key: "vec", value: toVectorBuffer(vector) }],
    returnFields: [
      { fieldIdentifier: "question" },
      { fieldIdentifier: "answer" },
      { fieldIdentifier: "distance" },
    ],
    dialect: 2,
  });
  const doc = docs[0];
  if (!doc) return undefined;
  const fields = Object.fromEntries(doc.value.map((f) => [String(f.key), String(f.value)]));
  return { question: fields.question, answer: fields.answer, distance: Number(fields.distance) };
}

/** Stores both levels with the same time to live. */
export async function store(
  client: GlideClusterClient,
  scope: CacheScope,
  question: string,
  vector: number[],
  answer: string,
  ttlSeconds: number,
): Promise<void> {
  const key = `${KEY_PREFIX}${randomUUID()}`;
  await client.hset(key, [
    { field: "question", value: question },
    { field: "answer", value: answer },
    { field: "topic", value: scope.topic },
    { field: "kbv", value: scope.kbVersion },
    { field: "bike", value: scope.bikeType },
    { field: "embedding", value: toVectorBuffer(vector) },
  ]);
  await client.expire(key, ttlSeconds);
  await client.set(exactKey(scope, question), answer, {
    expiry: { type: TimeUnit.Seconds, count: ttlSeconds },
  });
}
Enter fullscreen mode Exit fullscreen mode

Step 6: Ask the language model

For the model call I use the Bedrock Converse API. It has the same format for all text models, so you can swap the model ID later without touching the code. Moreover, the low temperature produces factual, consistent answers, which is exactly right for a cache.

// src/answer.ts
import { BedrockRuntimeClient, ConverseCommand } from "@aws-sdk/client-bedrock-runtime";
import { required } from "./valkey";

const bedrock = new BedrockRuntimeClient({});

const SYSTEM_PROMPT = `You are the FAQ assistant of the fictional “Lindenau Bike Station”, a bike rental.
Answer briefly, kindly and only with general information.
Never mention data about individual customers and never make up prices.`;

export interface ModelAnswer {
  text: string;
  inputTokens: number;
  outputTokens: number;
}

/** Asks the language model via the Converse API. */
export async function askModel(question: string): Promise<ModelAnswer> {
  const response = await bedrock.send(
    new ConverseCommand({
      modelId: required("MODEL_ID"),
      system: [{ text: SYSTEM_PROMPT }],
      messages: [{ role: "user", content: [{ text: question }] }],
      inferenceConfig: { maxTokens: 400, temperature: 0.2 },
    }),
  );
  const text = (response.output?.message?.content ?? [])
    .map((block) => block.text ?? "")
    .join("")
    .trim();
  return {
    text,
    inputTokens: response.usage?.inputTokens ?? 0,
    outputTokens: response.usage?.outputTokens ?? 0,
  };
}
Enter fullscreen mode Exit fullscreen mode

Step 7: The Lambda handler and the metrics

The handler ties everything together: validate the input, pass personal questions straight through, then the exact level, the semantic level and finally the model. For every request, it writes a metric in Embedded Metric Format. That is a JSON log line which CloudWatch automatically turns into metrics, without any extra API call. Thus, you can later see per source how many requests came from the cache.

// src/metrics.ts
export type Source = "exact" | "semantic" | "model" | "no-cache";

/** Writes a metric in Embedded Metric Format; CloudWatch turns it into metrics. */
export function recordRequest(source: Source, distance?: number): void {
  const metrics: { Name: string; Unit: string }[] = [{ Name: "Requests", Unit: "Count" }];
  const values: Record<string, number> = { Requests: 1 };
  if (distance !== undefined && Number.isFinite(distance)) {
    metrics.push({ Name: "Distance", Unit: "None" });
    values.Distance = distance;
  }
  console.log(
    JSON.stringify({
      _aws: {
        Timestamp: Date.now(),
        CloudWatchMetrics: [{ Namespace: "BikeStation/FAQ", Dimensions: [["Source"]], Metrics: metrics }],
      },
      Source: source,
      ...values,
    }),
  );
}
Enter fullscreen mode Exit fullscreen mode
// src/handler.ts
import type { APIGatewayProxyEventV2, APIGatewayProxyResultV2 } from "aws-lambda";
import { getClient, required } from "./valkey";
import { embed } from "./embed";
import { askModel } from "./answer";
import { recordRequest } from "./metrics";
import { detectBikeType, isCacheable, isHit, isTopic, ttlWithJitter, type CacheScope } from "./policy";
import { lookupExact, lookupSemantic, store } from "./semantic-cache";

const MAX_DISTANCE = Number(process.env.MAX_DISTANCE ?? "0.1");
const TTL_SECONDS = Number(process.env.TTL_SECONDS ?? "82800"); // 23 hours

export async function handler(event: APIGatewayProxyEventV2): Promise<APIGatewayProxyResultV2> {
  let body: { question?: unknown; topic?: unknown };
  try {
    body = JSON.parse(event.body ?? "{}");
  } catch {
    return json(400, { message: "Body is not valid JSON" });
  }
  const question = typeof body.question === "string" ? body.question.trim() : "";
  if (question.length < 5 || question.length > 500) {
    return json(400, { message: "Field question is missing or too long (5–500 characters)" });
  }
  const topic = isTopic(body.topic) ? body.topic : "general";

  // Personal questions always go straight to the model (or better: to a business system).
  if (!isCacheable(question)) {
    const fresh = await askModel(question);
    recordRequest("no-cache");
    return json(200, { answer: fresh.text, source: "no-cache" });
  }

  const scope: CacheScope = { topic, kbVersion: required("KB_VERSION"), bikeType: detectBikeType(question) };
  const client = await getClient();

  const exact = await lookupExact(client, scope, question);
  if (exact !== undefined) {
    recordRequest("exact");
    return json(200, { answer: exact, source: "cache-exact" });
  }

  const vector = await embed(question);
  const match = await lookupSemantic(client, scope, vector);
  if (match && isHit(match.distance, MAX_DISTANCE)) {
    recordRequest("semantic", match.distance);
    return json(200, { answer: match.answer, source: "cache-semantic", distance: match.distance });
  }

  const fresh = await askModel(question);
  if (fresh.text.length > 0) {
    await store(client, scope, question, vector, fresh.text, ttlWithJitter(TTL_SECONDS));
  }
  recordRequest("model", match?.distance);
  return json(200, { answer: fresh.text, source: "model", nearestDistance: match?.distance ?? null });
}

function json(statusCode: number, body: unknown): APIGatewayProxyResultV2 {
  return { statusCode, headers: { "content-type": "application/json" }, body: JSON.stringify(body) };
}
Enter fullscreen mode Exit fullscreen mode

The time to live of 23 hours with a bit of randomness follows the recommendation in the AWS blog: if your content is updated daily, entries should expire before that and not all in the same minute. On a miss, I also return the distance of the nearest neighbor. This is especially valuable during calibration.

Step 8: Tests without AWS

I test the rules directly and the handler with fakes for cache, embedding and model. This lets you check in seconds that personal questions are never stored and that a distance that is too large really leads to the model.

// test/policy.test.ts
import { describe, expect, it } from "vitest";
import {
  buildLookupQuery,
  detectBikeType,
  exactKey,
  isCacheable,
  isHit,
  normalizeQuestion,
  ttlWithJitter,
} from "../src/policy";

const scope = { topic: "rental" as const, kbVersion: "v3", bikeType: "none" };

describe("normalizeQuestion", () => {
  it("unifies spelling and punctuation", () => {
    expect(normalizeQuestion("  How LONG can I?!  ")).toBe("how long can i");
  });
});

describe("exactKey", () => {
  it("returns the same key for equivalent spellings", () => {
    expect(exactKey(scope, "How long can I?")).toBe(exactKey(scope, "how long can i"));
  });

  it("separates by knowledge version", () => {
    expect(exactKey(scope, "Question")).not.toBe(exactKey({ ...scope, kbVersion: "v4" }, "Question"));
  });
});

describe("isCacheable", () => {
  it("caches general questions", () => {
    expect(isCacheable("Can I return the bike at another station?")).toBe(true);
  });

  it("skips personal and time-sensitive questions", () => {
    expect(isCacheable("Where is my booking 4711815?")).toBe(false);
    expect(isCacheable("Write to lea@example.org")).toBe(false);
    expect(isCacheable("Is a cargo bike still free today?")).toBe(false);
  });
});

describe("detectBikeType", () => {
  it("detects bike types including synonyms", () => {
    expect(detectBikeType("Do I need a helmet for the e-bike?")).toBe("ebike");
    expect(detectBikeType("Do you have a cargobike?")).toBe("cargo");
    expect(detectBikeType("Do I need a license for the cargo bike?")).toBe("cargo");
  });

  it("sorts several types in a stable order", () => {
    expect(detectBikeType("Tandem or pedelec?")).toBe("ebikeandtandem");
  });

  it("reports none when no type occurs", () => {
    expect(detectBikeType("How long can I rent?")).toBe("none");
  });
});

describe("buildLookupQuery", () => {
  it("filters on topic, knowledge version and bike type", () => {
    expect(buildLookupQuery(scope)).toBe(
      "(@topic:{rental} @kbv:{v3} @bike:{none})=>[KNN 1 @embedding $vec AS distance]",
    );
  });

  it("rejects injected characters", () => {
    expect(() => buildLookupQuery({ ...scope, kbVersion: "v3} | @topic:{prices" })).toThrow(/Invalid/);
  });
});

describe("isHit", () => {
  it("only counts small distances as hits", () => {
    expect(isHit(0.05, 0.1)).toBe(true);
    expect(isHit(0.1, 0.1)).toBe(true);
    expect(isHit(0.25, 0.1)).toBe(false);
    expect(isHit(Number.NaN, 0.1)).toBe(false);
    expect(isHit(undefined, 0.1)).toBe(false);
  });
});

describe("ttlWithJitter", () => {
  it("stays between 90 and 100 percent of the base", () => {
    expect(ttlWithJitter(1000, 0.1, () => 0)).toBe(1000);
    expect(ttlWithJitter(1000, 0.1, () => 0.999)).toBe(901);
  });
});
Enter fullscreen mode Exit fullscreen mode
// test/handler.test.ts
import { beforeEach, describe, expect, it, vi } from "vitest";
import type { APIGatewayProxyEventV2, APIGatewayProxyStructuredResultV2 } from "aws-lambda";
import { handler } from "../src/handler";

const fake = vi.hoisted(() => ({
  lookupExact: vi.fn(),
  lookupSemantic: vi.fn(),
  store: vi.fn(),
  askModel: vi.fn(),
  embed: vi.fn(),
}));

vi.mock("../src/valkey", () => ({ getClient: async () => ({}), required: () => "v1" }));
vi.mock("../src/embed", () => ({ embed: fake.embed }));
vi.mock("../src/answer", () => ({ askModel: fake.askModel }));
vi.mock("../src/metrics", () => ({ recordRequest: () => undefined }));
vi.mock("../src/semantic-cache", () => ({
  lookupExact: fake.lookupExact,
  lookupSemantic: fake.lookupSemantic,
  store: fake.store,
}));

function call(body: unknown) {
  const event = { body: JSON.stringify(body) } as APIGatewayProxyEventV2;
  return handler(event) as Promise<APIGatewayProxyStructuredResultV2>;
}

beforeEach(() => {
  vi.clearAllMocks();
  fake.embed.mockResolvedValue([0.1, 0.2]);
  fake.askModel.mockResolvedValue({ text: "Fresh from the model", inputTokens: 50, outputTokens: 20 });
});

describe("handler", () => {
  it("answers from the exact level without an embedding", async () => {
    fake.lookupExact.mockResolvedValue("From the cache");
    const res = await call({ question: "How long can I rent a bike?", topic: "rental" });
    expect(JSON.parse(res.body!).source).toBe("cache-exact");
    expect(fake.embed).not.toHaveBeenCalled();
  });

  it("uses a semantic hit below the threshold", async () => {
    fake.lookupExact.mockResolvedValue(undefined);
    fake.lookupSemantic.mockResolvedValue({ answer: "Up to 7 days", question: "x", distance: 0.04 });
    const res = await call({ question: "What is the maximum rental period?", topic: "rental" });
    expect(JSON.parse(res.body!).source).toBe("cache-semantic");
    expect(fake.askModel).not.toHaveBeenCalled();
  });

  it("asks the model when the distance is too large and stores the answer", async () => {
    fake.lookupExact.mockResolvedValue(undefined);
    fake.lookupSemantic.mockResolvedValue({ answer: "old", question: "x", distance: 0.3 });
    const res = await call({ question: "Do you have child seats?", topic: "rental" });
    expect(JSON.parse(res.body!).source).toBe("model");
    expect(fake.store).toHaveBeenCalledOnce();
  });

  it("bypasses the cache for personal questions", async () => {
    const res = await call({ question: "Where is my booking 4711815?", topic: "rental" });
    expect(JSON.parse(res.body!).source).toBe("no-cache");
    expect(fake.lookupExact).not.toHaveBeenCalled();
    expect(fake.store).not.toHaveBeenCalled();
  });

  it("rejects invalid JSON", async () => {
    const res = await handler({ body: "{broken" } as APIGatewayProxyEventV2);
    expect((res as APIGatewayProxyStructuredResultV2).statusCode).toBe(400);
  });
});
Enter fullscreen mode Exit fullscreen mode
npm test
Enter fullscreen mode Exit fullscreen mode

You should see 17 passing tests in two files.

Step 9: Deployment with the Serverless Framework

The serverless.yml gets by without a VPC block. Instead, the function’s role only needs to connect to the cache and call two models. For the inference profile, it needs permissions on the profile itself and on the base model in the destination regions, which is why there is a wildcard for the region. The built-in esbuild bundling stays off, because GLIDE ships a native library that has to live in the node_modules folder.

# serverless.yml
service: bike-station-faq
frameworkVersion: "4"

build:
  esbuild: false # we compile with tsc ourselves because GLIDE ships a native library

provider:
  name: aws
  runtime: nodejs24.x
  architecture: x86_64
  region: eu-central-1
  memorySize: 512
  timeout: 15
  environment:
    CACHE_NAME: ${param:cacheName, 'bike-station-cache'}
    CACHE_HOST: ${param:cacheHost}
    MODEL_ID: ${param:modelId, 'eu.amazon.nova-lite-v1:0'}
    KB_VERSION: ${param:kbVersion, 'v1'}
    MAX_DISTANCE: "0.1"
    TTL_SECONDS: "82800"
  iam:
    role:
      statements:
        - Effect: Allow
          Action: elasticache:Connect
          Resource:
            - arn:aws:elasticache:${aws:region}:${aws:accountId}:serverlesscache:${param:cacheName, 'bike-station-cache'}
            - arn:aws:elasticache:${aws:region}:${aws:accountId}:user:default.iam-user
        - Effect: Allow
          Action: bedrock:InvokeModel
          Resource:
            - arn:aws:bedrock:${aws:region}::foundation-model/amazon.titan-embed-text-v2:0
            - arn:aws:bedrock:${aws:region}:${aws:accountId}:inference-profile/${param:modelId, 'eu.amazon.nova-lite-v1:0'}
            - arn:aws:bedrock:*::foundation-model/amazon.nova-lite-v1:0

package:
  patterns:
    - "!**"
    - dist/src/**
    - node_modules/**
    - "!node_modules/.bin/**"
    - "!node_modules/typescript/**"
    - "!node_modules/@typescript/**"
    - "!node_modules/vitest/**"
    - "!node_modules/@vitest/**"
    - "!node_modules/vite/**"
    - "!node_modules/esbuild/**"
    - "!node_modules/@esbuild/**"
    - "!node_modules/tsx/**"
    - "!node_modules/rollup/**"
    - "!node_modules/@rollup/**"

functions:
  faq:
    handler: dist/src/handler.handler
    events:
      - httpApi:
          method: POST
          path: /question
Enter fullscreen mode Exit fullscreen mode
export CACHE_HOST=bike-station-cache-abc123.public.serverless.euc1.cache.amazonaws.com
npm run deploy

# First question: comes from the model and lands in the cache
curl -s -X POST "https://abc123.execute-api.eu-central-1.amazonaws.com/question" \
  -H "content-type: application/json" \
  -d '{"question":"How long can I rent a bike?","topic":"rental"}'

# Same meaning, different words: should come from the semantic cache
curl -s -X POST "https://abc123.execute-api.eu-central-1.amazonaws.com/question" \
  -H "content-type: application/json" \
  -d '{"question":"What is the maximum rental period for a bike?","topic":"rental"}'

# Clean up
serverless remove
aws elasticache delete-serverless-cache --serverless-cache-name bike-station-cache --region eu-central-1
Enter fullscreen mode Exit fullscreen mode

The first response contains "source":"model", the second ideally "source":"cache-semantic" including the distance. If the second question lands just above the threshold, the field nearestDistance shows it. That is exactly what the next section is about. If you work on a Mac, the same note as in the vector search article applies: build first, then install the Linux variant of the native packages and deploy, or run the deployment in a CI pipeline on Linux right away.

Calibrating the threshold

You should not simply adopt the value 0.1 from the example. Indeed, suitable distances depend on the embedding model, the dimension and your language. This is my approach: I collect 30 to 50 question pairs from real requests and mark for each pair whether the same answer would fit. The small script calculates the distance for each pair.

// scripts/calibrate.ts
import { embed } from "../src/embed";

/** Question pairs with your judgment: should the same answer fit? */
const pairs: { a: string; b: string; same: boolean }[] = [
  { a: "How long can I rent a bike?", b: "What is the maximum rental period?", same: true },
  { a: "Can I return the bike at another station?", b: "Is it possible to drop it off somewhere else?", same: true },
  { a: "Do you have child seats?", b: "Can I book a child seat as an extra?", same: true },
  { a: "What happens if I get a flat tire?", b: "What happens if I bring the bike back late?", same: false },
  { a: "Do I need a helmet for the e-bike?", b: "Do I need a license for the cargo bike?", same: false },
];

function cosineDistance(x: number[], y: number[]): number {
  let dot = 0;
  for (let i = 0; i < x.length; i++) dot += x[i] * y[i];
  return 1 - dot; // vectors are normalized
}

async function main(): Promise<void> {
  for (const pair of pairs) {
    const [va, vb] = await Promise.all([embed(pair.a), embed(pair.b)]);
    const d = cosineDistance(va, vb).toFixed(3);
    console.log(`${pair.same ? "same     " : "different"} ${d}  ${pair.a} | ${pair.b}`);
  }
}

main().catch((error: unknown) => {
  console.error(error);
  process.exit(1);
});
Enter fullscreen mode Exit fullscreen mode
npm run calibrate
Enter fullscreen mode Exit fullscreen mode

In the output, you look for the boundary. All “same” pairs should be below it, all “different” pairs above it. If the ranges overlap, you have two options: make the threshold stricter or, usually better, introduce another hard filter. Repeat the measurement whenever you switch the embedding model or the dimension. In that case you need a new index anyway.

Common mistakes and how to fix them

Symptom Cause Fix
Wrong answer with high confidence Threshold too generous or missing filter for a decisive term Add the key term as a TAG filter, recalibrate the threshold
Old answer after a price change Knowledge version not increased Increase KB_VERSION and redeploy; old entries expire via the TTL
Hit rate close to zero Threshold too strict, topic sent inconsistently or TTL too short Evaluate distances in the nearestDistance field, fix the topic in the user interface
WRONGPASS when connecting Cache name signed with uppercase letters or missing permission Use a lowercase name, check elasticache:Connect for cache and user
AccessDeniedException on Converse Permissions only on the profile, not on the base model in the destination regions Allow both ARNs, as shown in the serverless.yml
Answers with personal data in the cache Rule for personal questions does not apply Extend the rule, delete affected entries, increase the knowledge version

One point is missing from the table because it is not an error in the strict sense: if two identical questions arrive at the same time, both ask the model and store two nearly identical entries. Granted, that costs one extra call, but it does no harm. Only under very high load would I add a short lock here. I show what that looks like in my article Prevent a cache stampede. How to set time budgets for slow model calls is covered in the article Setting the Lambda timeout right.

What does the cache cost and when does it pay off?

There is no flat number for the cost, but there is a simple calculation. A hit on level 1 costs almost nothing. On level 2, a hit costs one embedding and one search. Every miss, in turn, costs embedding, search, model call and two writes. So the cache pays off when the saved model calls are worth more than the additional embeddings plus the base cost of the cache.

For the base cost of ElastiCache Serverless, the pricing page shows 0.084 US dollars per GB-hour for Valkey in US East, with a minimum of 100 MB billed. A nearly empty cache therefore comes to about 6 US dollars per month, plus ECPUs for every access. An entry with 512 dimensions takes at least 4 KB in the index, because the vector is held twice in memory, plus question and answer. Consequently, with a few thousand FAQ entries, storage stays small. On the other hand, embedding and model prices vary greatly by model and region. Look them up in the Bedrock price list and plug them into the calculation together with your measured hit rate. You can estimate the Lambda cost per request with my Lambda cost calculator.

My advice: first measure for a week without a cache how often similar questions really occur. After that, the metrics from step 7 give you the hit rate per level. If it stays below roughly ten percent, a token budget or a cheaper model is often the better lever.

My take

I consider the semantic cache one of the underrated patterns for working with language models. First of all, it is quick to build, needs no change to the model and has an immediate effect on cost and response time. Especially for teams putting their first AI feature into production, it is a pragmatic lever.

However, I disagree with the common claim that you just pick a threshold and are done. In fact, a semantic cache is a business decision, not a purely technical one. Which terms flip an answer, when content goes stale and which questions are personal is something the domain team knows better than any vector search. I would therefore always start the cache with a list of such terms, a knowledge version and a small calibration set. And I would only use it for general questions. As soon as an answer depends on the person, it does not belong in a shared store.

Not every request needs an instant answer: for large volumes without time pressure, Bedrock batch inference is worth a look, as it costs half the price for selected models.

FAQ

What is the difference between a semantic cache and prompt caching?

A semantic cache returns a stored answer for a similar question and skips the model call entirely. By contrast, prompt caching still calls the model but processes a repeated prompt prefix more cheaply and quickly.

Which threshold is right for a semantic cache?

That depends on embedding model, dimension, language and domain. Start strict, for example with a cosine distance of 0.1, and calibrate with 30 to 50 labeled question pairs from real requests.

Do I need a dedicated vector database for a semantic cache?

Not necessarily. ElastiCache Serverless for Valkey version 9 or later supports vector search with filters directly in the cache. For very large, rarely queried datasets, S3 Vectors or OpenSearch are the better places.

How do I remove outdated answers from the cache?

The simplest way is a time to live per entry plus a version label for your content. When you increase the version, old entries no longer match the filter and expire via the TTL.

Your next step

Take the logs of your AI feature from the past week and count how many questions repeat in substance. Then write down the five terms that completely change an answer in your domain. With these two lists, you will know whether a semantic cache pays off for you and which filters it needs.

I am curious: which terms would lead to false hits in your project? Feel free to share in the comments.

Next in the series

Previous part: Step Functions Distributed Map: Process CSV with Lambda · Next part: Prevent a cache stampede: Lambda, Valkey and Bedrock · All parts of the series “Serverless patterns in practice”

All code from this article, ready to run: smirowstanitzok/sascha-ms-examples

Sources

This post was created with AI assistance.


👉 Next step: Not sure which AWS service fits your use case? My AWS service finder asks a few questions and points you to the right one.

📬 Stay updated: New articles on serverless, AI and leading tech teams via RSS, browser push or the weekly newsletter (in German).

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en

Top comments (0)