I upload images of places, wrestlers, and general objects to the application. When uploading images of people, I want the Gemini model to generate alt text about their appearance, posture, and attire. When uploading images of places, I want the Gemini model to generate alt text with their official names, coordinates, and locations. To support this, I installed the @mediapipe/tasks-decision library, which uses the EmbeddingGemma 2 model (embeddinggemma-2-text-vision-440m-litert-lm) to classify the category of a picture (people, general, or place). The category decides which Firebase AI Logic GenerativeModel and prompt generate the text response.
Architecture & Mental Model
-
Local Model Cache & Classification: When a user uploads an image for analysis,
ImageClassifierServicechecks the browser'sCacheStorage(decision-model-v1) for the EmbeddingGemma 2 model, downloads and caches it viaHttpClienton a cache miss, and usesDecisionMakerwith the provided choices to determine the image's category (people,general, orplace). -
Category Configuration Lookup: The category selects the prompt and
AiModelOptions(optionalsystemInstruction, responseschema, andtools) fromCATEGORY_MODEL_CONFIGS. -
GenerativeModelLookup & Content Generation:VisionServicepassesoptionstoAiModelFactory.getModel(options)to reuse or create the matching Firebase AI LogicGenerativeModel, then callsgenerateContent([prompt, imagePart])with the category's prompt to generate the text response.
flowchart TD
IMG["Selected Image (File)"] --> PREP["preprocessImageForVision() -> imageBytes + base64"]
PREP --> CLS["ImageClassifierService.classifyImage(imageBytes)"]
CLS --> CACHE_CHK{"Local model in CacheStorage?"}
CACHE_CHK -- "Miss" --> DL["HttpClient GET -> cache.put() -> Blob URL"]
CACHE_CHK -- "Hit" --> BLOB["cache.match() -> Blob URL"]
DL & BLOB --> DM["DecisionMaker.evaluateChoice(imageBytes, IMAGE_CATEGORY_QUESTION)"]
DM --> CAT{"Best Category"}
CAT -- "people" --> CFG_P["CATEGORY_MODEL_CONFIGS['people']"]
CAT -- "general" --> CFG_G["CATEGORY_MODEL_CONFIGS['general']"]
CAT -- "place" --> CFG_L["CATEGORY_MODEL_CONFIGS['place']"]
CFG_P & CFG_G & CFG_L --> FAC["AiModelFactory.getModel(options)"]
FAC --> GEN["aiModel.generateContent([prompt, imagePart])"]
1. Installing @mediapipe/tasks-decision, Serving WASM Assets, and Adding decisionModelPath to Remote Config
1.1. Installing @mediapipe/tasks-decision
I installed the latest @mediapipe/tasks-decision package (1.2.3), which provides the API to evaluate the category of an image input.
npm install @mediapipe/tasks-decision
1.2. Serving WASM Assets in angular.json
The decision tasks require loading the WASM files at runtime. Instead of pointing to the WASM files on a CDN, I used the files installed in node_modules/@mediapipe/tasks-decision/wasm. Since the service cannot load files directly from node_modules in the browser, I configured angular.json to copy node_modules/@mediapipe/tasks-decision/wasm to /wasm (WASM_PATH = '/wasm') so the application can access them.
// angular.json (projects.ng-firebase-tts.architect.build.options.assets)
{
"glob": "**/*",
"input": "node_modules/@mediapipe/tasks-decision/wasm",
"output": "/wasm"
}
1.3. Adding decisionModelPath to Firebase Remote Config
I added the decisionModelPath parameter to Firebase Remote Config and public/remote-config-defaults.json, defaulting to https://huggingface.co/litert-community/embeddinggemma-2-text-vision-440m-litert-lm/resolve/main/embeddinggemma-2-text-vision-440m.litertlm for on-device image classification. I also added the decisionModelPath property to AppRemoteConfig, which ConfigService updates when fetching the model path dynamically from Firebase Remote Config.
// public/remote-config-defaults.json
{
"decisionModelPath": "https://huggingface.co/litert-community/embeddinggemma-2-text-vision-440m-litert-lm/resolve/main/embeddinggemma-2-text-vision-440m.litertlm",
...
}
// src/app/core/firebase/config.service.ts (decisionModelPath in #appConfig & fetchRemoteConfig)
#appConfig: AppRemoteConfig = {
...
decisionModelPath: remoteConfigDefaults.decisionModelPath,
};
...
this.#appConfig = {
...
decisionModelPath: getValue(rc, 'decisionModelPath').asString(),
};
2. Downloading, Caching, and Running the Local Model in ImageClassifierService
2.1. Class Fields and Lifecycle Cleanup
ImageClassifierService encapsulates the image-classification logic that VisionService uses in step one of the image-analysis pipeline:
-
#cachedModelPath: Stores the path of the active EmbeddingGemma 2 model so the service can detect whendecisionModelPathchanges in Firebase Remote Config. -
#decisionMakerPromise: Stores thePromise<DecisionMaker>so the service initializes the local model at most once per active model path. -
DestroyRef.onDestroy(): Registers a cleanup callback that callscloseActiveDecisionMaker()before the application shuts down to avoid a WASM memory leak.
// src/app/core/vision/image-classifier.service.ts (fields & lifecycle)
export class ImageClassifierService {
#window = inject(WINDOW);
#configService = inject(ConfigService);
#destroyRef = inject(DestroyRef);
#decisionMakerPromise: Promise<DecisionMaker> | null = null;
#cachedModelPath: string | null = null;
#http = inject(HttpClient);
constructor() {
this.#destroyRef.onDestroy(() => this.closeActiveDecisionMaker());
}
2.2. Lazy Initialization, Caching, and Prewarming (initializeDecisionMaker & resolveCachedModelUrl)
initializeDecisionMaker initializes the MediaPipe WASM runtime, loads the cached model, and prewarms the classification options:
-
Loading the WASM Runtime: Dynamically imports
DecisionMakerandFilesetResolverfrom@mediapipe/tasks-decisionand callsFilesetResolver.forDecisionTasks(WASM_PATH)to load the WASM files from/wasm. -
Retrieving the Cached Model (
this.resolveCachedModelUrl()): Opens thedecision-model-v1(DECISION_MODEL_CACHE_NAME) bucket inCacheStorage, deletes any stale model URLs, and checks for a cached response matchingmodelAssetPath:-
Cache Miss: Makes an HTTP request to download the model as a
Blob, stores it in theCacheStoragebucket, and creates ablob:URL. -
Cache Hit: Creates and returns a
blob:URL from the cached response immediately.
-
Cache Miss: Makes an HTTP request to download the model as a
-
Creating and Prewarming the Session: Calls
DecisionMaker.createFromOptions()to create theDecisionMakerinstance with the resolveddelegate('GPU'or'CPU'),backendType: 3(the LiteRT-LM backend), andMAX_NUM_TOKENS, then callsprewarmChoice(IMAGE_CATEGORY_QUESTION)to prewarm the WASM session with thegeneral,people, andplaceoptions. Finally,URL.revokeObjectURL(resolvedModelPath)revokes the temporaryblob:URL once initialization finishes.
// src/app/core/vision/image-classifier.service.ts (initializeDecisionMaker & resolveCachedModelUrl)
private async initializeDecisionMaker(modelAssetPath: string, delegate: 'GPU' | 'CPU'): Promise<DecisionMaker> {
const { DecisionMaker, FilesetResolver } = await import('@mediapipe/tasks-decision');
const wasmFileset = await FilesetResolver.forDecisionTasks(WASM_PATH);
const resolvedModelPath = await this.resolveCachedModelUrl(modelAssetPath);
try {
const decisionMaker = await DecisionMaker.createFromOptions(wasmFileset, {
baseOptions: {
modelAssetPath: resolvedModelPath,
delegate,
},
backendType: 3,
maxNumTokens: MAX_NUM_TOKENS,
});
await decisionMaker.prewarmChoice(IMAGE_CATEGORY_QUESTION);
return decisionMaker;
} finally {
if (resolvedModelPath.startsWith('blob:')) {
URL.revokeObjectURL(resolvedModelPath);
}
}
}
private async resolveCachedModelUrl(modelAssetPath: string): Promise<string> {
const cacheStorage = this.#window?.caches;
if (!cacheStorage) {
return modelAssetPath;
}
try {
const cache = await cacheStorage.open(DECISION_MODEL_CACHE_NAME);
const keys = await cache.keys();
await Promise.all(keys.filter((req) => req.url !== modelAssetPath).map((req) => cache.delete(req)));
let response = await cache.match(modelAssetPath);
if (!response) {
const httpResponse = await firstValueFrom(
this.#http.get(modelAssetPath, { observe: 'response', responseType: 'blob' }),
);
const downloadedBlob = httpResponse.body;
if (!downloadedBlob) {
return modelAssetPath;
}
response = new Response(downloadedBlob, {
status: httpResponse.status,
headers: {
'Content-Type': httpResponse.headers.get('Content-Type') ?? 'application/octet-stream',
},
});
await cache.put(modelAssetPath, response.clone());
}
const blob = await response.blob();
return URL.createObjectURL(blob);
} catch {
return modelAssetPath;
}
}

Caption: The downloaded embeddinggemma-2-text-vision-440m-litert-lm model stored inside the browser's decision-model-v1 CacheStorage.
2.3. Reusing the Instance and Freeing WASM Memory (getValidDecisionMaker & closeActiveDecisionMaker)
-
Checking the Active Model Path (
getValidDecisionMaker()): Compares#cachedModelPathagainstdecisionModelPathin Firebase Remote Config and returns the existing#decisionMakerPromisewhen unchanged. -
Refreshing on Stale Path: Closes the old
DecisionMakerwhen#cachedModelPathis stale, updates#cachedModelPath, and callsinitializeDecisionMaker()to construct a newPromise<DecisionMaker>. -
Releasing WASM Memory (
closeActiveDecisionMaker()): Awaitsthis.#decisionMakerPromise, callsdecisionMaker.close()to free the WASM engine and model memory, and resets#decisionMakerPromiseand#cachedModelPathtonull.
// src/app/core/vision/image-classifier.service.ts (getValidDecisionMaker & closeActiveDecisionMaker)
private async getValidDecisionMaker(delegate: 'GPU' | 'CPU'): Promise<DecisionMaker> {
const modelAssetPath = this.#configService.appConfig.decisionModelPath;
if (this.#decisionMakerPromise && this.#cachedModelPath !== modelAssetPath) {
await this.closeActiveDecisionMaker();
}
if (!this.#decisionMakerPromise) {
this.#cachedModelPath = modelAssetPath;
this.#decisionMakerPromise = this.initializeDecisionMaker(modelAssetPath, delegate).catch((error) => {
this.#decisionMakerPromise = null;
this.#cachedModelPath = null;
throw error;
});
}
return this.#decisionMakerPromise;
}
private async closeActiveDecisionMaker(): Promise<void> {
if (!this.#decisionMakerPromise) {
return;
}
try {
const decisionMaker = await this.#decisionMakerPromise;
decisionMaker.close();
} catch {
// Ignore initialization or WASM teardown errors during cleanup
} finally {
this.#decisionMakerPromise = null;
this.#cachedModelPath = null;
}
}
2.4. Defining the Choice Question (IMAGE_CATEGORY_QUESTION)
Before calling DecisionMaker, I define the classification rubric and token budget in image-classifier.const.ts:
-
Three Balanced Options (
options): Maps the image toplace,people, orgeneralusing concise, parallel descriptions so no single category dominates by wording length alone. -
Tie-Breaker Rule (
instructions): Instructs the model to selectplacewhen an outdoor landscape, monument, or landmark appears behind people, andpeopleotherwise when a human or masked figure is visible. -
Image Input Prior (
normalizePrior: false): SetsnormalizePriortofalsebecause the input is an image ({ imageBytes }) rather than text; null-context prior normalization is intended for text-only choice prompts. -
Token Budget (
MAX_NUM_TOKENS = 512): Limits the context window to512tokens because EmbeddingGemma 2 maps a single image to 280 soft tokens by default, leaving ~232 tokens for the shortcontext,instructions, and threeoptionswithout allocating the default4096-token window in browser memory. -
Deterministic Scoring (
temperature: 0.05): Keeps probability scoring near-deterministic across repeated runs on the same photo.
// src/app/core/vision/image-classifier.const.ts
import type { ChoiceQuestion } from '@mediapipe/tasks-decision';
export const WASM_PATH = '/wasm';
export const MAX_NUM_TOKENS = 512;
export const DECISION_MODEL_CACHE_NAME = 'decision-model-v1';
export const IMAGE_CATEGORY_QUESTION: ChoiceQuestion = {
context: 'Analyze the uploaded image for accessibility alt-text routing.',
instructions:
'Select the primary subject category of this image. If an outdoor landscape, architecture, monument, landmark, or travel destination is visible behind people, select place. Otherwise, if any person, masked figure, or group of people is visible, select people.',
options: [
{
label: 'place',
description: 'An outdoor landscape, natural scenery, architecture, monument, landmark, or travel destination.',
},
{
label: 'people',
description: 'A person, portrait, face, masked or costumed figure, performer, or group of people.',
},
{
label: 'general',
description: 'A close-up of food, drink, animal, pet, vehicle, or household object.',
},
],
temperature: 0.05,
normalizePrior: false,
};
2.5. Resolving the Execution Delegate (resolveDelegate)
-
Checking WebGPU Support: Checks whether the browser supports WebGPU (
'gpu' in this.#window.navigator) and returns'GPU'when available. -
Skipping CPU for
.litertlmModels: Returnsundefinedwhen WebGPU is unavailable and the model path ends with.litertlm(since.litertlmvision models require WebGPU, causingclassifyImage()to fall back to'general'). -
CPU Fallback: Returns
'CPU'when WebGPU is unavailable and the model path does not end with.litertlm.
// src/app/core/vision/image-classifier.service.ts (resolveDelegate)
private resolveDelegate(): 'GPU' | 'CPU' | undefined {
const hasWebGpu = Boolean(this.#window?.navigator && 'gpu' in this.#window.navigator);
if (hasWebGpu) {
return 'GPU';
}
if (this.#configService.appConfig.decisionModelPath.endsWith('.litertlm')) {
return undefined;
}
return 'CPU';
}
2.6. Classifying the Image with Safe Fallbacks (classifyImage)
-
Evaluating Category Probabilities: Calls
decisionMaker.evaluateChoice()to evaluate{ imageBytes }againstIMAGE_CATEGORY_QUESTIONand return the probabilities of thegeneral,people, andplacecategories. - Selecting the Winner: Iterates over the returned entries and returns the category with the highest probability.
-
Defaulting to
'general': Returns'general'whenimageBytesis empty,delegateisundefined, no valid probability is returned, or an error occurs.
// src/app/core/vision/image-classifier.service.ts (classifyImage)
async classifyImage(imageBytes: Uint8Array): Promise<ImageSubjectCategory> {
if (imageBytes.byteLength === 0) {
return 'general';
}
const delegate = this.resolveDelegate();
if (!delegate) {
return 'general';
}
try {
const decisionMaker = await this.getValidDecisionMaker(delegate);
const { probabilities } = await decisionMaker.evaluateChoice({ imageBytes }, IMAGE_CATEGORY_QUESTION);
let bestCategory: ImageSubjectCategory = 'general';
let highestProbability = -1;
for (const [key, probability] of Object.entries(probabilities)) {
if (probability > highestProbability && (key === 'place' || key === 'people' || key === 'general')) {
highestProbability = probability;
bestCategory = key;
}
}
return bestCategory;
} catch (error) {
console.warn('[ImageClassifierService] On-device classification failed, falling back to general:', error);
return 'general';
}
}
3. Prompts, System Instructions, and Tools for Each Generative Model
-
Category Configuration Map (
CATEGORY_MODEL_CONFIGS): Maps eachDecisionMakercategory (general,people,place) to itspromptandAiModelOptions(optionalsystemInstruction, responseschema, andtools). -
Domain-Specific Personas (
systemInstruction): OmitssystemInstructionforgeneral, while providingpeopleSystemInstructionandplaceSystemInstructionto define the personas and verification rules of thepeopleandplaceGenerativeModelinstances.
| Category | systemInstruction |
prompt |
tools |
|---|---|---|---|
general |
None (uses prompt instructions directly) | altTextPrompt |
[{ googleSearch: {} }] |
people |
peopleSystemInstruction (strict anti-guessing rule for faces/masks) |
peopleAltTextPrompt |
[{ googleSearch: {} }] |
place |
placeSystemInstruction (landmark & coordinate verification, no fabrication) |
placeAltTextPrompt |
[{ googleSearch: {} }] |
// src/app/core/vision/vision.const.ts
export const CATEGORY_MODEL_CONFIGS: Record<ImageSubjectCategory, CategoryModelConfig> = {
general: {
options: {
schema: ImageAnalysisSchema,
tools: [{ googleSearch: {} }],
},
prompt: altTextPrompt,
},
people: {
options: {
schema: ImageAnalysisSchema,
tools: [{ googleSearch: {} }],
systemInstruction: peopleSystemInstruction,
},
prompt: peopleAltTextPrompt,
},
place: {
options: {
schema: ImageAnalysisSchema,
tools: [{ googleSearch: {} }],
systemInstruction: placeSystemInstruction,
},
prompt: placeAltTextPrompt,
},
};
Full prompt and systemInstruction strings: src/app/core/vision/vision.const.ts
4. Classifying Images and Generating Alt Text in VisionService
4.1. Classifying and Selecting the Model Config in VisionService
-
Injecting the Classifier: Injects
ImageClassifierServiceintoVisionServiceand assigns it tothis.#imageClassifierService. -
Classifying the Image: Calls
this.#imageClassifierService.classifyImage(imageBytes)to determine the category of the preprocessedimageBytes(Uint8Array). -
Selecting the Category Config: Indexes
CATEGORY_MODEL_CONFIGSbycategoryto retrieve thepromptandGenerativeModeloptions. -
Generating the Alt Text: Passes
optionstothis.#aiModelFactory.getModel(options)to obtain aGenerativeModelinstance, then callsaiModel.generateContent([prompt, imagePart])to generate the text response.
// src/app/core/vision/vision.service.ts (analyzeImage)
readonly #imageClassifierService = inject(ImageClassifierService);
async analyzeImage(image: File): Promise<ImageAnalysisResponse> {
if (!image) {
throw Error('image is required to generate texts.');
}
const { data, mimeType, optimizationMetrics, imageBytes } = await preprocessImageForVision(image, this.#window);
const category = await this.#imageClassifierService.classifyImage(imageBytes);
const { options, prompt } = CATEGORY_MODEL_CONFIGS[category] || CATEGORY_MODEL_CONFIGS['general'];
const imagePart: InlineDataPart = {
type: 'inlineData',
inlineData: {
data,
mimeType,
},
};
const aiModel = await this.#aiModelFactory.getModel(options);
const result = await aiModel.generateContent([prompt, imagePart]);
...
4.2. Generated Alt-Text Results for people, general, and place
I uploaded an image of six Japanese wrestlers, and the classifier returned the people category. The people system instruction tells the model to use the Google Search tool to verify visible text clues and find a real-world fact based on the generated tags. The people prompt asks the model to include a person's name in the alt text only when the name is printed in the image or 100% certain and otherwise to describe the person's features, posture, and attire.

Caption: Output for an image of six Japanese wrestlers when classified as people.
I uploaded an image of a butterfly, and the classifier returned the general category. The general prompt (altTextPrompt) generated the alt text and tags and used the Google Search tool to verify a real-world fact based on the tags.

Caption: Output for an image of a butterfly when classified as general.
I uploaded an image of Kek Lok Si Temple in Penang, Malaysia, and analyzed the image to generate the alt text and tags. DecisionMaker classified the image input as place and used the place prompt (placeAltTextPrompt) to generate the location name, coordinates, and architectural features without fabricating unverified coordinates. The place system instruction (placeSystemInstruction) directs the model to use the Google Search tool to verify the official place name and coordinates and to find a real-world fact based on the generated tags.

Caption: Output for Kek Lok Si Temple in Penang, Malaysia when classified as place.
Takeaways
-
Two-Tier Model Caching: Storing the EmbeddingGemma 2 model in
CacheStorage(decision-model-v1) avoids re-downloading the.litertlmfile across browser reloads, while#decisionMakerPromiseandAiModelFactoryreuse initialized model instances in memory. -
Lazy
GenerativeModelConstruction:VisionServiceonly creates theGenerativeModelfor a category (people,general, orplace) when an image is classified into that category, avoiding unnecessary model creation at startup. -
Smaller Initial Bundle (
main.js): Dynamically callingawait import('@mediapipe/tasks-decision')insideinitializeDecisionMaker()keeps the MediaPipe library out of the initialmain.jsbundle until the user analyzes an image. -
Explicit WASM Memory Cleanup: Calling
URL.revokeObjectURL()afterDecisionMaker.createFromOptions()anddecisionMaker.close()insideDestroyRef.onDestroy()prevents browser memory leaks from blob URLs and WASM/WebGPU instances.
Top comments (0)