DEV Community

Connie Leung for Google Developer Experts

Posted on AI-assisted

Stop Using One Prompt for Every Photo! Local Image Classification Before Calling Firebase AI Logic GenerativeModel

I upload images of places, wrestlers, and general objects to the application. When uploading images of people, I want the Gemini model to generate alt text about their appearance, posture, and attire. When uploading images of places, I want the Gemini model to generate alt text with their official names, coordinates, and locations. To support this, I installed the @mediapipe/tasks-decision library, which uses the EmbeddingGemma 2 model (embeddinggemma-2-text-vision-440m-litert-lm) to classify the category of a picture (people, general, or place). The category decides which Firebase AI Logic GenerativeModel and prompt generate the text response.


Architecture & Mental Model

  • Local Model Cache & Classification: When a user uploads an image for analysis, ImageClassifierService checks the browser's CacheStorage (decision-model-v1) for the EmbeddingGemma 2 model, downloads and caches it via HttpClient on a cache miss, and uses DecisionMaker with the provided choices to determine the image's category (people, general, or place).
  • Category Configuration Lookup: The category selects the prompt and AiModelOptions (optional systemInstruction, response schema, and tools) from CATEGORY_MODEL_CONFIGS.
  • GenerativeModel Lookup & Content Generation: VisionService passes options to AiModelFactory.getModel(options) to reuse or create the matching Firebase AI Logic GenerativeModel, then calls generateContent([prompt, imagePart]) with the category's prompt to generate the text response.
flowchart TD
    IMG["Selected Image (File)"] --> PREP["preprocessImageForVision() -> imageBytes + base64"]
    PREP --> CLS["ImageClassifierService.classifyImage(imageBytes)"]
    CLS --> CACHE_CHK{"Local model in CacheStorage?"}
    CACHE_CHK -- "Miss" --> DL["HttpClient GET -> cache.put() -> Blob URL"]
    CACHE_CHK -- "Hit" --> BLOB["cache.match() -> Blob URL"]
    DL & BLOB --> DM["DecisionMaker.evaluateChoice(imageBytes, IMAGE_CATEGORY_QUESTION)"]
    DM --> CAT{"Best Category"}
    CAT -- "people" --> CFG_P["CATEGORY_MODEL_CONFIGS['people']"]
    CAT -- "general" --> CFG_G["CATEGORY_MODEL_CONFIGS['general']"]
    CAT -- "place" --> CFG_L["CATEGORY_MODEL_CONFIGS['place']"]
    CFG_P & CFG_G & CFG_L --> FAC["AiModelFactory.getModel(options)"]
    FAC --> GEN["aiModel.generateContent([prompt, imagePart])"]

1. Installing @mediapipe/tasks-decision, Serving WASM Assets, and Adding decisionModelPath to Remote Config

1.1. Installing @mediapipe/tasks-decision

I installed the latest @mediapipe/tasks-decision package (1.2.3), which provides the API to evaluate the category of an image input.

npm install @mediapipe/tasks-decision
Enter fullscreen mode Exit fullscreen mode

1.2. Serving WASM Assets in angular.json

The decision tasks require loading the WASM files at runtime. Instead of pointing to the WASM files on a CDN, I used the files installed in node_modules/@mediapipe/tasks-decision/wasm. Since the service cannot load files directly from node_modules in the browser, I configured angular.json to copy node_modules/@mediapipe/tasks-decision/wasm to /wasm (WASM_PATH = '/wasm') so the application can access them.

// angular.json (projects.ng-firebase-tts.architect.build.options.assets)
{
  "glob": "**/*",
  "input": "node_modules/@mediapipe/tasks-decision/wasm",
  "output": "/wasm"
}
Enter fullscreen mode Exit fullscreen mode

1.3. Adding decisionModelPath to Firebase Remote Config

I added the decisionModelPath parameter to Firebase Remote Config and public/remote-config-defaults.json, defaulting to https://huggingface.co/litert-community/embeddinggemma-2-text-vision-440m-litert-lm/resolve/main/embeddinggemma-2-text-vision-440m.litertlm for on-device image classification. I also added the decisionModelPath property to AppRemoteConfig, which ConfigService updates when fetching the model path dynamically from Firebase Remote Config.

// public/remote-config-defaults.json
{
  "decisionModelPath": "https://huggingface.co/litert-community/embeddinggemma-2-text-vision-440m-litert-lm/resolve/main/embeddinggemma-2-text-vision-440m.litertlm",
  ...
}
Enter fullscreen mode Exit fullscreen mode
// src/app/core/firebase/config.service.ts (decisionModelPath in #appConfig & fetchRemoteConfig)
  #appConfig: AppRemoteConfig = {
    ...
    decisionModelPath: remoteConfigDefaults.decisionModelPath,
  };
  ...
  this.#appConfig = {
    ...
    decisionModelPath: getValue(rc, 'decisionModelPath').asString(),
  };
Enter fullscreen mode Exit fullscreen mode

2. Downloading, Caching, and Running the Local Model in ImageClassifierService

2.1. Class Fields and Lifecycle Cleanup

ImageClassifierService encapsulates the image-classification logic that VisionService uses in step one of the image-analysis pipeline:

  • #cachedModelPath: Stores the path of the active EmbeddingGemma 2 model so the service can detect when decisionModelPath changes in Firebase Remote Config.
  • #decisionMakerPromise: Stores the Promise<DecisionMaker> so the service initializes the local model at most once per active model path.
  • DestroyRef.onDestroy(): Registers a cleanup callback that calls closeActiveDecisionMaker() before the application shuts down to avoid a WASM memory leak.
// src/app/core/vision/image-classifier.service.ts (fields & lifecycle)
export class ImageClassifierService {
  #window = inject(WINDOW);
  #configService = inject(ConfigService);
  #destroyRef = inject(DestroyRef);
  #decisionMakerPromise: Promise<DecisionMaker> | null = null;
  #cachedModelPath: string | null = null;
  #http = inject(HttpClient);

  constructor() {
    this.#destroyRef.onDestroy(() => this.closeActiveDecisionMaker());
  }
Enter fullscreen mode Exit fullscreen mode

2.2. Lazy Initialization, Caching, and Prewarming (initializeDecisionMaker & resolveCachedModelUrl)

initializeDecisionMaker initializes the MediaPipe WASM runtime, loads the cached model, and prewarms the classification options:

  • Loading the WASM Runtime: Dynamically imports DecisionMaker and FilesetResolver from @mediapipe/tasks-decision and calls FilesetResolver.forDecisionTasks(WASM_PATH) to load the WASM files from /wasm.
  • Retrieving the Cached Model (this.resolveCachedModelUrl()): Opens the decision-model-v1 (DECISION_MODEL_CACHE_NAME) bucket in CacheStorage, deletes any stale model URLs, and checks for a cached response matching modelAssetPath:
    • Cache Miss: Makes an HTTP request to download the model as a Blob, stores it in the CacheStorage bucket, and creates a blob: URL.
    • Cache Hit: Creates and returns a blob: URL from the cached response immediately.
  • Creating and Prewarming the Session: Calls DecisionMaker.createFromOptions() to create the DecisionMaker instance with the resolved delegate ('GPU' or 'CPU'), backendType: 3 (the LiteRT-LM backend), and MAX_NUM_TOKENS, then calls prewarmChoice(IMAGE_CATEGORY_QUESTION) to prewarm the WASM session with the general, people, and place options. Finally, URL.revokeObjectURL(resolvedModelPath) revokes the temporary blob: URL once initialization finishes.
// src/app/core/vision/image-classifier.service.ts (initializeDecisionMaker & resolveCachedModelUrl)
  private async initializeDecisionMaker(modelAssetPath: string, delegate: 'GPU' | 'CPU'): Promise<DecisionMaker> {
    const { DecisionMaker, FilesetResolver } = await import('@mediapipe/tasks-decision');
    const wasmFileset = await FilesetResolver.forDecisionTasks(WASM_PATH);
    const resolvedModelPath = await this.resolveCachedModelUrl(modelAssetPath);

    try {
      const decisionMaker = await DecisionMaker.createFromOptions(wasmFileset, {
        baseOptions: {
          modelAssetPath: resolvedModelPath,
          delegate,
        },
        backendType: 3,
        maxNumTokens: MAX_NUM_TOKENS,
      });

      await decisionMaker.prewarmChoice(IMAGE_CATEGORY_QUESTION);

      return decisionMaker;
    } finally {
      if (resolvedModelPath.startsWith('blob:')) {
        URL.revokeObjectURL(resolvedModelPath);
      }
    }
  }

  private async resolveCachedModelUrl(modelAssetPath: string): Promise<string> {
    const cacheStorage = this.#window?.caches;
    if (!cacheStorage) {
      return modelAssetPath;
    }

    try {
      const cache = await cacheStorage.open(DECISION_MODEL_CACHE_NAME);
      const keys = await cache.keys();
      await Promise.all(keys.filter((req) => req.url !== modelAssetPath).map((req) => cache.delete(req)));

      let response = await cache.match(modelAssetPath);
      if (!response) {
        const httpResponse = await firstValueFrom(
          this.#http.get(modelAssetPath, { observe: 'response', responseType: 'blob' }),
        );
        const downloadedBlob = httpResponse.body;
        if (!downloadedBlob) {
          return modelAssetPath;
        }
        response = new Response(downloadedBlob, {
          status: httpResponse.status,
          headers: {
            'Content-Type': httpResponse.headers.get('Content-Type') ?? 'application/octet-stream',
          },
        });
        await cache.put(modelAssetPath, response.clone());
      }

      const blob = await response.blob();
      return URL.createObjectURL(blob);
    } catch {
      return modelAssetPath;
    }
  }
Enter fullscreen mode Exit fullscreen mode

Browser Cache Storage Showing Cached EmbeddingGemma 2 Model
Caption: The downloaded embeddinggemma-2-text-vision-440m-litert-lm model stored inside the browser's decision-model-v1 CacheStorage.

2.3. Reusing the Instance and Freeing WASM Memory (getValidDecisionMaker & closeActiveDecisionMaker)

  • Checking the Active Model Path (getValidDecisionMaker()): Compares #cachedModelPath against decisionModelPath in Firebase Remote Config and returns the existing #decisionMakerPromise when unchanged.
  • Refreshing on Stale Path: Closes the old DecisionMaker when #cachedModelPath is stale, updates #cachedModelPath, and calls initializeDecisionMaker() to construct a new Promise<DecisionMaker>.
  • Releasing WASM Memory (closeActiveDecisionMaker()): Awaits this.#decisionMakerPromise, calls decisionMaker.close() to free the WASM engine and model memory, and resets #decisionMakerPromise and #cachedModelPath to null.
// src/app/core/vision/image-classifier.service.ts (getValidDecisionMaker & closeActiveDecisionMaker)
  private async getValidDecisionMaker(delegate: 'GPU' | 'CPU'): Promise<DecisionMaker> {
    const modelAssetPath = this.#configService.appConfig.decisionModelPath;

    if (this.#decisionMakerPromise && this.#cachedModelPath !== modelAssetPath) {
      await this.closeActiveDecisionMaker();
    }

    if (!this.#decisionMakerPromise) {
      this.#cachedModelPath = modelAssetPath;
      this.#decisionMakerPromise = this.initializeDecisionMaker(modelAssetPath, delegate).catch((error) => {
        this.#decisionMakerPromise = null;
        this.#cachedModelPath = null;
        throw error;
      });
    }

    return this.#decisionMakerPromise;
  }

  private async closeActiveDecisionMaker(): Promise<void> {
    if (!this.#decisionMakerPromise) {
      return;
    }

    try {
      const decisionMaker = await this.#decisionMakerPromise;
      decisionMaker.close();
    } catch {
      // Ignore initialization or WASM teardown errors during cleanup
    } finally {
      this.#decisionMakerPromise = null;
      this.#cachedModelPath = null;
    }
  }
Enter fullscreen mode Exit fullscreen mode

2.4. Defining the Choice Question (IMAGE_CATEGORY_QUESTION)

Before calling DecisionMaker, I define the classification rubric and token budget in image-classifier.const.ts:

  • Three Balanced Options (options): Maps the image to place, people, or general using concise, parallel descriptions so no single category dominates by wording length alone.
  • Tie-Breaker Rule (instructions): Instructs the model to select place when an outdoor landscape, monument, or landmark appears behind people, and people otherwise when a human or masked figure is visible.
  • Image Input Prior (normalizePrior: false): Sets normalizePrior to false because the input is an image ({ imageBytes }) rather than text; null-context prior normalization is intended for text-only choice prompts.
  • Token Budget (MAX_NUM_TOKENS = 512): Limits the context window to 512 tokens because EmbeddingGemma 2 maps a single image to 280 soft tokens by default, leaving ~232 tokens for the short context, instructions, and three options without allocating the default 4096-token window in browser memory.
  • Deterministic Scoring (temperature: 0.05): Keeps probability scoring near-deterministic across repeated runs on the same photo.
// src/app/core/vision/image-classifier.const.ts
import type { ChoiceQuestion } from '@mediapipe/tasks-decision';

export const WASM_PATH = '/wasm';
export const MAX_NUM_TOKENS = 512;
export const DECISION_MODEL_CACHE_NAME = 'decision-model-v1';

export const IMAGE_CATEGORY_QUESTION: ChoiceQuestion = {
  context: 'Analyze the uploaded image for accessibility alt-text routing.',
  instructions:
    'Select the primary subject category of this image. If an outdoor landscape, architecture, monument, landmark, or travel destination is visible behind people, select place. Otherwise, if any person, masked figure, or group of people is visible, select people.',
  options: [
    {
      label: 'place',
      description: 'An outdoor landscape, natural scenery, architecture, monument, landmark, or travel destination.',
    },
    {
      label: 'people',
      description: 'A person, portrait, face, masked or costumed figure, performer, or group of people.',
    },
    {
      label: 'general',
      description: 'A close-up of food, drink, animal, pet, vehicle, or household object.',
    },
  ],
  temperature: 0.05,
  normalizePrior: false,
};
Enter fullscreen mode Exit fullscreen mode

2.5. Resolving the Execution Delegate (resolveDelegate)

  • Checking WebGPU Support: Checks whether the browser supports WebGPU ('gpu' in this.#window.navigator) and returns 'GPU' when available.
  • Skipping CPU for .litertlm Models: Returns undefined when WebGPU is unavailable and the model path ends with .litertlm (since .litertlm vision models require WebGPU, causing classifyImage() to fall back to 'general').
  • CPU Fallback: Returns 'CPU' when WebGPU is unavailable and the model path does not end with .litertlm.
// src/app/core/vision/image-classifier.service.ts (resolveDelegate)
  private resolveDelegate(): 'GPU' | 'CPU' | undefined {
    const hasWebGpu = Boolean(this.#window?.navigator && 'gpu' in this.#window.navigator);
    if (hasWebGpu) {
      return 'GPU';
    }
    if (this.#configService.appConfig.decisionModelPath.endsWith('.litertlm')) {
      return undefined;
    }
    return 'CPU';
  }
Enter fullscreen mode Exit fullscreen mode

2.6. Classifying the Image with Safe Fallbacks (classifyImage)

  • Evaluating Category Probabilities: Calls decisionMaker.evaluateChoice() to evaluate { imageBytes } against IMAGE_CATEGORY_QUESTION and return the probabilities of the general, people, and place categories.
  • Selecting the Winner: Iterates over the returned entries and returns the category with the highest probability.
  • Defaulting to 'general': Returns 'general' when imageBytes is empty, delegate is undefined, no valid probability is returned, or an error occurs.
// src/app/core/vision/image-classifier.service.ts (classifyImage)
  async classifyImage(imageBytes: Uint8Array): Promise<ImageSubjectCategory> {
    if (imageBytes.byteLength === 0) {
      return 'general';
    }

    const delegate = this.resolveDelegate();
    if (!delegate) {
      return 'general';
    }

    try {
      const decisionMaker = await this.getValidDecisionMaker(delegate);
      const { probabilities } = await decisionMaker.evaluateChoice({ imageBytes }, IMAGE_CATEGORY_QUESTION);

      let bestCategory: ImageSubjectCategory = 'general';
      let highestProbability = -1;

      for (const [key, probability] of Object.entries(probabilities)) {
        if (probability > highestProbability && (key === 'place' || key === 'people' || key === 'general')) {
          highestProbability = probability;
          bestCategory = key;
        }
      }

      return bestCategory;
    } catch (error) {
      console.warn('[ImageClassifierService] On-device classification failed, falling back to general:', error);
      return 'general';
    }
  }
Enter fullscreen mode Exit fullscreen mode

3. Prompts, System Instructions, and Tools for Each Generative Model

  • Category Configuration Map (CATEGORY_MODEL_CONFIGS): Maps each DecisionMaker category (general, people, place) to its prompt and AiModelOptions (optional systemInstruction, response schema, and tools).
  • Domain-Specific Personas (systemInstruction): Omits systemInstruction for general, while providing peopleSystemInstruction and placeSystemInstruction to define the personas and verification rules of the people and place GenerativeModel instances.
Category systemInstruction prompt tools
general None (uses prompt instructions directly) altTextPrompt [{ googleSearch: {} }]
people peopleSystemInstruction (strict anti-guessing rule for faces/masks) peopleAltTextPrompt [{ googleSearch: {} }]
place placeSystemInstruction (landmark & coordinate verification, no fabrication) placeAltTextPrompt [{ googleSearch: {} }]
// src/app/core/vision/vision.const.ts
export const CATEGORY_MODEL_CONFIGS: Record<ImageSubjectCategory, CategoryModelConfig> = {
  general: {
    options: {
      schema: ImageAnalysisSchema,
      tools: [{ googleSearch: {} }],
    },
    prompt: altTextPrompt,
  },
  people: {
    options: {
      schema: ImageAnalysisSchema,
      tools: [{ googleSearch: {} }],
      systemInstruction: peopleSystemInstruction,
    },
    prompt: peopleAltTextPrompt,
  },
  place: {
    options: {
      schema: ImageAnalysisSchema,
      tools: [{ googleSearch: {} }],
      systemInstruction: placeSystemInstruction,
    },
    prompt: placeAltTextPrompt,
  },
};
Enter fullscreen mode Exit fullscreen mode

Full prompt and systemInstruction strings: src/app/core/vision/vision.const.ts


4. Classifying Images and Generating Alt Text in VisionService

4.1. Classifying and Selecting the Model Config in VisionService

  • Injecting the Classifier: Injects ImageClassifierService into VisionService and assigns it to this.#imageClassifierService.
  • Classifying the Image: Calls this.#imageClassifierService.classifyImage(imageBytes) to determine the category of the preprocessed imageBytes (Uint8Array).
  • Selecting the Category Config: Indexes CATEGORY_MODEL_CONFIGS by category to retrieve the prompt and GenerativeModel options.
  • Generating the Alt Text: Passes options to this.#aiModelFactory.getModel(options) to obtain a GenerativeModel instance, then calls aiModel.generateContent([prompt, imagePart]) to generate the text response.
// src/app/core/vision/vision.service.ts (analyzeImage)
  readonly #imageClassifierService = inject(ImageClassifierService);

  async analyzeImage(image: File): Promise<ImageAnalysisResponse> {
    if (!image) {
      throw Error('image is required to generate texts.');
    }

    const { data, mimeType, optimizationMetrics, imageBytes } = await preprocessImageForVision(image, this.#window);
    const category = await this.#imageClassifierService.classifyImage(imageBytes);
    const { options, prompt } = CATEGORY_MODEL_CONFIGS[category] || CATEGORY_MODEL_CONFIGS['general'];

    const imagePart: InlineDataPart = {
      type: 'inlineData',
      inlineData: {
        data,
        mimeType,
      },
    };

    const aiModel = await this.#aiModelFactory.getModel(options);
    const result = await aiModel.generateContent([prompt, imagePart]);
    ...
Enter fullscreen mode Exit fullscreen mode

4.2. Generated Alt-Text Results for people, general, and place

I uploaded an image of six Japanese wrestlers, and the classifier returned the people category. The people system instruction tells the model to use the Google Search tool to verify visible text clues and find a real-world fact based on the generated tags. The people prompt asks the model to include a person's name in the alt text only when the name is printed in the image or 100% certain and otherwise to describe the person's features, posture, and attire.

Generated Alt Text for People Category
Caption: Output for an image of six Japanese wrestlers when classified as people.

I uploaded an image of a butterfly, and the classifier returned the general category. The general prompt (altTextPrompt) generated the alt text and tags and used the Google Search tool to verify a real-world fact based on the tags.

Generated Alt Text for General Category
Caption: Output for an image of a butterfly when classified as general.

I uploaded an image of Kek Lok Si Temple in Penang, Malaysia, and analyzed the image to generate the alt text and tags. DecisionMaker classified the image input as place and used the place prompt (placeAltTextPrompt) to generate the location name, coordinates, and architectural features without fabricating unverified coordinates. The place system instruction (placeSystemInstruction) directs the model to use the Google Search tool to verify the official place name and coordinates and to find a real-world fact based on the generated tags.

Generated Alt Text for Kek Lok Si Temple (Place Category)
Caption: Output for Kek Lok Si Temple in Penang, Malaysia when classified as place.


Takeaways

  • Two-Tier Model Caching: Storing the EmbeddingGemma 2 model in CacheStorage (decision-model-v1) avoids re-downloading the .litertlm file across browser reloads, while #decisionMakerPromise and AiModelFactory reuse initialized model instances in memory.
  • Lazy GenerativeModel Construction: VisionService only creates the GenerativeModel for a category (people, general, or place) when an image is classified into that category, avoiding unnecessary model creation at startup.
  • Smaller Initial Bundle (main.js): Dynamically calling await import('@mediapipe/tasks-decision') inside initializeDecisionMaker() keeps the MediaPipe library out of the initial main.js bundle until the user analyzes an image.
  • Explicit WASM Memory Cleanup: Calling URL.revokeObjectURL() after DecisionMaker.createFromOptions() and decisionMaker.close() inside DestroyRef.onDestroy() prevents browser memory leaks from blob URLs and WASM/WebGPU instances.

Resources

Top comments (0)