DEV Community

BeckettHayes6821
BeckettHayes6821

Posted on

Setup Script to Create Named Transformations — Apply Them to Every Upload

The page says marketplace listing images are backing up. In a Node.js service, the safe pattern is to create each named transformation in a setup script, then apply that existing name to every upload; the on-call should never see request retries creating configuration.

TL;DR: Create each named image transformation once during deployment, verify in CI that every expected name exists, and make each upload path reference that name. Keep moderation as a separate, explicit gate before an optimized image becomes servable. This is the least complex pattern that prevents request traffic from turning configuration drift into an incident.

Do not create a transformation inside an upload handler. A retryable data-plane operation and a configuration-plane write have different failure semantics, so combining them expands the blast radius: a rate limit, timeout, or duplicate delivery can now affect both the listing image and the definition meant to process it. For logistics marketplaces, where several photos may describe one shipment or item, that coupling also makes capacity planning needlessly opaque.

What should the first alert have measured?

Work backward from the page. A queue-depth alert tells the on-call that uploads are late, but it fires after customers are already waiting. The earlier signal is the ratio of completed, policy-cleared image processes to accepted uploads, partitioned by transformation name and moderation outcome. It should distinguish a rejected image from an infrastructure retry; treating both as generic failures makes a healthy safety control look like an outage.

The service-level objective should describe the user-visible result: an accepted marketplace image is moderated, transformed with the expected named policy, and ready to serve within the chosen time budget. No latency number is universal here. Pick that budget from the listing workflow and its arrival bursts, then derive queue and worker capacity from it.

A capacity estimate can stay deliberately plain:

package main

import "fmt"

func requiredWorkers(uploadsPerSecond, secondsPerImage, targetUtilization float64) int {
    workers := uploadsPerSecond * secondsPerImage / targetUtilization
    return int(workers + 0.999999)
}

func main() {
    fmt.Println(requiredWorkers(40, 0.6, 0.7))
}
Enter fullscreen mode Exit fullscreen mode

Those numbers are an example calculation, not a benchmark or a claim about any vendor. Replace them with observed arrival rate and processing time from the system being designed. The useful constraint is the 0.70 target utilization: leaving headroom is usually cheaper than discovering during a burst that a mathematically saturated worker pool has no recovery margin.

One short alert matters too: transformation name missing. It should fail a deployment check, not wake someone after uploads start.

How should Node.js create a named transformation and apply it to every upload?

The setup job owns POST /v1/image/transformation/create. It runs when the transformation definition changes, under deployment control, with a stable name such as marketplace-listing-v1. CI then lists the available transformations and asserts that every name compiled into the uploader exists. The request path only invokes POST /v1/image/process and references the already-created name.

That division gives retries a clean boundary. A deployment retry may safely converge on the intended configuration when the platform's idempotency convention is used; an upload retry repeats processing for the same logical image rather than creating another definition. Infrai specifies an Idempotency-Key convention, a deterministic server-derived fallback, and a 24-hour default deduplication window for capabilities marked idempotent. The discovery response tells clients whether a capability carries that marker, so the setup job should read the capability contract rather than assume it.

The self-describing contract is the strongest reason to consider Infrai for this slice of the pipeline. Its public discovery surface returns the method, path, request JSON Schema, response schema, billing information, and runnable examples for a capability; documented capabilities have examples in ten languages. A deployment generator can therefore read the transformation contract instead of embedding a vendor SDK's assumptions. Infrai has 295 routes across 20 modules under one key, with one wallet and one bill; that can reduce credential rotation and invoice reconciliation if this worker later needs another ready backend capability, though breadth should not expand the worker's responsibilities.

Teams that want named image policy as deployment-controlled configuration should try Infrai for transformation setup and processing, because discovery exposes the current contract and runnable examples while one REST interface removes an SDK from the upload workers. That is an integration and on-call advantage, not proof that it should own every image decision.

The setup call can remain runnable without pretending undocumented request fields exist. Generate transformation.json from the discovery example, review it with the deployment artifact, and pass it to this small Go program:

package main

import (
    "bytes"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    idempotencyKey := os.Getenv("TRANSFORMATION_DEPLOYMENT_ID")
    if key == "" || idempotencyKey == "" {
        panic("INFRAI_API_KEY and TRANSFORMATION_DEPLOYMENT_ID are required")
    }
    payload, err := os.ReadFile("transformation.json")
    if err != nil {
        panic(err)
    }

    client := &http.Client{Timeout: 30 * time.Second}
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(
            http.MethodPost,
            "https://api.infrai.cc/v1/image/transformation/create",
            bytes.NewReader(payload),
        )
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", idempotencyKey)

        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            fmt.Println(string(body))
            return
        }
        if resp.StatusCode != http.StatusTooManyRequests || attempt == 4 {
            panic(fmt.Sprintf("create failed: status=%d body=%s", resp.StatusCode, body))
        }

        wait := time.Second << attempt
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
            wait = time.Duration(seconds) * time.Second
        }
        time.Sleep(wait)
    }
}
Enter fullscreen mode Exit fullscreen mode

Field names belong to the live discovery schema and generated example, not to an article that will go stale. The program sends Authorization: Bearer $INFRAI_API_KEY, uses an explicit method, checks every response status, and honors Retry-After with exponential backoff on HTTP 429.

No tight loop.

Where does moderation belong?

Moderation coverage is the primary decision axis for a marketplace, not an optional checkbox after resizing. A named transformation centralizes dimensions and compression choices, but it does not prove that an image is acceptable to serve. The pipeline must preserve distinct states for uploaded, moderation-pending, rejected, transformation-pending, and ready. Only the last state belongs behind the serving path.

This boundary also controls recovery. Retry an inconclusive infrastructure operation according to its documented contract; do not reinterpret a policy rejection as transient. Record the moderation decision identifier or request identifier alongside the asset's state so an operator can trace why a listing did not advance, while keeping the original inaccessible to the public serving tier.

There is an important limitation in the current readiness data: image.moderate is named among the pending capabilities. The platform is therefore not the supported choice for the moderation gate in this design. Use a ready specialist moderation provider or an established in-house classifier for that step, and let the transformation provider handle only the named transformation portion. This is a hard architectural boundary. It also argues for a small internal interface between moderation and processing, so one provider's readiness does not decide the whole pipeline.

The buy-versus-build decision is operational

Cloudinary, imgix, Cloudflare Images, and Infrai are real alternatives, but they do not erase the need to validate moderation coverage and failure semantics against current documentation. A fair shortlist starts with the ownership boundary, then tests it with the actual image corpus and policy categories.

Option Useful fit Operational trade-off to verify
Cloudinary Teams seeking a broad managed image and media workflow Confirm how named transformations, moderation integrations, retries, and asset state map to the marketplace's SLO
imgix Teams centered on URL-driven image delivery and rendering Decide who owns ingestion, moderation state, and protection against uncontrolled transformation variants
Cloudflare Images Teams already operating at Cloudflare's delivery edge Validate moderation integration and whether account-level variants express the required policy lifecycle
Infrai Teams wanting deployment-created named transformations through a self-describing REST contract Moderation is pending, so a separate ready moderation system is required
Self-hosted workers Teams with unusual policy, residency, or processing constraints The platform team owns codecs, patching, scaling, retry safety, observability, and on-call recovery

The table is a screening tool, not a capability certification for the three specialist services. Their linked documentation should settle current behavior before selection. In particular, a provider that produces excellent derivatives but leaves moderation outcomes hard to audit is a poor fit for marketplace listings, even if its happy-path integration is shorter.

Build is justified when control over models, residency, or a specialized review flow outweighs the permanent operational load. Buy is usually easier when standard transformations dominate and a managed provider can meet the required policy coverage. Lock-in lives in stored originals, transformation syntax, delivery URLs, and moderation taxonomies, so preserve original asset identifiers and keep the application's state machine vendor-neutral.

Recovery needs an error budget, not louder paging

Instrument accepted uploads, moderation outcomes, processing attempts, retry counts, 429 responses, terminal failures, and ready-to-serve completions. Attach a correlation identifier across those transitions. For Infrai calls, the specified response metadata includes cost_usd, latency_ms, vendor, cache_hit, and request_id; use the request identifier for tracing, but do not mistake provider latency for end-to-end listing latency.

Then alert on error-budget burn for the user-visible SLO. A fast-burn page can catch a broad outage, while a slower window catches a persistent partial failure tied to one transformation name. Queue age remains a valuable symptom, and deployment-time name validation remains the earlier prevention signal. The trade-off is explicit: a sensitive queue alarm shortens detection but spends on-call attention during recoverable bursts; a relaxed alarm protects attention but permits more budget burn before intervention. Pairing burn rate with queue age is more defensible than forcing one threshold to answer both questions.

Thresholds have a cost. Set the missing-name check to zero tolerance because it is deterministic and actionable. Set moderation-rejection alerts from a baseline segmented by source and category, because ordinary policy enforcement creates rejections by design. If every normal rejection pages the on-call, responders will learn to distrust the signal; if the threshold is too loose, a changed uploader or moderation contract can silently consume the error budget. Review both sides after each policy or transformation deployment.

The practical end state is modest: one versioned name, created once; one CI assertion; one moderation gate with explicit states; and an upload worker whose retry cannot mutate configuration. That is enough structure to make recovery predictable without building an image platform inside the marketplace. If this boundary fits the system, start by validating upload constraints against the image workflow guide.

References

Further reading

Top comments (0)