Multimodal inference

Language models are one part of the network. Providers also serve embeddings, vision encoders, speech recognition and synthesis, segmentation, detection, time-series forecasting, video and generative media. Each modality has its own pool of providers, prices and runtimes, and each is reachable over the OpenAI-compatible API, JSON-RPC, the CLI, MCP and A2A.

Modalities

Inference calls bill the caller: over HTTP they take a DPoP-bound access token or an HTTP 402 payment, and over JSON-RPC they are signed by the paying account.

Each modality also has catalog, load and unload methods, for example tenzro_listTextEmbeddingCatalog, tenzro_loadTextEmbeddingModel and tenzro_unloadTextEmbeddingModel. Catalog calls are open. Loading and unloading decide what a node runs, so they are operator (admin) actions.

Embeddings

curl https://rpc.tenzro.com/v1/embeddings \
  -H 'content-type: application/json' \
  -H "Authorization: DPoP $TENZRO_BEARER_JWT" -H "DPoP: $TENZRO_DPOP_PROOF" \
  -d '{
    "model": "qwen3-embedding-0.6b",
    "input": ["the quick brown fox", "a fast auburn fox"],
    "dimensions": 512
  }'

input is a string or an array. dimensions truncates the vector on models trained for it (Qwen3-Embedding, EmbeddingGemma). A node serves embeddings from its own runtime when the model is loaded there, and routes to a network provider otherwise.

# Operator: load a catalog encoder on this node
tenzro embed-text load --model qwen3-embedding-0.6b

# Embed, locally or through the network
tenzro embed-text run --model qwen3-embedding-0.6b --input "the quick brown fox" --normalize

Speech

Transcription takes a multipart upload, exactly as OpenAI SDKs send it:

curl https://rpc.tenzro.com/v1/audio/transcriptions \
  -H "Authorization: DPoP $TENZRO_BEARER_JWT" -H "DPoP: $TENZRO_DPOP_PROOF" \
  -F file=@./interview.wav \
  -F model=parakeet-tdt-0.6b-v3 \
  -F response_format=verbose_json

response_format is json, text, verbose_json, srt or vtt; the last three include per-segment time ranges. Catalog families: Moonshine, Distil-Whisper, Whisper, Parakeet and Canary.

Speech synthesis returns raw audio bytes, not JSON:

curl https://rpc.tenzro.com/v1/audio/speech \
  -H 'content-type: application/json' \
  -H "Authorization: DPoP $TENZRO_BEARER_JWT" -H "DPoP: $TENZRO_DPOP_PROOF" \
  -d '{"model":"qwen3-tts","input":"Settlement complete.","voice":"<voice>","response_format":"wav"}' \
  --output reply.wav

Vision, detection and segmentation

Vision-language models accept images as image_url parts on chat messages, with the bytes inlined as a data: URI. Encoders (CLIP, SigLIP2, DINOv3) produce image embeddings and image-text similarity scores over JSON-RPC.

The Tenzro routes take the image base64-encoded in a JSON body:

{ "model": "rf-detr-base", "image_base64": "<base64>", "score_threshold": 0.25 }

Detection (RF-DETR, D-FINE) returns boxes as [x1, y1, x2, y2] with a label and score. Segmentation takes exactly one of prompts (points and boxes, for SAM 2, EdgeSAM and MobileSAM) or text_prompt (a noun phrase, for open-vocabulary SAM 3), and returns masks base64-encoded.

Forecasting

{ "model": "timesfm-2.5", "history": [101.2, 102.8, 104.0, 103.1], "horizon": 3, "quantiles": [0.1, 0.5, 0.9] }

The response carries the median path in point and one row of quantiles per step. Catalog families: TimesFM 2.5, Chronos-2 and TiRex.

Media generation

Images, video, audio and 3D assets are rendered one output per call, on the node you call or on a provider serving the model. The answer carries what the provider's signed receipt commits to and the price; the output itself is published to content-addressed storage and fetched by its hash. Media providers are ordinary providers with a different capability: the same identity, bond, reputation and settlement underwrite them.

The catalog also lists action_policy and world_model entries. Inputs are named in params.inputs by their SHA-256 and the role they play (first_frame, last_frame, reference, edit_source, mask, observation), and their bytes travel alongside in inputs_base64, so a render commits to the exact bytes it was conditioned on. Every render is seeded: params.seed is required.

Pricing. The unit of work is the pixel-step: width times height times denoising steps times frames. A render is metered and charged on the serving provider's price card; the answer reports the metered units and the charge as cost_wei.

Verification. A render can ask for none, receipt, optimistic or tee. The receipt commits to mime, output_sha256, output_bytes, a perceptual fingerprint and tolerance_bps, the distance a re-render must agree within. At optimistic, a share of renders is re-executed on another provider and a disagreement is filed as an inference fault.

Split-expert models. Some video models are built as two experts, one for the high-noise start of the denoising schedule and one for the low-noise remainder. Their catalog entries carry a non-null expert_pair, and they are not served as a single render.

From the CLI:

tenzro media-gen catalog --kind text2image --rpc https://rpc.tenzro.com
tenzro media-gen render --model z-image-turbo --kind text2image \
  --params ./params.json --out ./render.png --rpc https://rpc.tenzro.com

params.json holds the render parameters with a seed, for example {"prompt": "a lighthouse in fog", "width": 1024, "height": 1024, "steps": 8, "seed": 7}. Add --input <FILE> once per input params.inputs names, and --json to print the whole answer with its receipt and settlement.

Over JSON-RPC, tenzro_mediaGen_listCatalog lists the catalog and tenzro_mediaGen_render renders:

curl -s https://rpc.tenzro.com -H 'content-type: application/json' -d '{
  "jsonrpc": "2.0", "id": 1, "method": "tenzro_mediaGen_render",
  "params": [{
    "model_id": "z-image-turbo",
    "kind": "text2image",
    "params": {"prompt": "a lighthouse in fog", "width": 1024, "height": 1024, "steps": 8, "seed": 7},
    "inputs_base64": [],
    "verification": "receipt"
  }]
}'

The answer carries output_blake3. Fetch the output with tenzro_iroh_fetchBlob, passing the URI tenzro://blob/<output_blake3>, or with tenzro iroh fetch --uri tenzro://blob/<output_blake3> --out ./render.png.

Licenses and hardware

Every catalog entry carries a license tier, and a node refuses to load a model whose license the operator has not accepted. See Model registry.

The non-language modalities run on ONNX Runtime with hardware execution providers (TensorRT, CUDA, CoreML) where available, falling back to CPU. A GPU build on a machine without the matching driver still serves, on CPU.