Multimodal inference

Language models are one part of the network. Providers also serve embeddings, vision encoders, speech recognition and synthesis, segmentation, detection, time-series forecasting, video and generative media. Each modality has its own pool of providers, prices and runtimes, and each is reachable over the OpenAI-compatible API, JSON-RPC, the CLI, MCP and A2A.

Modalities

tenzro_visionEmbed, tenzro_visionSimilarity and tenzro_embed are accepted as aliases. Inference calls bill the caller: over HTTP they take an API key or HTTP 402 payment, and over JSON-RPC they are signed by the paying account.

Each modality also has catalog, list, load and unload methods, for example tenzro_listTextEmbeddingCatalog, tenzro_listTextEmbeddingModels, tenzro_loadTextEmbeddingModel and tenzro_unloadTextEmbeddingModel. Catalog and list calls are open. Loading and unloading decide what a node runs, so they are operator (admin) actions.

Embeddings

curl https://rpc.tenzro.xyz/v1/embeddings \
  -H 'content-type: application/json' \
  -H "X-Tenzro-Api-Key: $TENZRO_API_KEY" \
  -d '{
    "model": "qwen3-embedding-0.6b",
    "input": ["the quick brown fox", "a fast auburn fox"],
    "dimensions": 512
  }'

input is a string or an array. dimensions truncates the vector on models trained for it (Qwen3-Embedding, EmbeddingGemma). A node serves embeddings from its own runtime when the model is loaded there, and routes to a network provider otherwise.

# Operator: load a catalog encoder on this node
tenzro embed-text load --model qwen3-embedding-0.6b

# Embed, locally or through the network
tenzro embed-text run --model qwen3-embedding-0.6b --input "the quick brown fox" --normalize

Speech

Transcription takes a multipart upload, exactly as OpenAI SDKs send it:

curl https://rpc.tenzro.xyz/v1/audio/transcriptions \
  -H "X-Tenzro-Api-Key: $TENZRO_API_KEY" \
  -F file=@./interview.wav \
  -F model=parakeet-tdt-0.6b-v3 \
  -F response_format=verbose_json

response_format is json, text, verbose_json, srt or vtt; the last three include per-segment time ranges. Catalog families: Moonshine, Distil-Whisper, Whisper, Parakeet and Canary.

Speech synthesis returns raw audio bytes, not JSON:

curl https://rpc.tenzro.xyz/v1/audio/speech \
  -H 'content-type: application/json' \
  -H "X-Tenzro-Api-Key: $TENZRO_API_KEY" \
  -d '{"model":"qwen3-tts","input":"Settlement complete.","voice":"<voice>","response_format":"wav"}' \
  --output reply.wav

Vision, detection and segmentation

Vision-language models accept images as image_url parts on chat messages, with the bytes inlined as a data: URI. Encoders (CLIP, SigLIP2, DINOv3) produce image embeddings and image-text similarity scores over JSON-RPC.

The Tenzro routes take the image base64-encoded in a JSON body:

{ "model": "rf-detr-base", "image_base64": "<base64>", "score_threshold": 0.25 }

Detection (RF-DETR, D-FINE) returns boxes as [x1, y1, x2, y2] with a label and score. Segmentation takes exactly one of prompts (points and boxes, for SAM 2, EdgeSAM and MobileSAM) or text_prompt (a noun phrase, for open-vocabulary SAM 3), and returns masks base64-encoded.

Forecasting

{ "model": "timesfm-2.5", "history": [101.2, 102.8, 104.0, 103.1], "horizon": 3, "quantiles": [0.1, 0.5, 0.9] }

The response carries the median path in point and one row of quantiles per step. Catalog families: TimesFM 2.5, Chronos-2 and TiRex.

Media generation

Images and video are rendered as jobs rather than request-response calls. A requester posts a job with a price ceiling, a media worker claims it, renders it, publishes the output to content-addressed storage and signs a receipt over what it produced. Media workers are ordinary providers with a different capability: the same identity, bond, reputation and settlement underwrite them.

Image-conditioned jobs bind the reference image's hash into the job id, so a job commits to the exact bytes it was conditioned on.

Pricing. The unit of work is the pixel-step: width times height times denoising steps times frames. A quote is a base fee plus a rate per pixel-step, and a job whose ceiling is below the quote is refused at admission.

Split-expert rendering. Some video models are built as two experts, one for the high-noise start of the denoising schedule and one for the low-noise remainder. The network can render such a job across two machines, each holding one expert: the first hands a single intermediate latent to the second, signs the handoff with the number of steps it completed, and payment splits in proportion to the steps each half rendered. Two smaller accelerators can render what neither could hold alone.

Over HTTP, POST /v1/images/generations posts a job, waits for it and returns the image base64-encoded with a tenzro receipt block. If the wait runs out, it returns 504 naming the job to poll. POST /v1/videos returns a queued job at once: poll GET /v1/videos/{id} until it reports completed, then download from GET /v1/videos/{id}/content.

curl https://rpc.tenzro.xyz/v1/images/generations \
  -H 'content-type: application/json' \
  -H "X-Tenzro-Api-Key: $TENZRO_API_KEY" \
  -d '{
    "model": "flux2-klein-4b",
    "prompt": "a plaster studio room at dawn",
    "size": "1024x1024",
    "requester_did": "did:tenzro:human:<you>",
    "requester_address": "0x<your-address>"
  }'

The job is bound to the requester's identity, which owns the price ceiling, is the only party that can cancel it, and is who settlement charges.

From the CLI:

tenzro media-gen catalog
tenzro media-gen quote --kind text2image --params ./params.json
tenzro media-gen post-job --spec ./spec.json
tenzro media-gen get-job --job-id <id>
tenzro media-gen fetch-output --job-id <id> --out ./render.png

JSON-RPC groups the tenzro_mediaGen_* methods by who calls them. Discovery (listCatalog, quote, listWorkers) is open. Requester methods (postJob, listJobs, getJob, cancelJob, getReceipt, fetchOutput) act for the paying account. Worker methods (enrollWorker, claimJob, markRunning, failJob, publishOutput, recordHandoff, submitReceipt, fetchLatent) are signed by the worker.

Licences and hardware

Every catalog entry carries a licence tier, and a node refuses to load a model whose licence the operator has not accepted. See Model registry.

The non-language modalities run on ONNX Runtime with hardware execution providers (TensorRT, CUDA, CoreML) where available, falling back to CPU. A GPU build on a machine without the matching driver still serves, on CPU.