Training

Decentralised training on Tenzro Network 1: how runs, rounds, trainers and syncers work, the trust tiers, and how to run a trainer.

Tenzro Train is decentralised training on Tenzro Network 1. A sponsor posts a training run and funds it. Trainers, from a single home GPU to a data centre, enrol in the run, train on their share of the data and submit updates. A committee of syncers merges those updates round by round and commits the result to the ledger. Every round is replayable from its seed, so anyone can check what happened.

Tenzro Train is built for the kind of training the Machine Economy needs most: fine-tuning and post-training open models on specific data, including agent trajectories. See Training for agents.

Roles

Role	What it does	Bond type
Sponsor	Posts the run spec, supplies the dataset reference and funds the reward pool.	None
Trainer	Runs the inner training loop on its hardware and submits signed outer gradients.	`trainer`
Syncer	Serves on the round's committee: checks submissions, aggregates them and signs the round result.	`syncer`

Trainers and syncers bond before they enrol, and their identities are rooted in the machine's hardware key, like every node on the network. A node takes the training role alongside any others it runs. See Operators and roles.

How a run works

Tenzro Train uses decoupled, low-communication training. Trainers do many local steps between synchronisations, so the run tolerates ordinary internet links and machines that come and go.

Post. The sponsor posts a task spec: the model architecture and modality, trust tier, aggregation rule, number of trainers and quorum, inner steps per round, number of rounds, a grace window, the dataset reference and its hash, the objective and the reward pool.
Enrol. Trainers enrol with a signature from their machine key. In Verified and Confidential runs they also present a TEE attestation.
Inner loop. Each trainer takes the current weights, runs H local steps on its assigned shard and computes its outer gradient, the difference between its weights after and before the steps.
Submit. The trainer signs and submits the outer gradient, with the hash of its payload and, in the Open tier, an activation commitment.
Aggregate. The round's committee verifies each submission, clips it, and merges the accepted ones with the run's aggregation rule. A Nesterov outer optimiser applies the merged update.
Finalise. Once a quorum has submitted, the committee signs the new state root for the round and commits it on-chain. If the grace window ends without a quorum, the run moves on under a no-endorsement certificate that carries the previous state root forward.
Receipt. When the run completes, a receipt records the dataset hash, the round state roots and each trainer's contribution, and rewards are paid from the pool.

Rounds never wait for the slowest trainer: submissions that land inside the grace window count, and stragglers catch up in the next round.

Trust tiers

Tier	Who can train	What backs a submission
Open	Any bonded machine, GPU or CPU	Bond, Byzantine-robust aggregation and activation commitments that challengers can re-execute
Verified	Machines with a TEE	A TEE attestation on every submission, verified against the vendor's root
Confidential	Machines with a TEE	Dataset shards encrypted for the trainer's attested enclave, so the host never sees the data in the clear

Supported TEE platforms are Intel TDX, AMD SEV-SNP, AWS Nitro and NVIDIA confidential GPUs. Attestation is evidence about the workload; the trainer's identity is still its hardware-rooted key. See TEE.

Aggregation rules

Mean: the average of accepted updates.
LoraAlternating: an alternating-freeze rule for LoRA and QLoRA adapter runs.
TrimmedMean, CoordinateMedian and Krum: Byzantine-robust rules that limit the influence of bad updates.

Open runs use Mean or LoraAlternating. Verified and Confidential runs can use any rule. Every run can set a per-update norm cap, so no single trainer can move the model far in one round.

Checking the work

In the Open tier, each outer gradient carries an activation commitment: the loss at every inner step and the largest coordinates of the update, bound into the trainer's signature. The committee checks the structure when the gradient arrives. Anyone can challenge a commitment by re-running the trainer's inner loop from the round's seed, starting weights and shard, and comparing the result within tolerance bands sized for floating-point differences between GPUs. A trainer whose submission fails the check is evicted and its bond is slashed. See Training for agents for how a round is replayed.

Objectives

Objective	Inner loop
`Supervised`	Standard supervised loss over the assigned shard. The default.
`ChatSft`	Supervised fine-tuning on chat and tool-call trajectories, with the loss on assistant turns only. Language models.
`RlPostTraining`	GRPO-style reinforcement learning: sample a group of completions per prompt, score them with the sponsor's reward function and update toward the better ones. Language models.

Whatever the objective, trainers submit the same kind of outer gradient, so aggregation, commitments and receipts work the same way.

Modalities and data

Runs can train language, vision and timeseries models, with full fine-tuning or LoRA and QLoRA adapters, including mixture-of-experts language models. Datasets are published to the network's content-addressed store and referenced by hash:

bash

tenzro iroh publish --file shard-3.parquet
# tenzro://blob/<hash>

Each trainer is assigned its own shard for each round, and fetched data is checked against the hash in the task spec before training starts.

Communication efficiency

A task can declare several ways to cut the data each round moves: Int8 or Int4 quantisation of updates; top-k sparsification with error feedback, so nothing dropped is lost; streaming synchronisation of one parameter shard per round; delayed application, which overlaps synchronisation with the next round's compute; and pipeline-parallel trainer groups, so no single machine has to hold the whole model. The reference trainer scales across the GPUs of one host automatically and supports the Muon inner optimiser.

Run a trainer

Build the trainer image from Dockerfile.trainer in the network repository, or install the reference trainer package, tenzro-trainer, next to your node.

Bond as a trainer:

bash

tenzro stake deposit <amount> --provider-type trainer

Enable training in the node config:

toml

[training]
enabled = true
max_concurrent_trainers = 1

Restart the node. Its trainer daemon discovers runs, enrols with the node's machine identity and starts a trainer process for each run, up to max_concurrent_trainers. Check it with tenzro train daemon-status.

Post and follow a run

bash

# Post a run from a JSON task spec (signed by the sponsor)
tenzro train post-task --spec run.json

# Follow it
tenzro train list-runs
tenzro train get-run --task-id <task-id>
tenzro train decide-round --task-id <task-id>   # wait | finalize | no_quorum
tenzro train get-receipt --task-id <task-id>

The JSON-RPC methods use the tenzro_training_ prefix. Reads such as tenzro_training_listRuns, tenzro_training_getRun, tenzro_training_decideRound and tenzro_training_getReceipt are open. Posting a run, enrolling and submitting gradients are owner calls, signed by the sponsor or trainer. Finalising a round requires the committee's signatures.

Training for agents: fine-tuning on agent trajectories and replaying rounds.
Models: serving the models you train.
Iroh: the content-addressed transport behind datasets and updates.

Roles

How a run works

Trust tiers

Aggregation rules

Checking the work

Objectives

Modalities and data

Communication efficiency

Run a trainer

Post and follow a run

Related