Skip to content
Tenzro
← All tutorials
Tutorial · Inference and training

Train and finalize a round

Run the full Tenzro Train loop on Network 1: enrol a trainer, run local steps, submit outer gradients, finalise rounds, replay and challenge, and read the receipt.

Advanced45 min

This tutorial walks the whole training loop from the trainer's and the syncer's side: a trainer enrols in a posted task, runs its local steps, submits an outer gradient, and the round is finalised by a witness committee. Every round is replayable from its seed, so you also learn how to re-run a contribution and challenge one that does not match.

Prerequisites

  • A posted task. Post one with Post a training task, or pick one from tenzro train list-runs.
  • A machine running tenzro-node with a hardware-rooted identity (TPM 2.0 or Secure Enclave) and a trainer bond. See Run a trainer node.
  • Python 3.10 or newer, and one GPU recommended.

1. Install the reference trainer

The reference trainer is a Python package that runs the local training loop with PyTorch and talks to your node over JSON-RPC. Install the extra for the modality you train.

bash
pip install 'tenzro-trainer[timeseries]'
# or: 'tenzro-trainer[language]', 'tenzro-trainer[vision]'

In production you do not launch it by hand. Set [training] enabled = true in the node config, and the node's trainer daemon discovers open runs that fit your hardware and starts a trainer for each. Check it with:

bash
tenzro train daemon-status

The steps below do by hand what the daemon does for you.

2. Enrol in the run

Enrolment registers your machine DID with the run. For the Verified and Confidential tiers, pass a TEE attestation; it is verified against the vendor's roots and, for Confidential, must commit to the enclave key the sponsor sealed the data to.

bash
tenzro train enroll-trainer \
  --task-id forecast-gpu-util-2026-10 \
  --trainer-did did:tenzro:machine:<controller>:<machine-id>

Expected output:

  Trainer enrolled!
  Trainer count: 6
  Status:        enrolling

3. Run the local loop

tenzro-trainer run fetches your shard, runs the task's local steps each round, computes the outer gradient (the difference between the weights after and before the local steps), serialises it to safetensors and submits it with a signature.

bash
tenzro-trainer run \
  --task-id forecast-gpu-util-2026-10 \
  --trainer-did did:tenzro:machine:<controller>:<machine-id> \
  --shard-uri tenzro://blob/<hash>

Shards can be tenzro://blob/... (fetched through your node), ipfs://, ar://, https:// or a local path. On a multi-GPU host, run the same command under torchrun; the language adapter shards the model across GPUs and only one rank talks to the node.

bash
torchrun --nproc-per-node 8 -m tenzro_trainer.cli run \
  --task-id forecast-gpu-util-2026-10 \
  --trainer-did did:tenzro:machine:<controller>:<machine-id> \
  --shard-uri file:///data/shard.parquet

If you already have an outer gradient as JSON, for example from your own training code, submit it directly:

bash
tenzro train submit-gradient --gradient gradient.json

4. Watch the syncer decide the round

Trainers run on different hardware and networks, so a round does not wait for everyone. It is bounded by the grace window in the task spec. Ask the syncer what it will do:

bash
tenzro train decide-round --task-id forecast-gpu-util-2026-10
# Decision: wait      -> Grace window remaining (ms): <n>
# Decision: finalize  -> Round: <r>
# Decision: no_quorum -> Round: <r>
  • wait: the window is still open.
  • finalize: enough gradients arrived for every fragment.
  • no_quorum: the window closed without a quorum. The run advances and carries the previous state root forward, so a stalled round never blocks the run.

5. Finalise the round

The syncer aggregates the accepted gradients under the task's rule, applies the outer optimiser step and hashes each updated fragment. Witnesses then finalise the round with those hashes. Finalising is idempotent: several witnesses can submit the same round and state root at once without double-finalising, and a conflicting state root is rejected, which makes forks visible.

bash
tenzro train finalize-round \
  --task-id forecast-gpu-util-2026-10 \
  --round 12 \
  --post-step-hashes '{"0":"0xabcd...","1":"0x1234..."}'

Expected output:

  Round finalized!
  State root: "0x9f3c..."

6. Replay a round and challenge it

Every round is replayable from its seed: given the round's checkpoint, the trainer's shard and the seed, re-running the local steps reproduces the trainer's activation commitment. To check a contribution, re-run it with the reference trainer, save the recomputed commitment as JSON and submit it.

bash
tenzro train challenge-commitment \
  --task-id forecast-gpu-util-2026-10 \
  --round 12 \
  --fragment 3 \
  --trainer-did did:tenzro:machine:<controller>:<other-machine> \
  --recomputed recomputed-commitment.json

If the commitments differ, the trainer is evicted and its buffered gradient is dropped before aggregation.

7. Read the receipt

After the last round, the syncer seals a receipt with the task spec, every round's state root, the final model hash, each trainer's contribution count and reward share, and a Merkle root over the run. Rewards are paid from the sponsor's escrow according to those shares.

bash
tenzro train get-receipt --task-id forecast-gpu-util-2026-10

Next steps