Documentation

Chain synchronization

Status: implemented — headers, the manifest, parallel bodies, serial application, the UTXO reorg.

This document describes how a node gets from genesis to the network’s tip and how it keeps up once it is running. The code: cmd/powmachinad/bodysync.go (the driver), internal/blocksync (parallel downloading), internal/store/restore.go (application and recovery after a restart).

Two chains, not one

A node holds two different kinds of progress and it is important not to mix them up:

	what it is	the mark in storage	who moves it
the header tip	the highest verified header of the active branch	`keyHeaderTip`	`SetActiveBranch`, `ApplyBlockDirect`
the UTXO tip	up to which block the UTXO set is finished	`keyTip`	`ApplyBlock*` in one batch with the UTXOs

The header tip can be far above the UTXO tip and that is a normal state, not an error: a header is 89 B, a body up to 352 kB (96 kB of transactions and up to 256 kB of settled shares).

The manifest

A verified chain of headers is a manifest: for every height it says what hash the block at that height has. Which means:

A block’s body can be downloaded from anyone. Even from an enemy.

The source does not have to be trustworthy — either it gives bytes whose hash and Merkle root match the header, or we throw it away (blocksync.VerifyBody). That is why bodies are pulled in parallel, out of order and even from a plain HTTP mirror (-blocksource).

The manifest is the height index in storage (keyHeight). There is a single rule about it, but a strict one: the index describes the active branch. That is why only store.SetActiveBranch writes it, which the node calls when the tip moves (indexActiveBranch), and not PutHeader: a header on its own does not say whether it is on the active branch. When every accepted header wrote the index, one header from a branch that lost was enough for the node to assemble a chain from two branches after a restart, according to the index, and refuse to start.

A block’s path

 peer ──inv(block)──▶ do we lack the header? ──getheaders──▶ headers ──▶ AddHeader (PoW, bits,
                                                                        lane rotation, MESS)
                                                                 │
                                                 SetActiveBranch │ (the manifest = the active branch)
                                                                 ▼
    the body driver: what is missing between the UTXO tip and the header tip ──getdata──▶ bodies
                                                                 │
                                           VerifyBody + PutBlock │ (stores it, does NOT apply it)
                                                                 ▼
                              ApplyStored: in height order, with full validation into the UTXOs

A body is never requested for a header we have not verified — we would have nothing to verify it against. A body that arrives anyway is discarded without ban points: between the request and the arrival the header can be lost to a reorg and that is not the peer’s fault.

Propagating a new block

A node announces a new tip onward whenever its active branch moves — not only when it mines a block itself. Without that, relay is one-hop: the block reaches the miner’s direct peers and stops there: the miner’s node and its direct peers grow while the nodes behind them stand two to four heights below the tip.

A fresh block is handed over whole, not announced: one headers message with its header and the block with its body right behind it, to every peer, at every hop — as long as the block is under maxBlockPushBytes (64 kB). An announcement alone means the peer has to ask for the headers, verify them, and only then may the body driver ask for the body: a round trip plus up to one driver tick per hop, and when one request goes unanswered, the next attempt is a whole bodyRequestTimeout away.

That chain of waits is not harmless. When a block’s header reaches every node in seconds and its body reaches none of them for minutes, nothing can be applied past the hole: the UTXO set of the whole network stops one block below it — with it every wallet and the explorer, while the tip keeps growing and the chain looks healthy.

The order within the push matters: a body whose header the peer does not know yet is dropped (storeBody), so the header goes first in the same stream. Above the push limit the peer gets an inv and pulls what it wants — which is also where compact blocks (BIP152) would belong one day.

A safeguard against a lost announcement: when the driver has nothing to fetch, it asks every peer for headers once per headerPollInterval (30 s). One message per peer every half minute is cheaper than a node that stands still until somebody new connects.

The body driver

What is to be downloaded is decided by state, not by the last message received:

the candidates = the heights from the UTXO tip + 1 up to the header tip that are missing a body (capped at bodyWindowBatches × MaxGetDataItems bodies in flight),
the batch per peer = MaxGetDataItems (128),
a request nobody answered within bodyRequestTimeout (20 s) is repeated — preferably to a different peer, and when there is only one, to the same one,
the oldest missing body is asked of every peer at once, the rest are split between them: a body verifies itself against the manifest, so a duplicate costs only bandwidth, while waiting for one silent peer costs a whole timeout — and nothing above that height can be applied meanwhile,
the driver runs every bodySyncInterval (2 s) and is additionally woken by receiving headers or an inv with a new block.

Why doing it “after headers are received” is not enough: one message carries up to MaxHeadersPerMessage (2,000) headers, but only 128 bodies fit in one getdata. A node that requests 128 bodies and never asks for the rest stays with its headers ahead and its bodies behind forever.

A single owner of application

Bodies are applied and the branch index is written exclusively by ApplyReadyBodies, under applyMu, in this order: the index → reconciling the UTXOs → application. That holds for the bulk download (-connect), for a body received from the network and for a driver tick alike.

When two places wrote the index (receiving headers and applying), two concurrent header messages overwrote the manifest with each other’s older view of the chain and application stopped with the headers ahead. It only showed up in the reorg test, and only occasionally — exactly the class of bug that is hardest to find on a live network.

Application and the reorg

Application is serial and validating (store.ApplyStored): it takes bodies from disk in height order, runs Block.ValidateBlock on each and writes it into the UTXO set in one atomic batch with the undo data. The invariant: blocks up to the UTXO tip are verified, bodies above it are not.

Before application the UTXOs are reconciled to the active branch (reconcileUTXOTip): while the UTXO tip does not match the block that is at that height on the active branch, it is unwound through store.RevertTip (the undo data is in storage with every applied block). Without this step a header reorg would leave the UTXOs on a dead branch and every subsequent application would end with “the block does not build on the tip” — forever, because nothing would bring it back from there.

RevertTip does not delete the height index: the manifest belongs to the header branch, not to whatever is currently applied. When it did delete it, a reorg also deleted the entries the new branch had written there moments earlier, and the driver had nothing left to go by.

The bulk download

-connect <host:port> and -blocksource <url> go through internal/blocksync: the manifest is split into batches, those are pulled from every source at once, a source that lies is dropped immediately and one that cannot keep up after MaxFailures. It is an accelerator, not an authority — every body goes through the same verification against the manifest as a body from a peer.

What it says about itself

Left alone, this whole path is silent: a body that fails verification is dropped without a word, and a request that is never answered looks exactly like one that was never sent — nothing to look at when it matters. So the node reports:


`powm_header_body_gap`	how far the UTXO set is behind the headers — the metric the alert stands on
`powm_oldest_missing_body_seconds`, `powm_bodies_inflight`	how long the longest wait has been going on, and how many are waiting
`powm_body_requests_total`, `powm_bodies_delivered_total`, `powm_body_timeouts_total`, `powm_body_mismatch_total`	counters of the driver
the log	one line per 45 s when a body is missing that long: the height, the age, how many peers were asked; and a line for every body that does not match the manifest

What the tests hold

test	what would break otherwise
`TestSyncsChainLongerThanGetDataBatch`	the sync stops at the first batch of bodies
`TestSyncsChainOverRealGetDataWindow` (`POWM_SLOW_TESTS=1`)	the same with the real 128
`TestOutOfOrderBodyWaitsForItsTurn`	an out-of-order body is applied or discarded
`TestBodySyncRetriesAfterTimeout`	a silent peer stops the synchronization
`TestMinedBlockReachesPeerWithBody`	relay does not deliver the body (`inv` → `getdata` without a header)
`TestBlockRelayCrossesTwoHops`	relay is one-hop — the nodes behind a neighbour stand still
`TestReorgRevertsUTXOToActiveBranch`	the UTXOs stay on a dead branch
`TestRestartAfterReorgLoadsHeavierBranch`	after a reorg and a restart the node comes up on the weaker branch
`TestLosingBranchHeadersDoNotRewriteManifest`	a header from a losing branch overwrites the manifest
`store.TestSetActiveBranchReplacesAndTruncates`	entries of the old branch stay in the index after a reorg
`TestInvalidBodyDoesNotStallNode`	a block with valid PoW and an invalid body stops the node
`TestInvalidBranchSurvivesRestart`	after a restart the invalid branch is accepted again
`TestSilentPeerDoesNotStallTheChain`	one peer that does not hand over one body stops the chain
`TestBlockWithTransactionsReachesPeer`	a body with transactions does not make it into the store
`TestMetricsShowHeaderBodyGap`	the gap between the headers and the UTXO set is invisible

A node as its own seed

When a node is a DNS seed itself, it gets its own address from the seed. A connection to itself is only revealed by the handshake (the nonce in version) — a node behind NAT does not know its public address — and it is revealed by the receiving side, because that side reads version first; the caller only sees a closed connection. That is why the receiving side remembers the address (AddrMan.MarkLocal, keyed by IP) and the addrman never accepts it again. Without that a public node knocks on itself every ten seconds.

What is still missing

Bodies are requested from every connection equally — with no measurement of which peer keeps up. The oldest missing one is asked of everybody, which covers the case that hurt, but a peer that never delivers still gets its share of the work.
The bulk download from HTTP mirrors (-blocksource) is only used by the -connect path; the background driver pulls bodies from peers exclusively.