A receipt for every model

What it makes possible

Three possibilities follow from being able to reproduce the work exactly. Each depends on having the program, the data, and arithmetic the checker knows how to follow.

A stranger’s machine can be checked. Someone who rents out a processor and returns a result can be asked for a receipt. For the computations it covers, the result can be compared with an independent recomputation, without re-renting the processor or taking the operator’s word for the answer.

A broken machine can be found. In the arrangements we have tested, a wrong result causes a comparison to fail at the recorded step and identifies the machine that returned it. A discrepancy becomes a place to investigate. At the scale of a thousand processors, that is the ambition: find the fault where it enters the work, before it becomes a distant change in a training graph.

A saved state can be deleted. If a checkpoint can be recomputed exactly from an earlier one, it need not always be stored. Keep one in a hundred; reconstruct the intervening states when they are needed. Within the runs covered by this work, recovery can happen on different hardware, including another provider’s, so long as it can follow the receipt. Our small computers check the recoveries every night. Keeping less only makes sense if the way back remains dependable.

What it does not do yet

It is slow. In our measurements, an ordinary computer takes about forty seconds to redo what the processor did in a tenth of one. That can be acceptable for checking a sample of steps, or for an audit. It cannot watch the whole run in real time. A faster checker, running on the same kind of hardware as the run, is the next step.

The work has been shown on one machine at a time and on one group of four, for models of the size a small lab trains. The sizes and arrangements a large lab uses — a billion parameters, many machines, the fastest arithmetic the hardware offers — are the next chapter. The results so far do not establish that those larger cases will work in the same way.

And every receipt so far has been checked by our own computers. The design lets anyone check one. No one outside has yet. That is the first thing we would like to change.

The name, and the goal

In 1948, Alan Turing described machines whose connections were arranged largely at random. He called them unorganised. In the same report, he considered how a machine could be shaped through training, using the word interference for the influences that help give it form. Organisation could be something a machine acquired.