Training
Full-batch Adam on every training step at once, exactly lag3's recipe (learning rate 0.003).
Every parameter, as it trains
One cube per parameter: all 8,170 of them, block by block in the order the forward pass uses them. A cube's height is the size of its weight, its colour the sign; the white glow is how much each parameter changed in the latest steps, so you can watch where gradient descent is working. Drag to turn it; pinch or ⌘-scroll to zoom.
The twelve programs
A program runs correctly exactly when the network gets every step of its run right, so each one is checked step by step against the universal machine's exact rule. None of them is in the training data.
The whole step table
Every step the universal machine can take on these machines: 14,849 of them, one cube each, stacked in the order the random machines first produced them. The training set is the bottom of the stack. Drag to turn it.
Loading three.js…
What it learns about each token
Each token's learned embedding, 8 numbers, shown by its three main directions of variation, moving
as it trains. Two tokens are marked: go_aS:I and go_bS:I, "write 0, stay" and "write 1,
stay" on a marked 1. With too little data the network misses a step for the first that it has seen for the
second, because to it they are unrelated tokens. Watch whether they end up close.