Σ
Where the 23,844 parameters are
Almost all of them are the two lookup tables at the ends, one row per token. The computing in between is 832 numbers.
One forward pass
A transformer of 23,844 parameters runs a universal Turing machine, and through it the sieve of Eratosthenes. It does that one forward pass at a time: read three tokens, write the next. Scroll to follow one pass through the whole network, every weight and every number it computes.
Loading three.js…
Almost all of them are the two lookup tables at the ends, one row per token. The computing in between is 832 numbers.