AttentionAI is a tiny transformer — the same family of model that powers
modern language AIs like ChatGPT, scaled way down. With roughly 2,000 changeable
numbers it is still thousands of times smaller than a real one, but it shows the
key trick that makes them work: attention.
Embeddings. First every word is turned into a list of numbers (a
"meaning vector") by an embedding layer. Words used in similar ways get
similar numbers, so the brain can compare them.
Attention. Then a layer computes three lists per word — query,
key, and value. The query of the last word is matched against the
keys of all the words in the sentence to decide how much to focus on each one. Those
focus weights are the glowing lines you see: the stronger the glow, the more that
word matters for predicting what comes next. It is single-head attention —
one spotlight instead of the many "heads" a real transformer uses.
Feed-forward layers. After attention combines the important words, the
result passes through two small feed-forward layers (like the TinyAI and
SmallAI networks) that do the final reasoning.
The output. A final head layer turns the result into a score for
every word in the vocabulary, and the highest one is the next-word prediction.
The whole thing — embedding, attention, feed-forward, head — is what gives this
brain its roughly 2,000 adjustable numbers.