“Each model is a Transformer which leverages mixture-of-experts to reduce the number of active parameters needed to process input. gpt-oss-120b activates 5.1B parameters per token … The models have 117b and 21b total parameters …
We tokenized the data using … o200k_harmony … The models were post-trained … including a supervised fine-tuning stage and a high-compute RL stage. … The weights … are freely available … and come natively quantized in MXFP4. This allows for the gpt-oss-120B model to run within 80GB of memory … Available under the flexible Apache 2.0 license.”
Spec table: context length 128k.
Circle every word you can explain today. By 11:55 you will circle them all.
Three ways to write the same decision
1.0 you write the rule yourself. 2.0 you give examples and the machine learns the rule — as numbers called weights. 3.0 you give an instruction in plain English to a model that has already learned from huge amounts of text. 2.0 and 3.0 are built from the same thing: neurons.
Eight past payments
payment
large? (over €1,000)
abroad?
new device?
fraud?
A
0
0
0
no
B
1
0
0
no
C
0
1
0
no
D
0
0
1
no
E
1
1
0
yes
F
1
0
1
yes
G
0
1
1
yes
H
1
1
1
yes
Bet
A first guess: weight 1 on “large”, 0 on the others, bias −1; block if the score is above zero. Which payments does it block?
Three red flags per payment. The hidden truth, which the model is not told: fraud when at least two flags are on.
The first guess, as one neuron
Learning from mistakes, by hand
Wrong → move each weight a little towards the right answer; right → leave it. η, the learning rate, is how big each move is.
Six corrections over three rounds; round 4 makes none. Nobody wrote “two red flags” — the weights found it. And they decided “large” counts half as much as the others. This neuron is the perceptron, 1958.
Yes or no is not enough
How wrong, as one number
The loss of one payment: small when the model gave the right answer a high probability. The loss of the model: the average over all payments.
payment
fraud?
p(fraud)
loss = −log p(right answer)
after training: p(fraud)
A
no
0.27
0.31
0.00
B
no
0.38
0.47
0.05
C
no
0.50
0.69
0.05
D
no
0.50
0.69
0.05
E
yes
0.62
0.47
0.96
F
yes
0.62
0.47
0.96
G
yes
0.73
0.31
0.96
H
yes
0.82
0.20
1.00
average
0.45
loss 0.03
That average is the one number training pushes down: 0.45 at the perceptron’s weights, 0.03 after 200 steps of the formula on the next slide.
The formula for every weight
w \;\leftarrow\; w - \eta \cdot (p - y) \cdot x \qquad\qquad \text{new weight} = \text{old weight} - \text{learning rate} \times \text{blame}
One step on payment E (large, abroad, fraud), with η = 1:
w large
w abroad
w device
bias
p(fraud) for E
before
0.5
1
1
−1
0.62
blame = (p − y)·answer
−0.38
−0.38
0.00
−0.38
after (η = 1)
0.88
1.38
1.00
−0.62
0.84
Same shape as the perceptron rule, with a probability instead of a yes/no. “New device” gets no blame: it was off for E, so it could not have caused the mistake.
Where the blame comes from: backpropagation
The learning rate
Add payment I — a customer on holiday: large, abroad, genuine — and no rule is perfect any more. Too small a step crawls; too large jumps over the best answer.
How every network learns
A pattern one neuron cannot draw
More activation functions
From 4 numbers to 117 billion
An LLM is the same machine
The same nudge, on text
Think of a mixer: one knob per possible next token. Each sentence turns the right knob up a little — exactly the step we took on payment E.
Where the knobs end up
Pre-training is compressing the internet
Repeat that nudge on trillions of tokens and the weights end up holding a lossy summary of the text: the gist survives, exact facts do not always.
Then it learns to be an assistant
1 · Pre-training
internet text → a base model that continues documents
2 · Post-training
~100,000 example conversations, then rewards → an assistant
Post-training is the same loop again. Only the examples change: now the right answer is a good reply.
One prediction, end to end
Every word, taken apart — with our fraud neuron
in the release
in terms of what we just built
Transformer
our neurons, plus attention, stacked in 36 layers
117B total parameters
our neuron had 4 numbers (3 weights and a bias); this has 117 billion
mixture-of-experts
128 smaller networks inside; a router picks 4 for each token
5.1B active per token
only those 4 run, so it costs like a 5-billion model
tokenized, o200k_harmony
how text becomes numbers before the first neuron
post-trained
more rounds of the same training loop, on new data
supervised fine-tuning
the loop, where the right answers are good conversations
RL
the loop, where the “right answer” is a reward score
context length 128k
how many tokens go in at once — a few hundred pages
quantized, MXFP4, 80GB
each weight stored in ~4 bits instead of 16: fits on one GPU
open-weight, Apache 2.0
you download the weights themselves, and may use them commercially
Part 4 What the model reads
It never sees a word
Bet
"Il margine è insostenibile" — four words. How many pieces?
Before the network sees anything, a tokenizer cuts the text into pieces and turns each piece into a number. You are billed, and limited, in these pieces — not in words.
The same sentence, twice
How the tokens talk to each other: attention
Bet
“The bank raised rates because it feared inflation.” — what is it?
Vaswani et al., Attention Is All You Need, 2017 — the paper behind every model since.
To process each token, the model looks at every other token in the context and decides how much each one matters — here, most of the weight goes to “bank”. Every token does this with every other one, which is why long contexts get expensive.
Everything it knows, every time
The context window is everything the model sees in one call: instructions, conversation so far, documents, its own answer. Nothing else exists for it. It has no memory between calls — any “memory” is text re-sent by the application.
Back to the release
Read it again. Which words are still dark?
Close
Homework 1, due Tue 29 Sep:
your starting point, in writing
the question you want answered by December
the tokenizer on five texts of yours
ten glossary words, in your words
three pitch-deck sentences to take apart
a Drive folder ese-ai-fintech, shared with me — see Setup
Questions: WhatsApp or Telegram.
Spare If there is time
Take it apart yourself
LAB · session.ipynb § B1 — tiktoken
one sentence in English, Italian, Russian
1,234,567.89
your surname, a listed company, a start-up
A taste of DeFi
Bet
A pool holds 10 ETH and 30,000 USDC — 3,000 per ETH on screen. You buy 1 ETH. What do you pay?
ETH × USDC stays constant: 10 × 30,000 = 300,000.
9 ETH left → 300,000 ÷ 9 = 33,333 USDC → you paid 3,333, 11% above the screen.
Why pieces, and not letters?
the same 5,000 characters, as
length
alphabet
bits
40,000
2
bytes
5,000
256
tokens
~1,300
~100,000
The network reads a sequence, and longer sequences cost more to process. Letters would make every text very long; whole words would need an impossible vocabulary. Pieces are the compromise: about 1,300 of them for 5,000 characters.
How a tokenizer is learned
Merge the most frequent pair. Repeat.
step
text
new symbol
start
aaabdaaabac
1
ZabdZabac
Z = aa
2
YbdYbac
Y = Za
3
XdXac
X = Yb
11 symbols → 5. Train it on English, and it learns English pieces.
This is byte-pair encoding: find the pair of symbols that appears most often, give it a new name, repeat. On English text the frequent pairs are English, so English ends up with long pieces and other languages with short ones — more pieces, higher cost.
Much of the weirdness is tokenization
it struggles with
because
spelling a word backwards
the word is one token, not letters
Russian, Italian
more, smaller pieces
arithmetic
1,234,567.89 is 7 arbitrary pieces
egg · Egg · EGG
the same word, different pieces
Many failures that look like stupidity come from the model seeing pieces, not letters or digits. When an answer involves spelling, counting characters or arithmetic, check it yourself.
