Week 2 — Inside an LLM

ESE · AI for Business and FinTech · 30 September 2026

The sections follow the slides. From section 3 you run a real language model, GPT-2 small (124 million weights), in this notebook.

!pip -q install transformers tiktoken

0. Last week, runningPayment E: large, abroad, same device — and it was fraud. The four steps from the slide: guess, how wrong, blame, nudge. Read each line’s comment; the numbers are the ones on the slide.

import numpy as npx = np.array([1, 1, 0])      # payment E: large = 1, abroad = 1, new device = 0y = 1                        # it was fraudw = np.array([0.5, 1.0, 1.0]); b = -1.0   # the weights the perceptron found last weeklr = 1.0                     # learning rate: how big each nudge isdef guess(w, b, x):    score = w @ x + b                   # weigh each flag, add them up (@ = multiply and add)    return 1 / (1 + np.exp(-score))     # squash the score into a probabilityp = guess(w, b, x)print("1 guess       p(fraud) =", round(p, 2))print("2 how wrong   loss     =", round(-np.log(p), 2))blame = (p - y) * xprint("3 blame       per flag =", blame.round(2), " bias:", round(p - y, 2))w, b = w - lr * blame, b - lr * (p - y)print("4 nudge       weights  =", w.round(2), " bias:", round(b, 2))print("  guess again p(fraud) =", round(guess(w, b, x), 2))

Repeat — the same four steps on all eight payments, 200 times. Watch the average loss fall.

X = np.array([[0,0,0],[1,0,0],[0,1,0],[0,0,1],[1,1,0],[1,0,1],[0,1,1],[1,1,1]])   # payments A–HY = np.array([0, 0, 0, 0, 1, 1, 1, 1])                                            # fraud = two flags or morew, b, lr = np.array([0.5, 1.0, 1.0]), -1.0, 3.0def avg_loss(w, b):    p = 1 / (1 + np.exp(-(X @ w + b)))    return np.mean(-np.log(np.where(Y == 1, p, 1 - p)))for step in range(201):    if step in (0, 10, 50, 200):        print(f"step {step:3}   average loss {avg_loss(w, b):.2f}")    p = 1 / (1 + np.exp(-(X @ w + b)))    w, b = w - lr * (p - Y) @ X / 8, b - lr * np.mean(p - Y)

1. Your tiny LLM, piece 1: a tokenizer

Find the pair of symbols that appears most often; give it a new name; repeat. This is byte-pair encoding, the method real tokenizers use.

from collections import Counter

def learn_merges(text, merges=8, show=True):
    seq = [c if c != " " else "_" for c in text]          # _ marks a space
    for step in range(1, merges + 1):
        pairs = Counter(zip(seq, seq[1:]))
        (a, b), n = max(pairs.items(), key=lambda kv: (kv[1], -list(pairs).index(kv[0])))
        if n < 2:
            break
        out, i = [], 0
        while i < len(seq):
            if i < len(seq) - 1 and (seq[i], seq[i + 1]) == (a, b):
                out.append(a + b); i += 2
            else:
                out.append(seq[i]); i += 1
        seq = out
        if show:
            print(f"merge {step}: {a!r} + {b!r} seen {n} times -> {a+b!r:10}  text is now {len(seq)} symbols")
    return seq

text = ("the bank left rates unchanged. the bank said rates will stay unchanged. "
        "rates are high and the bank expects rates to stay high.")
print(len(text), "characters to start")
pieces = learn_merges(text)

Your turn. A paragraph of your own in English, then the same in Russian. 30 merges each. Symbols per word?

mine_en = "paste an English paragraph here"
mine_ru = "вставьте сюда абзац на русском"
for name, t in [("English", mine_en), ("Russian", mine_ru)]:
    seq = learn_merges(t, merges=30, show=False)
    print(f"{name}: {len(seq) / len(t.split()):.1f} symbols per word")

What you pay for. The real tokenizer family of gpt-oss (o200k). Same sentence, two languages.

import tiktokenenc = tiktoken.get_encoding("o200k_base")for text in ["The ECB left rates unchanged at 2%.",             "ЕЦБ оставил ставки без изменений на уровне 2%."]:    ids = enc.encode(text)    print(f"{len(text.split())} words -> {len(ids)} tokens:", [enc.decode([t]) for t in ids])

2. Attention, by hand

“The bank raised rates because it” — it asks a question (query); every token shows a label (key) and carries a content (value). Two numbers per vector, so you can check every step.

import numpy as np, pandas as pd

tokens = ["The", "bank", "raised", "rates", "because", "it"]
query  = np.array([1.5, 0.0])                                    # "it": who is an actor that can fear?
keys   = np.array([[0, .2], [2, 0], [0, 1], [1, .5], [0, .5], [.2, 0]])
values = np.array([[0, 0], [1, 0], [0, 0], [0, 1], [0, 0], [0, 0]])   # (is an institution, is a price)

scores  = keys @ query                       # multiply and add: a neuron
weights = np.exp(scores) / np.exp(scores).sum()   # the knobs
print(pd.DataFrame({"score": scores, "e^score": np.exp(scores).round(2), "attention": weights.round(3)}, index=tokens))
print("\nnew meaning of 'it' =", (weights @ values).round(2), " (institution, price)")

Try: give rates the key (2, 0) too. Where does it look now — and what would a real model need to tell them apart?

3. A real model’s knobs

GPT-2 small: 124 million weights, 50,257 tokens, 12 blocks. The first run downloads it (about half a gigabyte).

import torch
from transformers import GPT2LMHeadModel, GPT2TokenizerFast
tok = GPT2TokenizerFast.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2", attn_implementation="eager")
model.train(False)                                   # inference mode
print(f"{sum(p.numel() for p in model.parameters()):,} weights")

def knobs(prompt, k=8):
    ids = tok(prompt, return_tensors="pt").input_ids
    with torch.no_grad():
        p = torch.softmax(model(ids).logits[0, -1], -1)
    top = torch.topk(p, k)
    print(prompt, "->")
    for v, i in zip(top.values, top.indices):
        print(f"   {tok.decode([int(i)])!r:14} {float(v):6.1%}")

knobs("The ECB left rates")

Bet first, then run: three prompts of your own — one from your work, one about markets, one about crypto. How sure is the model about the next word?

knobs("Bitcoin fell sharply after")

4. Temperature, and one token at a time

prompt = "The ECB left rates"
ids = tok(prompt, return_tensors="pt").input_ids

out = model.generate(ids, max_new_tokens=12, do_sample=False, pad_token_id=tok.eos_token_id)
print("T = 0  :", prompt + tok.decode(out[0][ids.shape[1]:]))

for T in [1.0, 1.0, 1.0, 2.0]:
    out = model.generate(ids, max_new_tokens=12, do_sample=True, temperature=T, top_k=0, pad_token_id=tok.eos_token_id)
    print(f"T = {T}:", prompt + tok.decode(out[0][ids.shape[1]:]))

A base model is not an assistant. GPT-2 was only pre-trained. Bet first: does it answer?

q = "What is the capital of France?"ids_q = tok(q, return_tensors="pt").input_idsout = model.generate(ids_q, max_new_tokens=30, do_sample=False, pad_token_id=tok.eos_token_id)print(q + tok.decode(out[0][ids_q.shape[1]:]))

Same model, before and after post-training. Qwen2.5-0.5B and its Instruct version: bet first — what will each write?

from transformers import AutoModelForCausalLM, AutoTokenizerquestion = "Should I buy Bitcoin today?"    # try "What is the capital of France?" toofor name in ["Qwen/Qwen2.5-0.5B", "Qwen/Qwen2.5-0.5B-Instruct"]:    # about 1 GB each, first run only    t = AutoTokenizer.from_pretrained(name)    m = AutoModelForCausalLM.from_pretrained(name)    if "Instruct" in name:                  # the assistant expects a conversation        text = t.apply_chat_template([{"role": "user", "content": question}], tokenize=False, add_generation_prompt=True)    else:                                   # the base model just continues the text        text = question    ids = t(text, return_tensors="pt").input_ids    out = m.generate(ids, max_new_tokens=40, do_sample=False, pad_token_id=t.eos_token_id)    print(f"--- {name}\n{t.decode(out[0][ids.shape[1]:], skip_special_tokens=True)}\n")

🔍 CHECK. Which of these sentences contain a number? Is any of those numbers a fact? How would you find out?

5. Meaning as a place

Every token is a row of 768 numbers. Cosine similarity: 1 means the same direction, 0 unrelated.

E = model.transformer.wte.weight.detach()
def similarity(a, b):
    va, vb = E[tok.encode(" " + a)[0]], E[tok.encode(" " + b)[0]]
    return float(torch.nn.functional.cosine_similarity(va, vb, dim=0))

for a, b in [("bank", "lender"), ("bank", "Paris"), ("Bitcoin", "crypto"), ("euro", "dollar")]:
    print(f"{a:8} – {b:8} {similarity(a, b):.2f}")

6. Two piles

Ask your assistant: “Summarise the last annual results of [a company you know]: revenue, profit, employees, the CEO’s name.” Split every sentence of the answer:

Verifiable — a number, date, name or source you can open Plausible — fluent, and nothing to check it against
… …

Check two items from the left pile against the company’s own report.

Before you leave: three lines, in your words

  1. …
  2. …
  3. …

Then File ▸ Save a copy in Drive, into your ese-ai-fintech folder.