Making it answer from your documents

Week 4 · Wednesday 14 October 2026 · 10:00–13:00

Niccolò Salvini, PhD

AI for Business and FinTech · European School of Economics, Florence

Where we are

1–3 23 Sep – 7 Oct data, a model, and an honest evaluation ✓
4 14 Oct call a model from code; make it cite your documents
5 21 Oct agents, tools, evals · the product lens
— 1 Nov case brief, 40% — on what you build today and next week

From today you are building the thing you will be assessed on.

Last week, in one slide

A feature is what you know at t; the target is what happens after. The base rate is the score to beat.

A shuffled split let the future leak into training. Nothing to the right of the wall may fit what is on the left.

A probability is not a decision: the threshold comes from the cost of each kind of error.

Calibration: when the model says 70%, does it happen 70% of the time?

Block A Four knobs, and they are the whole API

What you control on every call

system prompt who the model is, and what it must never do
the prompt what you are asking, this time
temperature 0 when the answer must be reproducible
schema the shape the answer must have, validated

And three numbers come back with every response, whether you look or not:

tokens in · tokens out · latency. Log all three from the first call. In week 10 you will need them to say what this costs at volume, and by then nobody remembers.

Structured output guarantees the type, not the truth

A valid JSON object can be entirely wrong.

The schema stops your code crashing. It does nothing about whether the number in the field is real.

First calls

LAB · session.ipynb § A

  1. The same prompt at T = 0 three times, then at T = 1 three times. Look at what changed.
  2. Ask for free text. Then ask for a JSON schema. Compare how much code you need after each.
  3. Print tokens and latency for every call, and keep printing them all term.

Block B Making it answer from documents it has never seen

The problem, stated plainly

The model has never read your contract, your filing, or last month’s board pack. It was trained before they existed, and it cannot open a file.

Bet before we build

Two ways to fix this: paste the whole document into the prompt, or retrieve just the relevant part. When does the first one stop working?

At about the point where it becomes expensive, slow, and worse — models attend badly to the middle of very long inputs. Retrieval is the alternative, and it is four steps.

Four steps, and each one can fail differently

Why the nearest passage is the right one

Not one word in common, and still the three nearest. That is what an embedding buys you.

Build it, with two bugs planted

LAB · session.ipynb § B

Corpus: risk sections from three filings, two whitepapers, a MiCA summary.

  1. Chunk it. Bug one is in the chunking — a fact gets cut in half.
  2. Embed and retrieve. Print the passages that came back, every time, before the answer.
  3. Generate with citations. Bug two is in the prompt — the model is allowed to use what it already knows.

Ten questions you already know the answer to

LAB · session.ipynb § B6

Write ten questions with answers you computed independently of the system.

Run them. Record hits and misses. For each miss, write retrieval or generation next to it.

That table is your evidence section in the case brief. Build it now, not the night before.

Block C The EU AI Act, in thirty minutes

It regulates the use, not the technology

Close

Homework 4 — extend the retrieval system with a corpus of your own, write ten test questions with expected answers, record hits and misses, and explain each miss as retrieval or generation.

Due Tuesday 20 October, 23:59.

Next week the model stops answering and starts acting — and everything that can go wrong, does.