100 topics · Read, check sources and follow your curiosity.

A little room to wonder

AI & Learning from Examples

Machine learning fits a model to examples. Its usefulness depends on the question, the data and how well it handles new cases beyond those examples.

Why can a model get every training example right and still fail on new data?

Overlapping paperclips appear in several colours and shapes.
Dietmar Rabich’s real paperclip photograph illustrates possible objects for a classification task. It is not the dataset used in the fictional length example, and colour alone does not establish a clip’s intended category.Dietmar Rabich · Source · CC BY-SA 4.0

Define the task and the examples

Machine learning is one approach within artificial intelligence. A model is fitted using data so it can make predictions or generate outputs. In supervised classification, examples include input features and labels describing the category the system is meant to predict.

Features are the recorded properties used as inputs. Labels come from a chosen task and collection process; they are not automatically complete or correct. Deciding what to classify and which features to record already shapes the result before a model is trained.

Keep learning and evaluation separate

A model can fit patterns in its training data that do not transfer well to new examples. Overfitting describes this problem. Training accuracy alone therefore does not show how a model will perform in use.

Separate validation and test data help examine generalisation. Repeatedly adjusting a model after looking at its final test results also weakens that test’s independence. The evaluation examples need to represent the intended task, not just be easy examples chosen to make a score look good.

Worked example: a fictional clip classifier

Imagine two fictional clip families, A and B. Training records give lengths in centimetres: 2→A, 3→A, 6→B, 7→B. A simple fitting rule places a threshold halfway between 3 and 6, at 4.5: predict A below 4.5 and B otherwise. All four training labels match this rule.

Now test unseen records: 2.5→A, 4→B, 5→A, 6.5→B. The model predicts A, A, B, B, so only the first and last are correct: 2 out of 4, or 50%. Length alone has not captured the family distinction. The records are invented, not measurements of the photograph, and this tiny score cannot estimate performance on all clips.

A fluent or confident output still needs checking

Different AI systems have different purposes; a classification score does not establish that a system understands a subject like a person. A generated explanation can sound plausible while containing unsupported claims. Compare important factual claims with appropriate evidence rather than treating confidence of wording as proof.

A small paper model can reveal these issues without opening an AI account or uploading data. Use fictional objects and labels, not personal profiles or information about other people. If a task changes, review whether the training examples, features and evaluation still fit it.

Only if you feel like trying

Test a second feature without changing the goal

  1. Recheck the four test predictions in the fictional example and calculate the score.
  2. Propose a second non-personal feature, such as the shape of the clip’s end, that might help distinguish the invented families.
  3. Explain what fresh examples you would need before claiming the new feature improves generalisation.

Optional. Work on paper, point to the example or talk it through. No account, purchase or public sharing is needed.