COMPSCI 714 — AI Architecture and Design: Exam Killer Book

How to Get an A in One Day (Feynman Method)

This book is reverse-engineered from every past exam (2024, 2025, Practice) with official marking schemes. Every concept is ranked by exam frequency.

Your One-Day Battle Plan (Feynman Whiteboard Method)

The Feynman Technique: Grab a blank sheet. Write the topic. Explain it out loud as if teaching a 12-year-old. When you get stuck, that's your gap. Go back, learn it, explain again.

Morning (3 hrs) — Build Understanding

Afternoon (3 hrs) — Practice Exam Questions

Evening (2 hrs) — Cheat Sheet + Final Review

What to Put on Your Cheat Sheet

Side 1 — Formulas & Calculations:

CNN CONV output:  floor((n + 2p - f) / s) + 1
CNN POOL output:  floor((n - f) / s) + 1
Valid padding: p = 0    Same padding: output = input size

Accuracy  = (TP + TN) / (TP + TN + FP + FN)
Precision = TP / (TP + FP)
Recall    = TP / (TP + FN)
F1        = 2 * P * R / (P + R)

Side 2 — Decision Trees & Key Points:

DIAGNOSIS FLOWCHART:
Train HIGH, Val LOW  → Overfitting (high variance)
  → Fix: regularisation, more data, data augmentation, smaller model
Train LOW, Val LOW   → Underfitting (high bias)
  → Fix: bigger model, more features, train longer, remove regularisation
Train HIGH, Val HIGH → Good fit!

OUTPUT ACTIVATION:
Multi-class (one label)    → Softmax
Multi-label (many labels)  → Sigmoid
Regression                 → Linear (no activation)

BATCH NORM EFFECTS: faster training, reduce vanishing gradients, 
regularisation effect, less sensitive to weight init

Exam Format

Golden rule: "Quality over quantity" — be concise. A 3-sentence precise answer beats a full-page ramble.

考前心理建设（Mental Preparation）

作为中国留学生，你的 ML 概念理解可能比很多本地学生都强。你唯一需要练的是：

1. 先说结论（不要铺垫）
2. 用题目的数字（不要泛泛而谈）
3. 连接词（because, therefore, however — 让逻辑清晰）
4. 不要怕犯语法错误（内容正确比语法完美重要100倍）

记住：考官打分看的是你理解不理解，不是你英语好不好。 一个有小语法错误但逻辑清晰的答案 >> 一个语法完美但空洞的答案。

Exam Question-by-Question Analysis

Source: 2025 S1 Test + 2024 S1 Test + Practice Test (with official answers)

2025 S1 Mid-Semester Test (20 marks, 6 questions)

Q1: Dataset Cleaning [2 marks]

The Trick: Attribute 4 has 9995/10000 missing values → remove entirely. Attribute 2 has extreme min/max relative to mean → outliers exist. No missing values in categorical/binary → no need for most-frequent imputation.

Q2: Evaluation and Design Choices [3 marks]

The Trick: Gap between training and validation = high variance (overfitting). Training loss still relatively high = possible high bias too. Each suggestion must target a DIFFERENT aspect.

Q3: Activation Functions [3 marks]

The Trick: Multi-label (multiple anomalies per image) = sigmoid (independent per output). NOT softmax (which forces probabilities to sum to 1).

Q4: Learning Rate [4 marks]

The Trick: Diverging (loss goes up) = 0.5. Slow descent = 0.001. Fast convergence to HIGH loss = 0.1 (overshoots optimum). Best convergence to LOW loss = 0.01.

Q5: Transformers [4 marks]

The Trick: (a) Mask prevents looking at future tokens → preserves autoregressive property during training. (b) [CLS] token aggregates info from all patches → efficient classification without processing all embeddings separately.

Q6: CNNs [4 marks]

Answer: 180. Pipeline: [35,35,3] → Conv1(valid,k=7,s=2) → [15,15,10] → Pool1(k=2,s=2) → [7,7,10] → Conv2(same,k=3,s=1) → [7,7,20] → Pool2(k=2,s=2) → [3,3,20] → Flatten = 180.

2024 S1 Mid-Semester Test (30 marks, 7 questions)

Q1: Data Preprocessing [4 marks]

Pipeline 1 (median imputer → standardisation → log transform): Numerical data, missing values, different scales, heavy-tailed distribution.

Pipeline 2 (most-frequent imputer → one-hot encoding): Categorical data, missing values, no ordinal relationship, not too many categories.

Q2: Design Choices [6 marks] — HIGHEST VALUE QUESTION

• More epochs: NO — worsens overfitting
• Larger dataset: YES — more diverse data helps generalise
• L2 regularisation: YES — penalises large weights, promotes simpler model

Q3: Evaluation [4 marks]

Results: Accuracy=60%, Recall=100%, Precision=56%. The model predicts almost everything as positive. Looks like it catches all positives (perfect recall) but actually just labels everything positive (terrible precision).

Q4: Learning Rate and Optimisers [4 marks]

Key answers: (1) Exponential decay — fast at start, fine-tune near optimum. (2) Momentum = exponentially decaying average of past gradients → smoother updates, speeds up convergence.

Q5: RNN and Transformer [4 marks]

Key: (1) Advantage: naturally captures order. Drawback: can't parallelise → slow for long sequences. (2) Transformer: processes all tokens in parallel via embeddings + adds positional encoding for order.

Q6: CNN Feature Map [4 marks]

Answers: Conv: ((50+0-5)/3)+1 = 16 → [16,16,10]. AvgPool: ((50-5)/5)+1 = 10 → [10,10,5]. MaxPool: same dimensions (only values differ).

Q7: DNN Training [4 marks]

Key: (1) Vanishing/exploding gradients + overfitting + longer training. (2) Batch norm, skip connections (ResNet), better optimisers (Adam), LSTM/GRU.

Practice Test (~32 marks, 7 questions)

Q1: Data Pre-processing [5 marks]

Two approaches to missing data + when to use each. Remove attribute (when mostly missing) or impute values (when reasonable amount missing).

Q2: DNN and Generalisation [5 marks]

High bias → underfitting → increase model, add data, transfer learning. High variance → overfitting → regularisation, more data, reduce model.

Q3: Design Choices [6 marks]

Underfitting scenario (train=val=50%). Increase size = YES. Zero init = NO (symmetry problem). Dropout = NO (regularisation doesn't help underfitting).

Q4: Evaluation [3 marks]

Accuracy=70%, Recall=33%. Accuracy misleading due to class imbalance — model bad at finding positives.

Q5: Batch Normalisation [5 marks]

Effects: speeds up training, reduces vanishing gradients, regularisation effect, reduces weight init sensitivity.

Q6: Attention and Transformers [4 marks]

Multi-head attention = stacks multiple attention heads with separate Q/K/V. Benefit: focuses on different aspects simultaneously.

Q7: CNNs [5 marks]

Reverse-engineer hyperparameters from diagram. Early layers = edge detectors (low-level features), deeper layers = complex features.

Exam Topic Frequency Map

The Heat Map: What WILL Be on Your Exam

Priority Guide

The 80/20 Rule: 4 Topics = ~65% of All Marks

1. Bias-Variance + Design Choices (~20% of all marks)

• Diagnose overfitting vs underfitting from numbers/curves
• For each fix: say YES/NO + link to the specific diagnosis
• Never confuse: regularisation fights overfitting, NOT underfitting

2. CNN Calculations (~15% of all marks)

• Two formulas: conv output + pool output
• Practice multi-layer pipeline calculations
• Know valid vs same padding

3. Transformer / Attention (~15% of all marks)

• Masked attention = prevent seeing future tokens
• Multi-head attention = multiple perspectives simultaneously
• ViT: patches → embeddings → [CLS] token → classification

4. Data Preprocessing (~15% of all marks)

• Which imputation for which data type
• When to remove attribute vs impute
• Read pipeline → infer data characteristics

+ 2 More for Safety (~20% more marks)

5. Learning Rate — curve shapes, momentum, LR schedules
6. Confusion Matrix — calculate accuracy/precision/recall, spot class imbalance traps

Total Marks by Topic (All Exams Combined)

Bias-Variance/DC    ████████████████████  20 marks
CNN                 █████████████         13 marks
Transformer         ████████████          12 marks
Data Preprocessing  ███████████           11 marks
Learning Rate       ████████              8 marks
Eval Metrics        ███████               7 marks
Batch Norm          █████                 5 marks
DNN Training        ████                  4 marks
RNN                 ████                  4 marks
Activation Func     ███                   3 marks

Teacher's Exam Style Analysis

Core Philosophy

"We privilege quality over quantity" — concise, clear, correct.

The teacher tests applied understanding, not memorisation. Every question gives a scenario and asks you to reason about it.

Question Patterns That Repeat Every Exam

Pattern 1: "Evaluate These Suggestions" (EVERY EXAM)

Format: Given model settings + results, evaluate 2-3 suggestions.

How to nail it:

1. FIRST: diagnose the problem (overfitting or underfitting?)
2. THEN: for each suggestion, say YES/NO
3. THEN: explain WHY by connecting to your diagnosis

Scoring: 2 marks each (1 for answer, 1 for reasoning connected to scenario)

Pattern 2: CNN Dimension Calculation (EVERY EXAM)

Format: Given architecture spec → compute output at each layer → find FC input size.

How to nail it: Write this for EVERY layer:

[Layer Name]
  Input:  [H, W, C]
  Formula: ((H + 2p - f) / s) + 1
  Output: [H', W', C']

Pattern 3: Transformer Two-Part Question (EVERY EXAM)

Format: (a) Explain mechanism X. (b) Why is it useful?

How to nail it: Part (a) = WHAT it does. Part (b) = WHY it matters (concrete benefit).

Pattern 4: Loss Curve / Metric Interpretation (2/3 EXAMS)

Format: Given graph or numbers → diagnose + suggest fix.

Traps the Teacher Sets (And How to Avoid Them)

Sentence Patterns in Questions → What They Want

Concepts That Always Appear Together

Bias-Variance ←→ Regularisation ←→ Design Choices
(one question covers all three — master the connections)

CNN Dimensions ←→ Valid/Same Padding ←→ FC Layer Size
(pipeline calculation from start to end)

Transformer ←→ Masked Attention ←→ Positional Encoding ←→ ViT
(mechanism + why it exists)

Loss Curves ←→ Learning Rate ←→ Optimizers
(visual diagnosis skill)

Confusion Matrix ←→ Class Imbalance ←→ Misleading Accuracy
(numbers game — always check precision AND recall)

Module A — Data Preprocessing

Exam Importance

MUST | Every exam has a data preprocessing question (2025 Q1, 2024 Q1, Practice Q1)

Feynman Draft

Imagine you're a chef and someone delivers raw ingredients to your kitchen. Some tomatoes are rotten (outliers), some boxes are missing labels (missing values), some ingredients are measured in grams while others are in kilograms (different scales). You can't cook with this mess — you need to clean and prepare everything first. That's data preprocessing.

The 4 things you might need to do:

1. Missing Values（缺失值） — Some cells in your spreadsheet are empty

   • Numerical data? → Fill with median (robust to outliers) or mean — this is called Imputation（插补/填补）
   • Categorical data? → Fill with most frequent value (mode)
   • Almost all missing? → Remove the entire column (attribute)

2. Outliers（异常值/离群值） — Values that are absurdly far from the rest

   • Look at: is max/min way larger than mean + a few standard deviations?
   • Example: mean=500, std=100, but max=50000 → definitely outliers

3. Scaling（缩放） — Features on different scales confuse the model

   • Standardisation（标准化） (z-score): $ x' = (x - \mu) / \sigma $ → mean=0, std=1
   • Use when features have different units/ranges

4. Encoding（编码） — Models need numbers, not text

   • One-hot encoding（独热编码）: turn "Red/Blue/Green" into [1,0,0], [0,1,0], [0,0,1]
   • Use for categorical data with NO natural ordering

Toy Example: Dataset with 10,000 samples:

Analysis:

• Most-frequent imputation? NO — binary and categorical have zero missing values
• Median imputation（中位数插补）? YES — Attr 2 (15 missing) and Attr 5 (23 missing) have some missing numerical values; median is better than mean because outliers exist
• Remove attribute（移除特征）? YES — Attr 4 has 9995/10000 missing → useless, imputation would create fake data
• Outlier removal（异常值移除）? YES — Attr 2's max (50000) is ~495 standard deviations from the mean!

Common Misconception: Students think "always impute" is the right answer. But if 99.95% of values are missing, imputation creates misleading data — just remove it.

Core Intuition: Preprocessing matches each data problem to the right cleaning tool — like choosing the right kitchen tool for each ingredient.

The Pipeline Reading Trick (2024 Exam Favourite)

The teacher loves giving you a pipeline diagram and asking "what does the raw data look like?"

Reverse-engineer the pipeline:

Example from 2024 Q1:

• Pipeline 1: median imputer → standardisation → log transform
  • → Numerical data, missing values, different scales, heavy-tailed
• Pipeline 2: most-frequent imputer → one-hot encoding
  • → Categorical data, missing values, no ordinal relationship

Past Exam Questions

2025 Q1 [2m]: Given dataset table, justify 4 cleaning steps (yes/no + why) 2024 Q1 [4m]: Given 2 pipelines, describe characteristics of raw data Practice Q1 [5m]: Describe 2 approaches to missing data + when each is preferred

中文思维 → 英文输出

本章 Chinglish 纠正

Whiteboard Self-Test

• Can you list 4 data cleaning operations and when to use each?
• Given a dataset summary table, can you justify each cleaning step?
• Given a pipeline diagram, can you describe what the raw data looks like?
• Do you know why median is preferred over mean for imputation with outliers?

Bias-Variance Tradeoff & Design Choices

Exam Importance

MUST | The single most tested topic: 4 questions across all exams, ~20 marks total

Feynman Draft

Imagine you're learning to throw darts at a bullseye.

• High bias（高偏差） = you consistently miss in the same direction. Your aim is systematically off. You're too rigid — like using only your wrist instead of your whole arm. This is underfitting（欠拟合） — your model is too simple to capture the real pattern.

• High variance（高方差） = your throws are scattered all over the board. Sometimes you hit the bullseye, sometimes the wall. You're too sensitive to tiny movements. This is overfitting（过拟合） — your model memorises the training data noise instead of learning the real pattern.

How do you diagnose this from training curves?

Toy Example with Numbers:

Common Misconception: "If validation accuracy is low, always add regularisation." WRONG! Regularisation helps overfitting (A), but makes underfitting (B) even WORSE because it constrains the model further.

Core Intuition: Bias（偏差） = model too simple for the problem. Variance（方差） = model too complex for the data amount.

The Design Choices Decision Tree (EXAM ESSENTIAL)

This is the teacher's favourite question format. Memorise this:

Step 1: DIAGNOSE
  Train >> Val? → Overfitting (high variance)
  Train ≈ Val ≈ low? → Underfitting (high bias)

Step 2: PRESCRIBE
  If OVERFITTING（过拟合）:
    ✅ Regularisation（正则化） (L1, L2, Dropout) → constrains model complexity
    ✅ More/diverse training data → helps generalise（泛化）
    ✅ Data augmentation（数据增强） → more variety without new data
    ✅ Batch normalisation（批量归一化） → regularising effect
    ✅ Early stopping（提前停止） → stop before overfitting
    ✅ Reduce model size → less capacity to memorise
    ❌ More epochs → makes it WORSE
    ❌ Bigger model → makes it WORSE

  If UNDERFITTING（欠拟合）:
    ✅ Increase model size (more layers/neurons) → more capacity（容量）
    ✅ Train longer → give it time to learn
    ✅ Remove/reduce regularisation → stop constraining
    ✅ Better features / more data → more signal
    ✅ Transfer learning（迁移学习） → start from pretrained model
    ❌ Regularisation → makes it WORSE
    ❌ Dropout → makes it WORSE
    ❌ Smaller model → makes it WORSE

Past Exam Questions with Answer Logic

2024 Q2 [6 marks] — Overfitting Scenario

Setup: 5 hidden layers, ReLU, 20 neurons/layer, 1000 epochs. Train=95%, Val=60%.

Practice Q3 [6 marks] — Underfitting Scenario

Setup: 2 hidden layers, ReLU, 5 neurons/layer, 2000 epochs, L1 regularisation. Train=50%, Val=50% (achievable=95%).

2025 Q2 [3 marks] — Curve Interpretation

Setup: Training curves after 20 epochs showing gap between train/val accuracy and diverging loss curves.

(a) Diagnose: High variance (overfitting) — clear gap between training and validation. Possibly also high bias if training loss is still high.

(b) Two changes (each targeting different aspect):

• Regularisation (e.g., L2, dropout) → reduces overfitting
• Data augmentation → more varied training data → better generalisation
• Batch normalisation → has regularising effect
• Increase model size (if bias is high) → more capacity to fit

How to Read Training Curves

Quick reference table:

English Expression Templates

Diagnosing:

• "The model displays high variance as there is a clear gap between training and validation accuracy."
• "This indicates overfitting, where the model fits the training data too closely but fails to generalise."

Prescribing:

• "Applying regularisation can help reduce overfitting by limiting model complexity."
• "Training on a larger dataset might help the model learn more general patterns."
• "This will not help because the model is already underfitting — adding regularisation would constrain it further."

中文思维 → 英文输出

本章 Chinglish 纠正

Whiteboard Self-Test

• Can you draw the bias-variance diagnosis table from memory?
• Given train=95%/val=55%, what's the diagnosis? What 3 things help?
• Given train=50%/val=50%, what's the diagnosis? Why does dropout NOT help?
• Can you explain why zero weight initialisation is bad?
• Can you explain why more epochs worsens overfitting?

Optimization: Learning Rate, Schedules & Optimizers

Exam Importance

HIGH | 2 out of 3 exams (2025 Q4, 2024 Q4) — 8 marks total

Feynman Draft

Imagine you're blindfolded on a hilly landscape, trying to find the lowest valley. You take steps downhill based on the slope you feel under your feet. That's Gradient Descent（梯度下降）.

The learning rate（学习率） is your step size:

• Too big (0.5): You leap so far you jump OVER the valley and end up on the other side, maybe even higher. Your loss goes UP. This is divergence（发散）.
• Too small (0.001): You take tiny baby steps. You'll eventually get there, but it takes forever. This is slow convergence（收敛缓慢）.
• Just right (0.01): You stride confidently into the valley. Fast convergence（收敛） to low loss.
• Slightly too big (0.1): You get near the valley but keep overshooting（超调） back and forth, settling at a suboptimal point.

The 4 Loss Curves — This Exact Question Was on 2025 AND 2024:

Common Misconception: "Curve 2 could be lr=0.001 because it gets stuck." While small lr CAN get stuck in local minima, the teacher's intended answer is: converging to a high loss = lr slightly too high (overshooting), not too low. Match the explanation consistently to the other curves.

Core Intuition: Learning rate controls step size — too big overshoots, too small is slow, just right converges fast to a good minimum.

Learning Rate Schedules（学习率调度） (2024 Q4.1)

What: Change the learning rate during training instead of keeping it fixed.

Why: Start with large steps (explore quickly), then shrink steps (fine-tune near optimum（最优点）).

Exam answer (1 example is enough): "Exponential learning rate decay reduces the lr as training progresses. This is beneficial because it allows taking large steps initially to move quickly towards an optimum, then smaller steps to avoid overshooting it."

Momentum（动量） (2024 Q4.2)

Analogy: Imagine pushing a ball downhill. Without momentum, the ball moves exactly where the current slope points — every tiny bump changes its direction. With momentum, the ball builds up speed and rolls smoothly past small bumps, heading in the general downhill direction.

Mechanism: Instead of updating weights（权重） using ONLY the current gradient（梯度）, momentum keeps a running average of past gradients:

$$v_t = \beta \cdot v_{t-1} + (1-\beta) \cdot \nabla L$$ $$w = w - lr \cdot v_t$$

Where $\beta$ (typically 0.9) controls how much past gradients matter.

Effects:

1. Smooths updates: Averages out noisy gradients → more stable direction
2. Accelerates convergence（加速收敛）: Builds up speed in consistent downhill directions
3. Escapes shallow local minima（局部最小值）: Momentum carries the ball through small bumps

Key Optimizers Quick Reference

Why Adam is the go-to optimizer (2024 Q7 — "better optimisers"): Adam adapts the learning rate for each parameter individually. Parameters with large gradients get smaller steps; parameters with small gradients get larger steps. This is especially helpful for deep networks where gradient magnitudes vary wildly across layers — it directly mitigates the vanishing/exploding gradient problem（梯度消失/梯度爆炸） at the optimiser level.

When SGD still wins: For very large-scale training (e.g., ImageNet), well-tuned SGD + momentum + lr schedule can generalise better than Adam. Adam sometimes converges to sharper minima, while SGD finds flatter (more generalisable) minima.

Past Exam Questions

2025 Q4 [4m]: Match 4 loss curves to learning rates 0.5, 0.1, 0.01, 0.001. Justify each. 2024 Q4 [4m]: (1) Give LR schedule example + why beneficial. (2) Explain momentum.

中文思维 → 英文输出

本章 Chinglish 纠正

Whiteboard Self-Test

• Can you draw 4 loss curves for different learning rates and label each?
• Can you explain why a diverging loss curve means the lr is too high?
• Can you name one LR schedule and explain why it helps?
• Can you explain momentum in your own words (not just the formula)?

Regularisation & Batch Normalisation

Exam Importance

HIGH | Tested directly (Practice Q5) and indirectly in every Design Choices question

Feynman Draft

Imagine you're studying for an exam. Overfitting（过拟合） is like memorising the textbook word-for-word — you ace the practice test but fail the real exam because you memorised answers instead of understanding concepts.

Regularisation（正则化） is like a study technique that forces you to actually understand: someone randomly covers parts of your notes (Dropout（随机失活）), or penalises you for writing overly complicated answers (L1/L2).

L1 and L2 Regularisation

Think of weights as "how much attention" the model pays to each feature.

• L2 (Ridge / 岭回归): Adds penalty proportional to weight² → pushes ALL weights to be small but non-zero — this is called weight decay（权重衰减）. Like telling someone "you can use all ingredients, but use them sparingly."

  $$L_{total} = L_{original} + \lambda \sum w_i^2$$

• L1 (Lasso): Adds penalty proportional to |weight| → pushes some weights to exactly 0. Like telling someone "pick only the most important ingredients and ignore the rest." Creates sparse（稀疏） models — performing automatic feature selection（特征选择）.

  $$L_{total} = L_{original} + \lambda \sum |w_i|$$

Dropout

During training, randomly "turn off" neurons with probability $p$ (typically 0.5). Forces the network to learn redundant representations — no single neuron can be relied on.

Key: Dropout is ONLY active during training. During inference, all neurons are used (but outputs are scaled by 1-p to compensate).

Batch Normalisation（批量归一化） (Practice Q5 — 5 marks)

What: Normalise（归一化） the inputs to each layer by subtracting mean and dividing by std of the current mini-batch（小批量）.

$$\hat{x} = \frac{x - \mu_{batch}}{\sqrt{\sigma^2_{batch} + \epsilon}}$$

Then apply learnable scale ($\gamma$) and shift ($\beta$): $y = \gamma \hat{x} + \beta$

4 Effects (know at least 2 for the exam):

Common Misconception: "Batch norm is just standardisation." No — it also has learnable parameters ($\gamma$, $\beta$) that let the network undo the normalisation if that's beneficial. And the normalisation per mini-batch introduces noise that has a regularising effect.

Core Intuition: Regularisation = purposely limiting model complexity to prevent memorisation and force generalisation.

When to Use What (Design Choices Context)

Early Stopping（提前停止）

What: Monitor validation loss during training. When it stops improving for $N$ consecutive epochs (patience), stop training — even if training loss is still decreasing.

Why it works: The point where validation loss starts rising is exactly the point where the model begins memorising training noise. Stopping there gives you the best generalisation.

In practice: Save a checkpoint of model weights at each validation improvement. When patience runs out, roll back to the best checkpoint.

L1 Sparsity（稀疏性） vs L2 Shrinkage（收缩） — Why the Difference?

Geometric intuition: L1's constraint region is a diamond (corners touch axes); L2's is a circle. The optimal point is where the loss contour（损失等高线） meets the constraint boundary. The diamond's sharp corners align with axes → weights are pushed to exactly 0. The circle has no corners → weights are pushed toward 0 but never reach it.

Practical consequence:

• Use L1 when you suspect many features are irrelevant (automatic feature selection)
• Use L2 when all features are somewhat useful (just reduce their magnitudes)
• The hyperparameter（超参数） λ controls regularisation strength（正则化强度）: higher λ = stronger penalty. If λ is too high → underfitting（欠拟合） (weights too constrained); too low → minimal regularisation effect.

Critical exam trap: If the model is underfitting (train=val=low), adding regularisation makes it WORSE by further constraining the model.

Past Exam Questions

Practice Q5 [5m]: Explain 2 effects of batch normalisation (2 marks each: name + explanation). 2024 Q2: L2 regularisation as a suggestion for overfitting → YES, explain why. Practice Q3: Dropout as a suggestion for underfitting → NO, explain why. 2025 Q2b: Suggest changes for overfitting → regularisation is a valid answer.

中文思维 → 英文输出

本章 Chinglish 纠正

Whiteboard Self-Test

• Can you explain L2 regularisation in one sentence?
• Can you explain why dropout doesn't help underfitting?
• Can you list 4 effects of batch normalisation?
• Can you explain the regularisation effect of batch norm (why mini-batch noise helps)?

MLP, Activation Functions & DNN Training

Exam Importance

MED | 2025 Q3 (activation functions), 2024 Q7 (DNN training challenges)

Feynman Draft: Neural Networks

Imagine a team of workers in a factory assembly line. Each worker receives inputs, does a simple calculation (multiply by a weight, add a bias), and passes the result through a "decision gate" (activation function（激活函数）) to the next worker.

One worker can only draw a straight line to separate things. But stack many workers in layers, and they can draw incredibly complex boundaries — that's a Deep Neural Network (DNN)（深度神经网络）.

Activation Functions (2025 Q3)

ReLU and the Dying ReLU Problem

ReLU (Rectified Linear Unit): $f(x) = \max(0, x)$

The problem: If a neuron's input is always negative, ReLU outputs 0 and its gradient is also 0. The neuron stops learning permanently — this is the Dying ReLU Problem（神经元死亡问题）. Look at the gradient plot above: the flat zero region on the left is the dead zone.

LeakyReLU (The Fix)

$$\text{LeakyReLU}(x) = \begin{cases} x & \text{if } x > 0 \ \alpha x & \text{if } x \leq 0 \end{cases}$$

How it helps: The small slope $\alpha$ (typically 0.01–0.1) means the gradient is never zero — dead neurons can recover.

Output Activation（输出激活函数）: Sigmoid vs Softmax (2025 Q3b)

2025 Q3b scenario: Manufacturing quality control — single image may contain multiple anomaly types simultaneously. This is multi-label → use sigmoid, because each anomaly is predicted independently.

Common Misconception: "Always use softmax for classification." Only when it's MULTI-CLASS (one label). For multi-label, softmax is wrong because it forces outputs to sum to 1 — detecting one anomaly would lower the probability of detecting others.

Why Deep Networks Are Hard to Train (2024 Q7)

The Problems:

1. Vanishing Gradients（梯度消失）: During backpropagation（反向传播）, gradients are multiplied through many layers via the chain rule. With sigmoid activation, the maximum derivative is only 0.25 — so gradients are multiplied by ≤0.25 at each layer. After 6 layers: $0.25^6 ≈ 0.0002$ — the gradient reaching early layers is nearly 0, so they can't learn. ReLU fixes this because its derivative is exactly 1 for positive inputs, preserving gradient magnitude perfectly.

2. Exploding Gradients（梯度爆炸）: Same multiplication, but with values > 1 → gradient grows exponentially → training becomes unstable.

3. Overfitting（过拟合）: More parameters = more capacity to memorise training data noise.

4. Longer training time: More computations per forward/backward pass.

The Solutions (name 2 for the exam):

Weight Initialisation（权重初始化）: Why Zero = Bad (Practice Q3)

If all weights are 0, then:

1. All neurons compute the same output (0)
2. All gradients are the same
3. All weights update by the same amount
4. All neurons remain identical forever → symmetry problem（对称性问题）

The network is essentially a single neuron repeated N times. It can't learn different features.

Correct initialisation: Random values, properly scaled:

• Xavier/Glorot: For sigmoid/tanh: $\text{Var}(w) = 1/n_{in}$
• He: For ReLU: $\text{Var}(w) = 2/n_{in}$

Why two different methods? Each is designed to keep the variance of activations stable across layers for a specific activation function:

• Xavier assumes the activation is roughly linear around 0 (true for sigmoid/tanh near their centre). It balances forward and backward signal variance.
• He accounts for the fact that ReLU kills half the inputs (outputs 0 for negative), so it doubles the variance to compensate. Using Xavier with ReLU → activations shrink to 0 in deep networks. Using He with sigmoid → activations may saturate.

Rule of thumb: Match initialisation to activation — He for ReLU/LeakyReLU, Xavier for sigmoid/tanh.

Architecture Diagram

中文思维 → 英文输出

本章 Chinglish 纠正

Whiteboard Self-Test

• Can you explain the dying ReLU problem and how LeakyReLU fixes it?
• When do you use sigmoid vs softmax for the output layer?
• Can you name 2 reasons why deep networks are hard to train?
• Can you name 2 solutions that make deep training easier?
• Why is initialising weights to 0 a bad idea?

CNN — Convolutional Neural Networks

Exam Importance

MUST | Every exam has a CNN calculation question (2025 Q6, 2024 Q6, Practice Q7)

Feynman Draft

Imagine you're looking at a photo and trying to find a cat. You don't examine every pixel individually — your eyes scan small regions looking for patterns: edges, then curves, then ears, then a face. A CNN works exactly like this.

A CNN（卷积神经网络） slides small "windows" (filters/kernels（卷积核/滤波器）) across the image. Each filter detects a specific pattern:

• Layer 1 filters: detect simple edges (horizontal, vertical, diagonal)
• Layer 2 filters: combine edges into shapes (corners, curves)
• Layer 3+ filters: combine shapes into objects (ears, eyes, faces)

After sliding filters, we shrink the image with pooling（池化） (like zooming out) to focus on "where" a pattern exists rather than its exact pixel position.

Toy Example: A 5x5 image with a 3x3 filter（特征图 = feature map）

Common Misconception: "More filters = bigger output feature map." NO — more filters increases the DEPTH (channels), not the spatial dimensions. Spatial size depends on kernel size, stride, and padding.

Core Intuition: CNN = sliding pattern detector. Shallow layers find edges, deep layers find objects.

Architecture Overview

The Two Formulas You MUST Memorize

Formula 1: Convolution Output Size

$$\text{output} = \left\lfloor \frac{n + 2p - f}{s} \right\rfloor + 1$$

Where:

• $n$ = input spatial dimension (height or width)
• $p$ = padding（填充） (valid = 0, same = computed so output = input)
• $f$ = filter/kernel size
• $s$ = stride（步幅）

Output depth = number of filters $n'_C$

Formula 2: Pooling Output Size

$$\text{output} = \left\lfloor \frac{n - f}{s} \right\rfloor + 1$$

Where:

• $n$ = input spatial dimension
• $f$ = pool kernel size
• $s$ = stride (usually = f)

Output depth = same as input depth (pooling doesn't change channels!)

Key difference: Pooling has NO padding (p=0 always).

Padding Types

Same padding shortcut: When stride = 1 and same padding → output spatial dimensions = input spatial dimensions. Just change the depth to the number of filters.

Worked Example: 2025 Q6 (The Exact Exam Question)

Architecture:

• Input: [35, 35, 3]
• Conv1: 10 filters, kernel=7, stride=2, valid padding
• MaxPool1: kernel=2, stride=2
• Conv2: 20 filters, kernel=3, stride=1, same padding
• MaxPool2: kernel=2, stride=2
• FC layer: ? inputs, 10 outputs

Step-by-step:

Layer: Conv1 (valid, p=0)
  Input:  [35, 35, 3]
  Calc:   (35 + 2*0 - 7) / 2 + 1 = 28/2 + 1 = 14 + 1 = 15
  Output: [15, 15, 10]    ← 10 from number of filters

Layer: MaxPool1
  Input:  [15, 15, 10]
  Calc:   (15 - 2) / 2 + 1 = 13/2 + 1 = 6.5 + 1 = 7.5 → floor = 7
  Output: [7, 7, 10]      ← depth unchanged

Layer: Conv2 (same padding, stride=1)
  Input:  [7, 7, 10]
  Calc:   same padding + stride 1 → spatial stays same
  Output: [7, 7, 20]      ← 20 from number of filters

Layer: MaxPool2
  Input:  [7, 7, 20]
  Calc:   (7 - 2) / 2 + 1 = 5/2 + 1 = 2.5 + 1 = 3.5 → floor = 3
  Output: [3, 3, 20]      ← depth unchanged

Flatten（展平）: 3 × 3 × 20 = 180

Answer: (ii) 180 ✓

Worked Example: 2024 Q6

1) Conv: Input [50,50,5], ten 5×5×5 filters, stride=3, padding=0

(50 + 2*0 - 5) / 3 + 1 = 45/3 + 1 = 15 + 1 = 16
Output: [16, 16, 10]

2) AvgPool: Input [50,50,5], 5×5 filter, stride=5

(50 - 5) / 5 + 1 = 45/5 + 1 = 9 + 1 = 10
Output: [10, 10, 5]   ← depth stays 5!

3) MaxPool: Same answer as AvgPool! Max vs average only changes VALUES, not dimensions.

Worked Example: Practice Q7

Given: Input [21,21,3] → Conv (no padding, s=2) → Output [9,9,100]

Find: n'C, f, nC

• n'C = 100 (depth of output = number of filters)
• f: (21 + 0 - f)/2 + 1 = 9 → (21 - f)/2 = 8 → 21 - f = 16 → f = 5
• nC = 3 (depth of input = filter width/depth)

Edge detection question: Early layers (close to input) detect edges because they see small local regions (receptive field（感受野）). Deeper layers combine these into complex features (shapes → objects).

Key Facts to Remember

中文思维 → 英文输出

本章 Chinglish 纠正

Whiteboard Self-Test

• Can you write both formulas from memory?
• Can you compute: Input [28,28,1] → Conv(16 filters, k=5, s=1, valid) → ?
• Can you compute: [24,24,16] → MaxPool(k=2, s=2) → ?
• What's the difference between valid and same padding?
• Why does max pooling not change the depth?
• Which layers detect edges? Why?

RNN / LSTM / GRU — Recurrent Neural Networks

Exam Importance

MED | Tested in 2024 Q5 (alongside Transformer comparison) — 4 marks

Feynman Draft

Imagine you're watching a movie and trying to understand the plot. You don't forget everything after each scene — you carry a running memory of what happened before. When a character says "he went back to the castle," you remember who "he" is and which castle from earlier scenes.

That's exactly what an RNN (Recurrent Neural Network)（循环神经网络） does. It processes a sequence (words, time steps, video frames) one element at a time, and passes a hidden state（隐藏状态） from one step to the next — like your running memory of the movie.

Input:    x₁        x₂        x₃        x₄
           ↓          ↓          ↓          ↓
State:  → [h₁] →→ [h₂] →→ [h₃] →→ [h₄] → output
           
Each box takes BOTH the current input AND the previous hidden state.
h₂ = f(W·h₁ + U·x₂ + b)

The Sequential Processing Trade-off (The Exact Exam Question — 2024 Q5):

Advantage: Because the RNN uses sequential processing（顺序处理）, processing one step at a time, it naturally captures the order of the sequence. You don't need to tell it "this word comes first, that word comes second" — it inherently knows because it processes them in order. The sequential structure IS the ordering mechanism.

Drawback: Because each step MUST wait for the previous step's hidden state to finish, you cannot parallelise（并行化） the computation. For a sequence of length 1000, you need 1000 sequential operations. This makes training very slow for long sequences.

Additionally, the hidden state must carry ALL information from the past through a single vector — for very long sequences, early information gets "washed out." This is related to the vanishing gradient problem（梯度消失问题）.

Why Vanilla RNNs Struggle with Long Sequences

During backpropagation through time (BPTT)（时间反向传播）, gradients are multiplied by the recurrent weight matrix at each time step:

$$\frac{\partial h_t}{\partial h_1} = \prod_{i=1}^{t-1} \frac{\partial h_{i+1}}{\partial h_i}$$

If these partial derivatives are < 1 → gradients vanish (early parts of the sequence get no learning signal) If they are > 1 → gradients explode (training becomes unstable)

Practical impact: A vanilla RNN trained on a 100-word sentence might "forget" what happened in the first few words by the time it reaches the end.

LSTM — Long Short-Term Memory

LSTM solves the vanishing gradient problem by adding gating mechanisms（门控机制） that control information flow:

┌──────────── Cell State (highway of information) ──────────────┐
│                                                                │
│    ┌─────┐      ┌─────┐      ┌─────┐                         │
│    │Forget│      │Input │      │Output│                        │
│    │ Gate │      │ Gate │      │ Gate │                        │
│    └──┬──┘      └──┬──┘      └──┬──┘                         │
│       │            │            │                              │
└───────┴────────────┴────────────┴──────────────────────────────┘

The cell state（细胞状态） acts like a conveyor belt — information can flow through unchanged if the gates allow it. This creates a direct path for gradients to flow back through time without being multiplied at each step → solves vanishing gradients.

GRU — Gated Recurrent Unit

GRU is a simplified version of LSTM with only 2 gates:

GRU vs LSTM: GRU has fewer parameters → faster to train, sometimes performs just as well. LSTM is more expressive for complex long-range dependencies.

How Transformers Fix the RNN Problem (2024 Q5.2)

The key answer for 2024 Q5.2:

1. The Transformer processes ALL input positions simultaneously using embeddings (not sequentially) → enables parallel computation → much faster
2. But this loses order information → solved by adding positional encoding to the embeddings
3. Self-attention creates direct connections between any two positions → no vanishing gradient over distance

Exam Answer Template for 2024 Q5

(1) Why is sequential processing both an advantage and drawback?

"Sequential processing is an advantage because it naturally captures the order of the input sequence — each hidden state implicitly encodes position information based on the processing order. However, it is also a drawback because each step depends on the previous hidden state, making it impossible to parallelise computation. For long sequences, this leads to very slow training and inference times."

(2) How does the Transformer alleviate this?

"The Transformer architecture processes all input positions in parallel by creating embeddings for each token simultaneously, rather than processing them sequentially. This dramatically speeds up computation. However, since parallel processing loses positional information, the Transformer adds positional encoding to the embeddings to integrate information about the sequence order."

Architecture Diagrams

RNN Unrolled Through Time:

LSTM Cell — Three Gates:

中文思维 → 英文输出

本章 Chinglish 纠正

Whiteboard Self-Test

• Can you draw the basic RNN unrolled diagram (input → hidden state → next step)?
• Can you explain sequential processing as BOTH advantage and drawback?
• Can you name the 3 gates in LSTM and what each does?
• Can you explain how the Transformer solves the parallelisation problem?
• Why do vanilla RNNs have trouble with long sequences?

Transformer & Attention Mechanism

Exam Importance

MUST | Every exam has a Transformer question (2025 Q5, 2024 Q5, Practice Q6)

Feynman Draft

Imagine you're reading a long book and someone asks: "What did the main character feel about the letter?"

You don't re-read every word. You skim for relevant parts — you pay more attention to sentences about the character and the letter, and less attention to descriptions of the weather. That's Attention（注意力机制）.

Now imagine you have 8 friends, and each one reads the book looking for something different: one tracks emotions, one tracks characters, one tracks locations, one tracks time. Then they share notes. That's Multi-Head Attention（多头注意力）— multiple "perspectives" on the same input.

The Transformer's Big Idea:

RNNs read words one by one (like reading a book left to right, can't skip ahead). This is slow. The Transformer reads ALL words at once (like seeing the whole page), then uses Attention to figure out which words relate to which. Much faster.

But wait — if you see all words at once, you lose the order! "Dog bites man" ≠ "Man bites dog". Solution: add Positional Encoding（位置编码）— a signal that tells the model "this word is in position 1, this one is position 2..."

Toy Example: "The cat sat on the mat"

With attention, when processing "sat", the model assigns weights:

"The"  → 0.05  (not very relevant)
"cat"  → 0.60  (WHO sat? very relevant!)
"sat"  → 0.10  (itself)
"on"   → 0.05  (grammar word)
"the"  → 0.05  (not very relevant)
"mat"  → 0.15  (WHERE sat? somewhat relevant)

Common Misconception: "Transformers are just faster RNNs." No — they work fundamentally differently. RNNs process sequentially (maintaining hidden state). Transformers process all positions in parallel (using attention to find relationships).

Core Intuition: Attention = learned "relevance weighting" between all pairs of inputs, processed in parallel.

Formal Definition: Scaled Dot-Product Attention

Scaled Dot-Product Attention（缩放点积注意力）:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V$$

Where:

• Q (Query): "What am I looking for?"
• K (Key): "What do I contain?"
• V (Value): "What information do I provide?"
• $d_k$: dimension of keys (scaling factor to prevent huge dot products)

The softmax creates attention weights (sum to 1) → multiply by V to get weighted combination.

Multi-Head Attention (考试高频)

Instead of one attention function, run h attention heads in parallel, each with its own learned Q, K, V weight matrices:

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h) W^O$$

Why multiple heads?

• Each head learns to focus on different aspects (syntax, semantics, position)
• Single head would have an averaging effect — blurs different types of relationships
• Multiple heads capture richer, more diverse patterns

Masked Attention in Decoder (2025 Q5a, 2024 Q5a)

What: In the decoder, when predicting token at position $t$, the attention is masked to prevent looking at positions $t+1, t+2, ...$

Why: During training, all tokens are available (teacher forcing（教师强迫）), but the model must learn to predict WITHOUT seeing the future. The mask sets future positions to $-\infty$ before softmax → attention weights become 0 for future tokens.

In plain English: It's like covering the right half of the answer sheet during an exam — you can only see what you've already written, not what comes next. This preserves the autoregressive（自回归） property: each prediction depends only on previous predictions.

Without mask:        With mask:
"I love cats"       "I love cats"
 ↕  ↕  ↕             →  →  →    (can only look LEFT)
All attend to all   Each token only attends to previous tokens

Vision Transformer (ViT) — Full Pipeline (2025 Q5b)

The Core Idea

CNNs use sliding filters to process images. ViT asks: what if we just cut the image into patches（图像块） and feed them into a standard Transformer? It turns out this works — and for large datasets, ViT matches or beats CNNs.

The ViT Pipeline (Step by Step)

Concrete Example: 224 × 224 image, patch size = 16 × 16

Step 1: Split into patches
   224 / 16 = 14 patches per side → 14 × 14 = 196 patches total
   Each patch is 16 × 16 × 3 (RGB) = 768 values

Step 2: Linear projection (patch embedding)
   Each patch (768 values) → linearly projected to a D-dimensional vector
   This is NOT just flattening — it's a learned linear layer
   Output: 196 vectors of dimension D

Step 3: Prepend [CLS] token
   Add one learnable vector at position 0
   Sequence is now: [CLS], patch_1, patch_2, ..., patch_196
   Total: 197 tokens

Step 4: Add positional embeddings
   Each of the 197 positions gets a learnable positional embedding (added, not concatenated)
   Without this: the model can't distinguish top-left patch from bottom-right

Step 5: Pass through Transformer encoder
   Standard encoder: Multi-Head Self-Attention → Add & Norm → FFN → Add & Norm
   Repeated N times (ViT-Base uses N=12)

   "Add & Norm" explained:
   - ADD = residual/skip connection: output = sublayer(x) + x
     Why: gradient flows directly through the '+' → prevents vanishing gradients in deep models
   - NORM = Layer Normalisation: normalise across features for each token
     Why: keeps activations stable → faster, more stable training

Step 6: Classification
   Take ONLY the [CLS] token's output → pass through MLP head → class prediction

Why Patches Instead of Pixels?

Self-attention complexity is O(n²) where n = number of tokens.

Patches reduce the sequence length by a factor of 256, making attention computationally tractable.

The [CLS] Token — What and Why

What: A special learnable embedding prepended to the patch sequence. It has no image content initially — it starts as random values and is learned during training.

How it works: Through self-attention across all encoder layers, the [CLS] token gradually aggregates information from ALL patches into a single global representation — like a "summary" token.

Why not just use all patch outputs? You could (some variants use Global Average Pooling over all patch embeddings instead). But [CLS] is more efficient: the MLP classification head only needs to read one vector instead of processing 196 vectors.

ViT vs CNN — Key Differences (Likely Exam Comparison)

Common Misconception: "ViT is always better than CNN." Wrong — ViT only beats CNN when trained on large datasets (e.g., ImageNet-21k, JFT-300M). On small datasets, CNN's inductive bias gives it a significant advantage. This is why ViT models are typically pre-trained on large data then fine-tuned on smaller target datasets.

Core Intuition: ViT trades CNN's built-in assumptions (locality, translation invariance) for the Transformer's flexibility — this pays off only when you have enough data to learn those patterns from scratch.

RNN vs Transformer (2024 Q5)

Exam answer structure for 2024 Q5:

1. Advantage of sequential: RNNs naturally capture sequence order through their step-by-step processing — no extra mechanism needed.
2. Drawback of sequential: Can't parallelise → slow for long sequences. Each step must wait for the previous one.
3. How Transformer fixes it: Uses embeddings to represent all positions at once (parallel), then adds positional encoding to restore order information that would otherwise be lost.

Past Exam Questions Summary

English Expression Templates

Explaining attention:

• "The attention mechanism allows the model to focus on the most relevant parts of the input sequence when making predictions."
• "Attention computes a weighted sum of values, where weights reflect the relevance of each input position."

Explaining masked attention:

• "Masking prevents each position from attending to future tokens, ensuring predictions depend only on known outputs."
• "This preserves the autoregressive property during training."

Explaining multi-head:

• "Multi-head attention runs several attention functions in parallel, each focusing on different aspects of the input."
• "This is beneficial because a single head would have an averaging effect over all types of relationships."

Architecture Diagrams

Transformer Encoder Block:

Scaled Dot-Product Attention:

中文思维 → 英文输出

本章 Chinglish 纠正

Whiteboard Self-Test

• Can you draw the Transformer encoder block (self-attention → add&norm → FFN → add&norm)?
• Can you explain Q, K, V in one sentence each?
• Can you explain masked attention and WHY it's needed?
• Can you explain the [CLS] token in ViT?
• Can you explain why multi-head attention is better than single-head?
• Can you compare RNN vs Transformer in 3 bullet points?

Evaluation Metrics — Confusion Matrix & Beyond

Exam Importance

HIGH | 2 out of 3 exams (2024 Q3, Practice Q4) — 7 marks total

Feynman Draft

Imagine you're a doctor testing patients for a disease. You have 100 patients. Your test says 25 are sick. But how good is your test really?

The Confusion Matrix（混淆矩阵） breaks down exactly what happened:

                    Actually Sick    Actually Healthy
Test says "Sick"     TP (True Pos)    FP (False Pos)（假阳性/误报） ← "False alarm"
Test says "Healthy"  FN (False Neg)   TN (True Neg)（真阴性）      ← "Missed case"
                     （真阳性）        （假阴性/漏报）

The 3 key metrics:

Toy Example (2024 Q3):

                True Positive    True Negative
Predicted Pos      500              400
Predicted Neg        0              100

• Accuracy = (500 + 100) / 1000 = 60%
• Precision = 500 / (500 + 400) = 500/900 = 55.6%
• Recall = 500 / (500 + 0) = 100%

Interpretation (this is worth marks!): The model has perfect recall (catches ALL positives) but terrible precision (55.6% of its "positive" predictions are wrong). It achieves this by predicting almost everything as positive — like a doctor who tells every patient they're sick. That's not useful!

Common Misconception: "High accuracy = good model." WRONG! If 99 out of 100 patients are healthy and your model always says "healthy", accuracy = 99%. But recall = 0% — you missed every sick person. Always check precision AND recall, especially with imbalanced classes.

Toy Example 2 (Practice Q4):

                True Positive    True Negative
Predicted Pos        5               20
Predicted Neg       10               65

• Accuracy = (5 + 65) / 100 = 70%
• Recall = 5 / (5 + 10) = 5/15 = 33.3%

Interpretation: Accuracy looks decent (70%), but recall is awful (33%). The model only finds 1 in 3 sick patients. This is because the data is class imbalanced（类别不平衡） — 85 negatives vs 15 positives. The model learns to mostly predict "negative" because that's the safe bet for accuracy.

The Class Imbalance Trap (Exam Favourite)

Pattern the teacher uses: Give you a confusion matrix where accuracy looks "OK" but precision or recall reveals the model is actually terrible at one class.

How to spot it:

1. Calculate all metrics
2. Check if positives and negatives are balanced
3. If one class dominates → accuracy is misleading → look at per-class metrics

How to answer the "What do you think?" question:

1. State the numbers (accuracy, precision, recall)
2. Observe: "The model is [good/bad] at classifying [positive/negative] examples"
3. Explain: "This is because [the model predicts most examples as X / the classes are imbalanced]"
4. Conclude: "If we care about [finding positives/avoiding false alarms], this model [does well / performs poorly]"

Quick Reference: All Formulas

$$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$

$$\text{Precision} = \frac{TP}{TP + FP}$$

$$\text{Recall (Sensitivity)} = \frac{TP}{TP + FN}$$

$$\text{F1 Score（F1分数）} = \frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

When to use F1: When you care about BOTH precision AND recall equally but the classes are imbalanced. F1 is the harmonic mean（调和平均数） — it penalises extreme imbalance between precision and recall. A model with precision=100%, recall=1% gets F1≈2%, not 50.5%.

Example from Mock Exam 2: Precision=70%, Recall=70% → F1 = 2×0.7×0.7/(0.7+0.7) = 70%. Equal precision and recall → F1 equals both. But if Precision=90%, Recall=10% → F1 = 2×0.9×0.1/(0.9+0.1) = 18% — F1 reveals the model is bad despite high precision.

Memory trick:

• Precision = "of all my Positive predictions, how many were right?" (P for Predicted)
• Recall = "of all Real positives, how many did I find?" (R for Real)

English Expression Templates

Calculating:

• "The accuracy is (TP+TN)/(TP+TN+FP+FN) = ..."

Interpreting:

• "The model achieves high recall but low precision, indicating it predicts most instances as positive."
• "Despite the seemingly acceptable accuracy of 70%, the model performs poorly at identifying positive instances, with a recall of only 33%."
• "This discrepancy is due to class imbalance in the dataset."

Confusion Matrix Diagram

中文思维 → 英文输出

本章 Chinglish 纠正

Whiteboard Self-Test

• Can you draw a confusion matrix and label TP, TN, FP, FN?
• Can you calculate accuracy, precision, and recall from any matrix?
• Can you explain why 70% accuracy might actually be a bad model?
• What does recall=100% with precision=56% tell you about the model?

✍️ Output Practice — Express It Yourself

How to use this chapter:

1. Pick a topic you just reviewed
2. Close all notes
3. Write your answer on paper or in a blank document — do NOT type into this page
4. Only open the self-check and reference answer after you have finished writing

The goal is not to get it perfect. The goal is to discover what you think you know but cannot yet say.

A1 — Bias-Variance Tradeoff

Level 1 — Core Intuition (30 seconds)

Prompt: What is overfitting? Explain it without using the word "overfitting."

Level 2 — Exam Paragraph (3 minutes)

Prompt: A model achieves 95% training accuracy but only 62% validation accuracy. Diagnose the problem and suggest two solutions. Explain why each solution works.

🧠 中文思路：95% vs 62% → 差距大 → 过拟合 → 两个解决办法各写一句为什么有效

Level 3 — Analysis (5–7 minutes)

Prompt: Your colleague says: "Whenever validation loss rises, just add dropout." Evaluate this advice. When is it correct? When does it backfire?

🧠 中文思路：先说什么时候对（过拟合时）→ 再说什么时候错（欠拟合时）→ 用 train/val 数字区分两种情况

A2 — Optimisation & Learning Rate

Level 1 — Core Intuition (30 seconds)

Prompt: What does the learning rate control? Explain it without using a formula.

Level 2 — Exam Paragraph (3 minutes)

Prompt: A training loss curve oscillates wildly and never converges. What is the most likely cause? How would you fix it?

🧠 中文思路：loss震荡不收敛 → 学习率太大 → 解释为什么（步子太大跳过最优点）→ 怎么修

Level 3 — Analysis (5–7 minutes)

Prompt: Compare SGD, SGD with momentum, and Adam. In what situation would you prefer each?

🧠 中文思路：三个优化器各写一段 → SGD最简单 → Momentum加了动量 → Adam自适应 → 各说什么时候好用

A3 — Regularisation

Level 1 — Core Intuition (30 seconds)

Prompt: What is L2 regularisation doing to the weights, in plain English?

Level 2 — Exam Paragraph (3 minutes)

Prompt: Explain two distinct effects of batch normalisation on model training.

🧠 中文思路：Batch norm两个效果 → 每个写"做了什么"+"为什么有用" → 不要和dropout搞混

Level 3 — Analysis (5–7 minutes)

Prompt: A model is underfitting (train=55%, val=54%). A teammate suggests adding dropout. Evaluate this suggestion.

🧠 中文思路：先判断欠拟合 → 然后说dropout会让情况更糟 → 因为dropout减少了模型容量 → 最后说应该怎么做

A4 — Data Preprocessing

Level 1 — Core Intuition (30 seconds)

Prompt: Why is median imputation preferred over mean imputation when a feature has outliers?

Level 2 — Exam Paragraph (3 minutes)

Prompt: A preprocessing pipeline contains: median imputer → standard scaler → log transform. What does this tell you about the raw data?

🧠 中文思路：从每个pipeline步骤反推原始数据特征 → median说明有异常值 → scaler说明量纲不同 → log说明分布偏

Level 3 — Analysis (5–7 minutes)

Prompt: You receive a dataset with 3 numerical columns (with outliers and missing values) and 2 categorical columns. Design a full preprocessing pipeline and justify each step.

🧠 中文思路：数值和分类分开处理 → 数值：插补→处理异常值→标准化 → 分类：插补→编码 → 每步说为什么

Numerical: Median imputer → Outlier handling → Standardisation
Categorical: Mode imputer → One-hot encoding

B1 — MLP & Backpropagation

Level 1 — Core Intuition (30 seconds)

Prompt: Why is ReLU preferred over sigmoid in hidden layers? One sentence.

Level 2 — Exam Paragraph (3 minutes)

Prompt: Explain the vanishing gradient problem and one technique that mitigates it.

🧠 中文思路：先解释梯度消失原因（链式法则乘很多层）→ 再说sigmoid导数最大0.25 → 最后说解决方案

Level 3 — Analysis (5–7 minutes)

Prompt: You initialise all weights in a network to 0. Describe exactly what happens during training and why this is a problem.

🧠 中文思路：全0权重 → 前向传播所有神经元输出一样 → 反向传播梯度一样 → 对称性问题 → 永远学不出不同特征

B2 — CNN

Level 1 — Core Intuition (30 seconds)

Prompt: What does a convolutional filter detect? One sentence.

Level 2 — Exam Paragraph (3 minutes)

Prompt: Calculate the output dimensions of: Input [32, 32, 3] → Conv layer (32 filters, kernel=3, stride=1, padding=0). Show your working.

🧠 中文思路：写公式 → 代入数字 → 算出空间维度 → 深度 = 滤波器数量 → 记住别把滤波器深度和输出深度搞混

Level 3 — Analysis (5–7 minutes)

Prompt: Why are CNNs more suitable than fully connected MLPs for image data? Give at least two structural reasons.

🧠 中文思路：两个核心优势 → 参数共享（同一个滤波器全图共用）→ 局部连接（只看周围像素）→ 用参数数量对比说明

B3 — RNN / LSTM / GRU

Level 1 — Core Intuition (30 seconds)

Prompt: Why do vanilla RNNs struggle with long sequences? One sentence.

Level 2 — Exam Paragraph (3 minutes)

Prompt: Explain sequential processing as both an advantage and a disadvantage of RNNs.

🧠 中文思路：优点→自然捕捉顺序 | 缺点→不能并行 → 实际影响：长序列训练很慢 → 对比Transformer

Level 3 — Analysis (5–7 minutes)

Prompt: Compare RNNs and Transformers. In what scenario would you still choose an RNN over a Transformer in 2024?

🧠 中文思路：Transformer强在并行+远程依赖 → RNN什么时候还有用？→ 资源有限的设备、流式数据、短序列小数据

B4 — Transformer & Attention

Level 1 — Core Intuition (30 seconds)

Prompt: Explain Query, Key, and Value — one sentence each, using an analogy if possible.

Level 2 — Exam Paragraph (3 minutes)

Prompt: Why does a Transformer need positional encoding? What happens if you remove it?

🧠 中文思路：attention是排列不变的（不管顺序）→ 所以需要位置编码 → 否则"狗咬人"="人咬狗" → 怎么加的？

Level 3 — Analysis (5–7 minutes)

Prompt: What is the architectural difference between a Transformer encoder and decoder? Why does the decoder need masked self-attention?

🧠 中文思路：encoder双向注意力 → decoder有遮蔽+交叉注意力 → 为什么遮蔽？训练时防止看到未来答案

B5 — Vision Transformer (ViT)

Level 1 — Core Intuition (30 seconds)

Prompt: Why does ViT split images into patches instead of processing individual pixels?

Level 2 — Exam Paragraph (3 minutes)

Prompt: Describe the full ViT pipeline: how does a 224×224 image become a class prediction? Include the role of the [CLS] token.

🧠 中文思路：6步pipeline → 切patch→投影→加CLS→加位置编码→Transformer编码→CLS输出分类

Level 3 — Analysis (5–7 minutes)

Prompt: Compare ViT and CNN for image classification. When would you prefer each? Discuss the role of inductive bias and dataset size.

🧠 中文思路：CNN有归纳偏置（局部性+平移不变性）→ ViT没有 → 小数据CNN好 → 大数据ViT好 → 原因是什么

B6 — Activation Functions (2025 Q3)

Level 1 — Core Intuition (30 seconds)

Prompt: What is the dying ReLU problem? One sentence.

Level 2 — Exam Paragraph (3 minutes)

Prompt: A manufacturing quality system needs to detect multiple anomaly types simultaneously in a single image. Which output activation function should be used — sigmoid or softmax? Explain why.

🧠 中文思路：判断多标签问题 → sigmoid（独立概率）→ softmax不行（概率和为1互相压制）

Level 3 — Analysis (5–7 minutes)

Prompt: Compare ReLU, LeakyReLU, and sigmoid as hidden layer activations. When would you use each, and what are the risks?

🧠 中文思路：三个激活函数各写一段 → ReLU默认好但有死亡问题 → Leaky修复 → Sigmoid只适合门控/输出层

B7 — Batch Normalisation (Practice Q5)

Level 1 — Core Intuition (30 seconds)

Prompt: What does batch normalisation do to the activations? One sentence.

Level 2 — Exam Paragraph (3 minutes)

Prompt: Explain two distinct effects of batch normalisation on model training. For each, describe the mechanism.

🧠 中文思路：两个不同的效果 → 加速训练（梯度健康→可用大学习率）→ 正则化（batch统计引入噪声）

Level 3 — Analysis (5–7 minutes)

Prompt: Your colleague says "Batch norm replaces dropout — we don't need both." Evaluate this claim.

🧠 中文思路：部分对但不完全对 → BN的正则化效果比dropout弱 → BN主要是训练稳定 → 实际上很多模型两个都用

E1 — Evaluation Metrics

Level 1 — Core Intuition (30 seconds)

Prompt: What does recall measure, in plain English? (No formula — just what it means.)

Level 2 — Exam Paragraph (3 minutes)

Prompt: A fraud detection model has 99% accuracy but only 10% recall. Is this a good model? Explain your reasoning.

🧠 中文思路：类别不平衡 → 99%准确率没用（全预测阴性就有99%）→ recall才是关键（漏了90%的欺诈）

Level 3 — Analysis (5–7 minutes)

Prompt: You are building a cancer screening model. The dataset has 95% negative cases and 5% positive cases. Design an evaluation strategy. Which metrics do you prioritise and why?

🧠 中文思路：类别不平衡 → 准确率没用 → recall最重要（漏诊比误诊更危险）→ 用PR-AUC不用ROC → 设低阈值

考场速查卡 — 从中文思维到英文输出

用法： 脑子想到中文 → 找到对应英文零件 → 用万能连接词拼起来。不需要翻译，只需要组装。

万能连接词（只需要这 5 个）

答题公式（每种题型一个模板）

题型 1："这个建议能不能改善模型？"

[YES/NO], [建议] is likely to [improve/not improve] validation accuracy.

The model is currently [overfitting/underfitting], as evidenced by
[说出你看到的证据].

[建议] [helps/does not help] because [一句话机制].

实战例子（你脑子里想的 → 写出来的）：

题型 2："解释一个概念"

[概念] is [一句话定义].

It works by [机制].

This is [beneficial/important] because [为什么有用].

高频概念直接背：

题型 3："看图说话（loss curves / metrics）"

The [loss curve / metric] shows that [你看到了什么].

This indicates [诊断].

[建议/解释] because [原因].

题型 4："CNN 计算"

不需要造句，只需要写步骤。模板：

Conv layer:
  Input: [H, W, C]
  Output: [floor((H + 2p - f) / s) + 1, same for W, num_filters]

Pool layer:
  Input: [H, W, C]
  Output: [floor((H - f) / s) + 1, same for W, C]  (depth unchanged)

Flatten: H × W × C = [answer]

题型 5："对比 A 和 B"

While [A] [A的特点], [B] [B的特点].

The key advantage of [A] is [优势].

However, [A]'s limitation is [缺点], which [B] addresses by [B的解决方式].

高频 Chinglish 修正（只改这几个就够）

最终检查：写完每道题看一眼

• 有没有写 YES/NO 然后接 because？
• 关键术语拼对了没？（regularisation, overfitting, gradient, convergence）
• 有没有用具体数字（"95% vs 60%"）而不是模糊描述（"high vs low"）？

逻辑答题万能框架（Universal Answer Logic Framework）

所有 "评价建议" 题的万能思路：

第一步（诊断）：看 train acc 和 val acc → 判断过拟合/欠拟合
第二步（判断）：这个建议能不能解决诊断出的问题？
第三步（输出）：YES/NO + 用英文写原因

答题逻辑链模板（每道题都能用）：

观察 → 诊断 → 建议 → 原因
Observe → Diagnose → Suggest → Explain

"The training accuracy is [X]% while validation accuracy is [Y]%."  （观察）
→ "This indicates [overfitting/underfitting]."                      （诊断）
→ "[Suggestion] is [likely/unlikely] to help."                      （建议）
→ "This is because [mechanism]."                                     （原因）

答题字数控制：

COMPSCI 714 — A4 Cheatsheet (Print Both Sides)

SIDE 1: FORMULAS + DIAGNOSIS + CONCEPTS

1. CNN Dimension Formulas (MUST — every exam)

Conv output:  floor((n + 2p - f) / s) + 1  × num_filters
Pool output:  floor((n - f) / s) + 1        × same_depth
Flatten:      H × W × C

• Valid padding: p = 0 (output shrinks)
• Same padding: output H,W = input H,W (p chosen automatically)
• MaxPool vs AvgPool: same output dimensions, only values differ
• Depth after Conv = number of filters; depth after Pool = unchanged

Worked example (2025 Q6):

[35,35,3] →Conv(valid,k=7,s=2)→ floor((35-7)/2)+1=15 → [15,15,10]
         →Pool(k=2,s=2)→ floor((15-2)/2)+1=7 → [7,7,10]
         →Conv(same,k=3,s=1)→ same H,W → [7,7,20]
         →Pool(k=2,s=2)→ floor((7-2)/2)+1=3 → [3,3,20]
         →Flatten: 3×3×20 = 180

2. Bias-Variance Diagnosis (MUST — ~20% of marks)

Fixes for OVERFITTING (high variance):

• More data / data augmentation ✓
• L2 regularisation (penalises large weights) ✓
• Dropout (randomly deactivates neurons) ✓
• Batch normalisation (regularising effect) ✓
• Reduce model size ✓
• Early stopping ✓
• More epochs ✗ (worsens it!)

Fixes for UNDERFITTING (high bias):

• Increase model size (more layers/neurons) ✓
• More/better features ✓
• Train longer ✓
• Reduce regularisation ✓
• Dropout ✗ (constrains already limited model!)
• Zero initialisation ✗ (symmetry problem — all neurons learn same thing)

3. Evaluation Metrics (HIGH)

Accuracy  = (TP + TN) / (TP + TN + FP + FN)
Precision = TP / (TP + FP)    "of predicted +, how many correct?"
Recall    = TP / (TP + FN)    "of actual +, how many found?"
F1        = 2 × (P × R) / (P + R)

Trap: High accuracy + low recall → class imbalance. Model predicts majority class. Trap: 100% recall + low precision → model predicts everything as positive.

4. Learning Rate Curves (HIGH)

Momentum: exponentially decaying average of past gradients → smoother updates, faster convergence.

LR Schedule (e.g. exponential decay): start high for fast progress, reduce to fine-tune near optimum.

Adam: momentum + adaptive per-parameter learning rates.

5. Activation Functions (MED)

Dying ReLU: negative input → output=0 → no gradient → neuron "dies". LeakyReLU fix: small slope (e.g. 0.01x) for negative inputs → neuron still gets gradient.

KEY: Multi-label (multiple outputs ON) → sigmoid. Multi-class (exactly one) → softmax.

6. Data Preprocessing (MUST)

When to remove vs impute:

• Remove: vast majority missing (e.g. 9995/10000)
• Impute: reasonable number missing (e.g. 15/10000)

Outlier detection: extreme min/max relative to mean+std → likely outliers

SIDE 2: ARCHITECTURES + ANSWER TEMPLATES

7. Transformer / Attention (MUST)

Self-attention: weighted sum of Values, where weights = relevance between Query and Key.

Multi-head attention: multiple attention heads with separate Q/K/V → each focuses on different aspects. Outputs concatenated.

Masked attention (decoder): prevents attending to future tokens → preserves autoregressive property during training (predict next token based only on previous).

Positional encoding: needed because Transformer processes all tokens in parallel → loses order information. Added to embeddings.

ViT [CLS] token: learnable token prepended to patch sequence → aggregates info from all patches via attention → fed to MLP for classification. Advantage: efficient, no need for global pooling.

8. RNN / LSTM (MED)

RNN: h_t = f(W·h_{t-1} + U·x_t + b). Sequential processing.

• Advantage: naturally captures order
• Drawback: can't parallelise → slow for long sequences
• Problem: vanishing gradients (long-range dependencies lost)

LSTM: 3 gates (forget, input, output) control information flow → solves vanishing gradient.

How Transformer fixes RNN drawback: processes all positions in parallel via embeddings + adds positional encoding for order info.

9. DNN Training Challenges (MED)

Why deep nets are hard to train:

1. Vanishing/exploding gradients
2. More prone to overfitting
3. Longer training time

Strategies to help:

• Skip connections / ResNet (y = F(x) + x, gradients flow through shortcut)
• Batch normalisation (normalises activations, keeps gradients healthy)
• Better optimisers (Adam, RMSProp)
• LSTM/GRU for sequences

Batch Norm effects: speeds up training, reduces vanishing gradients, regularisation effect, reduces sensitivity to weight initialisation.

10. Answer Templates

"Will this improve validation accuracy?"

[YES/NO], [suggestion] is [likely/unlikely] to improve validation accuracy.
The model is currently [overfitting/underfitting], as evidenced by
[train acc X% vs val acc Y%].
[Suggestion] [helps/does not help] because [mechanism].

"Explain a concept"

[Concept] is [one-sentence definition].
It works by [mechanism].
This is [beneficial/important] because [why].

"Interpret loss curves / metrics"

The [curve/metric] shows [observation].
This indicates [diagnosis].
This is because [cause].

CNN calculation

Layer: Input [H,W,C]
  formula: floor((H+2p-f)/s)+1 = ...
  Output: [H',W',C']
→ next layer...
→ Flatten: H×W×C = answer

11. Key English Phrases (copy-paste ready)

12. Common Traps to Avoid

• Multi-label ≠ multi-class → sigmoid, NOT softmax
• High recall + low precision = predicting everything positive (not a good model)
• Regularisation fights overfitting, NEVER helps underfitting
• Zero initialisation → symmetry problem → all neurons learn the same
• More epochs → more overfitting, not less
• MaxPool vs AvgPool → same dimensions, different values
• Accuracy alone is misleading with imbalanced classes

13. Marks-per-minute priority

1. Bias/Variance diagnosis + fixes    ~20%  ← ALWAYS on exam
2. CNN dimension calculation           ~15%  ← ALWAYS on exam
3. Transformer/Attention concepts      ~15%  ← ALWAYS on exam
4. Data preprocessing reasoning        ~15%  ← ALWAYS on exam
5. Learning rate curve matching        ~10%  ← usually on exam
6. Confusion matrix metrics            ~10%  ← usually on exam
7. Activation functions                ~5%
8. RNN vs Transformer                  ~5%
9. Batch Norm / DNN training           ~5%

English Expression Guide for the Exam

This exam is written and answered in English. These templates help you write clear, precise, high-scoring answers.

Universal Answer Structures

Structure 1: "Evaluate a Suggestion" (most common question type)

[YES/NO], [suggestion] is [likely/unlikely] to improve validation accuracy.

This is because the current model is [overfitting/underfitting], as evidenced by 
[training accuracy being much higher/lower than validation accuracy].

[Suggestion] works by [mechanism], which [helps/does not help] with 
[overfitting/underfitting] because [specific reason].

Example: "Yes, L2 regularisation is likely to improve validation accuracy. The current model is overfitting, as the training accuracy (95%) is much higher than the validation accuracy (60%). L2 regularisation penalises large weights, encouraging the model to learn a simpler, more generalisable representation, which helps reduce overfitting."

Structure 2: "Explain Concept X"

[Concept] is [one-sentence definition].

It works by [mechanism in 1-2 sentences].

This is beneficial/important because [why it matters].

Structure 3: "Interpret Model Performance"

The [metric] is [value], which indicates [what this means].

However, [other metric] reveals that [deeper insight].

This is because [explanation of model behavior, e.g., class imbalance, 
predicting everything as one class].

Structure 4: "Compare A and B"

While [A] [feature of A], [B] [feature of B].

The key advantage of [A] is [advantage], whereas [B] excels at [advantage].

However, [A]'s main drawback is [drawback], which [B] addresses by [solution].

Topic-Specific Sentence Templates

Data Preprocessing（数据预处理 — 考察你能否从pipeline步骤反推数据特征）

• "The use of [median/most frequent] imputation suggests the data is [numerical/categorical] with missing values."
• "The standardisation step indicates that features are on different scales."
• "The log transformation suggests the data has a heavy-tailed distribution."
• "Removing the attribute is justified because [X]% of values are missing, and imputation would create misleading information."
• "Outlier removal is appropriate because the maximum value ([X]) is significantly larger than expected given the mean ([Y]) and standard deviation ([Z])."

Bias-Variance（偏差-方差 — 考察你能否从train/val数字诊断问题并给出建议）

• "The model displays high variance, as there is a clear gap between the training and validation [accuracy/loss]."
• "This indicates overfitting, where the model fits the training data too closely but fails to generalise."
• "The model appears to have high bias, as both training and validation accuracies are low."
• "This suggests underfitting — the model is not complex enough to capture the underlying patterns."
• "Applying regularisation can help reduce overfitting by limiting the complexity of the model."
• "Increasing the model size may help address underfitting by giving the model more capacity to learn."

CNN（卷积神经网络 — 考察维度计算和结构优势解释）

• "The output dimensions of the convolutional layer are calculated as: floor((n + 2p - f) / s) + 1."
• "With valid padding (p=0), the spatial dimensions of the output will be smaller than the input."
• "With same padding, the output spatial dimensions match the input spatial dimensions when stride is 1."
• "The depth of the output equals the number of filters applied."
• "Pooling reduces the spatial dimensions while preserving the depth."
• "Max pooling and average pooling produce output with the same dimensions; only the values differ."

Transformer & Attention（注意力机制 — 考察位置编码、遮蔽、多头注意力的作用）

• "The masking in the decoder prevents each position from attending to future tokens."
• "This preserves the autoregressive property, ensuring that predictions depend only on previously generated tokens."
• "Multi-head attention runs several attention functions in parallel, each with its own learned weight matrices."
• "This allows the model to focus on different aspects of the input simultaneously."
• "The [CLS] token in ViT aggregates information from all image patches for the final classification."
• "Positional encoding is necessary because the Transformer processes all positions in parallel, losing inherent ordering."

Learning Rate & Optimizers（学习率与优化器 — 考察loss曲线诊断和优化器选择）

• "A diverging loss curve indicates a learning rate that is too high, causing the optimisation to overshoot."
• "A very slowly decreasing loss suggests the learning rate is too small."
• "Learning rate scheduling, such as exponential decay, allows faster initial convergence while avoiding overshooting near the optimum."
• "The momentum mechanism maintains an exponentially decaying average of past gradients, smoothing the optimisation trajectory."

Evaluation Metrics（评估指标 — 考察混淆矩阵计算和类别不平衡下的指标选择）

• "The accuracy is calculated as (TP + TN) / (TP + TN + FP + FN) = ..."
• "Despite the seemingly acceptable accuracy, the model performs poorly at identifying positive instances."
• "The high recall but low precision indicates the model predicts most instances as positive."
• "This discrepancy highlights the importance of examining metrics beyond accuracy, particularly with imbalanced datasets."

Activation Functions（激活函数 — 考察sigmoid/softmax/ReLU的选择和死亡ReLU问题）

• "The dying ReLU problem occurs when neurons consistently receive negative inputs, causing them to output zero and stop learning."
• "LeakyReLU mitigates this by introducing a small positive slope for negative inputs."
• "For multi-label classification, sigmoid is the appropriate output activation because each output is treated independently."
• "Softmax is unsuitable for multi-label problems because it forces all outputs to sum to 1."

Batch Normalisation（批量归一化 — 考察BN的两个效果：加速训练+正则化）

• "Batch normalisation speeds up training by normalising activations within each mini-batch."
• "It reduces the risk of vanishing and exploding gradients by keeping activations in a healthy range."
• "The normalisation over mini-batches introduces noise, which has a regularising effect."
• "It also reduces sensitivity to weight initialisation by automatically adjusting activation distributions."

Regularisation (L1, L2, Dropout, Early Stopping)（正则化 — 考察过拟合时的解决方案及其机制）

• "L2 regularisation penalises large weights, encouraging the model to learn a simpler, more generalisable representation."
• "L1 regularisation drives some weights to exactly zero, performing automatic feature selection."
• "Dropout randomly deactivates neurons during training, forcing the network to learn redundant, distributed representations."
• "This prevents co-adaptation, where specific neurons become overly reliant on each other."
• "Early stopping halts training when validation loss stops improving, preventing the model from memorising training noise."
• "Regularisation constrains model complexity, which helps when the model is overfitting but worsens underfitting."

RNN / LSTM / GRU（循环神经网络 — 考察顺序处理优缺点和梯度消失的解决）

• "RNNs process tokens sequentially, naturally capturing temporal order without additional mechanisms."
• "However, sequential processing prevents parallelisation, making training slow for long sequences."
• "Vanilla RNNs suffer from vanishing gradients because gradients are multiplied through many time steps."
• "LSTM mitigates this by introducing a cell state and gating mechanisms (forget, input, output gates) that control information flow."
• "The forget gate decides what information to discard, while the input gate controls what new information to store."
• "GRU simplifies LSTM by combining the forget and input gates into a single update gate, reducing the number of parameters."

逻辑连接词完整指南（Logic Connectors for Exam Answers）

中国学生最常犯的逻辑问题不是内容错误，而是句子之间缺少逻辑连接。下面的连接词能让你的答案读起来像母语者写的。

因果关系（Cause & Effect）

转折关系（Contrast & Concession）

递进关系（Addition & Elaboration）

总结关系（Conclusion & Summary）

Common Mistakes in English Writing

Power Words for High-Scoring Answers

Professional Vocabulary Quick Reference

Frequently Confused Terms

Key Terms by Topic

Data Preprocessing

Neural Networks

CNN

Transformer

Regularisation & Training

Optimisation

RNN / Sequence Models

Evaluation

Commonly Misspelled Words

Note: This is a New Zealand university — British/NZ spelling is expected (regularisation, normalisation, optimisation), not American spelling.

考试高频搭配（Collocations for Exams）

英文不是一个词一个词写的，是一组一组搭配着用的。背搭配比背单词更有效。

动词 + 名词 搭配

形容词 + 名词 搭配

常用介词搭配

Exam Writing Format & Strategy

Exam Rules Recap

Time management:

• 2025: ~20 marks in 55 min = ~2.75 min per mark
• 2024: ~30 marks in 55 min = ~1.83 min per mark
• Rule of thumb: 1 mark ≈ 2-3 minutes. If a question is worth 2 marks, spend ~5 minutes max.

The 5-Minute Reading Period Strategy

During the 5-minute reading time (no writing allowed):

1. Scan ALL questions — count marks, identify topics
2. Identify the calculation question (CNN dimensions) — mentally plan the steps
3. Identify the diagnosis question (bias/variance from curves) — start forming your answer
4. Plan your time allocation — more time on high-mark questions
5. Identify what you need from your cheat sheet — locate formulas you'll need

Answer Formatting Rules

Rule 1: Lead with the Answer

💡 中文思维习惯："先铺垫再给结论"。英文学术写作相反：先给结论，再解释原因。这是中国学生最需要改变的习惯。

❌ "There are many factors to consider. First, we need to think about..."
✅ "No, training for 2000 epochs will not help because the model is already overfitting."

Rule 2: Be Concise — Quality Over Quantity

💡 不要写"废话"凑字数。老师明确说了 "quality over quantity"。用3句精确的话比写半页模糊的要好。

❌ (half page of vague general knowledge)
✅ "This is an overfitting scenario (train 95%, val 60%). L2 regularisation will help 
    because it penalises large weights, promoting a simpler model that generalises better."

Rule 3: Link to the Specific Scenario

💡 不要只写"正则化能防止过拟合"这种教科书式的回答。必须引用题目给的具体数字（如 train=95%, val=60%）来支撑你的判断。

❌ "Regularisation helps prevent overfitting." (too generic)
✅ "Since the training accuracy (95%) is much higher than validation (60%), 
    indicating overfitting, L2 regularisation is likely to help by constraining 
    model complexity." (linked to given numbers)

Rule 4: Show Calculation Steps

💡 计算题一定要写出公式和代入过程，不能只写最终答案。即使算错了，步骤正确也能拿部分分。

❌ "The output is 16x16x10"
✅ "Conv output: ((50 + 2×0 - 5) / 3) + 1 = (45/3) + 1 = 15 + 1 = 16
    Output: [16, 16, 10] (10 from number of filters)"

Rule 5: For "What Do You Think?" — Go Beyond Numbers

💡 "你怎么看"这类题不能只算数字，要解释数字背后的含义。模型到底在做什么？为什么会这样？

❌ "Accuracy is 60% and recall is 100%."
✅ "The accuracy is 60% and recall is 100%. This means the model predicts almost 
    everything as positive — it catches all actual positives (perfect recall) but 
    at the cost of many false positives (precision only 56%). The model appears 
    to be performing well at detecting positives, but is actually just classifying 
    nearly everything as positive."

Answer Templates by Question Type

Type: "Evaluate This Suggestion" (2 marks each)

[YES/NO — 1 mark]
[Reasoning connected to scenario — 1 mark]

Template:

[Yes/No], [suggestion] is [likely/unlikely] to improve the validation accuracy. 
The model is currently [overfitting/underfitting] (training accuracy [X]%, 
validation accuracy [Y]%). [Suggestion] [mechanism: e.g., "penalises large weights" / 
"adds more training data"] which [helps/does not help] with [overfitting/underfitting] 
because [specific reason].

Type: "Calculate + Interpret" (Confusion Matrix)

Step 1: State formulas
Step 2: Plug in numbers
Step 3: Give result
Step 4: Interpret (what does the model actually DO?)
Step 5: Explain WHY (class imbalance? threshold too low?)

Type: "Explain Concept" (2-4 marks)

Paragraph 1: WHAT it is (definition)
Paragraph 2: HOW it works (mechanism)
Paragraph 3: WHY it matters (benefit/purpose)

Type: CNN Calculation (show workings)

For each layer:
  Input:   [H, W, C]
  Formula: ((H + 2p - f) / s) + 1 = ...
  Output:  [H', W', C']

Final: Flatten = H' × W' × C' = [number]

Time Allocation Guide

For a 20-mark exam (2025 format):

For a 30-mark exam (2024 format):

Last-Minute Reminders

1. Read the scenario carefully — the numbers matter (train acc, val acc, missing values count)
2. Diagnose BEFORE prescribing — always state overfitting/underfitting first
3. Show your work on calculations — partial marks are possible
4. Cross out wrong work — the exam says "cross out work you don't want assessed"
5. Write clearly — illegible answers get 0 marks
6. Use overflow pages if needed, but note which question on the original page

中国学生考试写作常见问题（Common Issues for Chinese Students）

问题 1：先写"背景"再给"结论"

❌ 中文习惯："首先，正则化是一种技术……它的作用是……所以我认为……"
✅ 英文习惯："Yes, L2 regularisation will help. This is because..."

→ 英文考试要求：第一句话就给结论，然后解释原因。

问题 2：过度使用 "can"

❌ "Regularisation can help to improve the model."
✅ "Regularisation is likely to improve validation accuracy by constraining model complexity."

→ "can" 太弱。用 "is likely to" 或 "will" 更果断、更学术。

问题 3：缺少因果连接

❌ "The model is overfitting. We should use dropout."（两句话之间没有逻辑连接）
✅ "The model is overfitting, as evidenced by the gap between training and validation accuracy. Therefore, applying dropout is likely to help by reducing co-adaptation."

→ 用 "as evidenced by"、"therefore"、"because" 把句子连起来。

问题 4：用中文直译

❌ "The model learned too good on the training data"
✅ "The model fits the training data too closely"

❌ "The performance is not good enough"
✅ "The model fails to generalise to unseen data"

❌ "We can use a more big model"  
✅ "Increasing the model size would help"

问题 5：不敢下判断

❌ "Maybe this suggestion could possibly help..."
✅ "Yes, this suggestion is likely to improve validation accuracy."

→ 考试要明确 YES/NO。模糊的回答拿不到分。

Mock Exam 1 — 20 Marks (2025 Format)

Instructions: 60 minutes total. 5 minutes reading + 55 minutes writing. Double-sided handwritten cheat sheet allowed. No calculators. Attempt ALL questions. Be concise and clear.

Question 1: Data Preprocessing [3 marks]

Consider the following dataset summary with 8,000 samples:

For each of the following cleaning steps, explain whether it makes sense to apply it to this dataset. Briefly justify your answers. [0.5 mark per step, + 0.5 for overall justification]

• (a) Missing value replacement based on most frequent value.
• (b) Missing value replacement based on mean value.
• (c) Removing an attribute.
• (d) Standardisation.
• (e) Outlier removal.

Question 2: Bias-Variance and Design Choices [4 marks]

You trained a neural network with the following settings: 8 hidden layers with ReLU units, 128 neurons per layer, trained for 500 epochs with a batch size of 32 and no regularisation. After training, the training accuracy is 98% and the validation accuracy is 52%.

(a) Diagnose the model's performance in terms of bias and variance. [1 mark]

(b) For each of the following suggestions, explain whether it is likely to improve the validation accuracy and why. [1 mark per suggestion]

• Adding dropout (rate=0.5) to each hidden layer.
• Reducing the number of hidden layers from 8 to 2.
• Training for 1000 epochs instead of 500.

Question 3: Evaluation Metrics [3 marks]

A spam detection model is evaluated on a test set of 2,000 emails (200 spam, 1,800 not spam). The confusion matrix is:

(a) Calculate the accuracy, precision, and recall. [1.5 marks]

(b) Is this model suitable for deployment as a spam filter? Explain your reasoning, considering what each metric means in this context. [1.5 marks]

Question 4: Learning Rate and Batch Normalisation [4 marks]

(a) Explain in your own words what is the momentum mechanism in gradient descent optimisation, and describe its effect on training. [2 marks]

(b) Give two effects of batch normalisation on the training of a neural network. For each effect, briefly explain the mechanism behind it. [2 marks]

Question 5: Transformers and ViT [3 marks]

(a) In a Transformer model, the self-attention mechanism computes Query (Q), Key (K), and Value (V) matrices. Explain what role each plays in the attention computation. [1.5 marks]

(b) Explain why multi-head attention is preferred over a single attention head in Transformer models. [1.5 marks]

Question 6: CNN Architecture [3 marks]

Consider the following CNN architecture:

• Input: [32, 32, 3]
• Conv1: 8 filters, kernel size = 5, stride = 1, padding = 0 (valid)
• MaxPool1: kernel size = 2, stride = 2
• Conv2: 16 filters, kernel size = 3, stride = 1, padding = 1 (same)
• MaxPool2: kernel size = 2, stride = 2
• Fully connected layer: ? inputs → 10 outputs

(a) What is the number of inputs to the fully connected layer? [1 mark]

(b) Show your calculation steps for each layer. [2 marks]

END OF MOCK EXAM 1

Mock Exam 1 — Answer Key & Detailed Explanations

Question 1: Data Preprocessing [3 marks]

(a) Most frequent value imputation: YES for A1 (categorical with 150 missing values). Most frequent value strategy is appropriate for categorical data with a moderate number of missing values. NO need for A4 (binary, no missing values).

(b) Mean value imputation: Not ideal — for A2 (30 missing, numerical), imputation is needed but median would be better than mean because the max (210.5) is unusually high relative to the mean (45.2), suggesting outliers. Mean is sensitive to outliers. For A5, no missing values so no imputation needed.

(c) Removing an attribute: YES for A3 — it has 7,850 out of 8,000 values missing (98.1%). Imputing so many values would create almost entirely fabricated data, which is misleading and unlikely to help the model.

(d) Standardisation: YES — A2 has values around 45, while A5 has values around 1,200 with a huge range. Features on different scales should be standardised so no single feature dominates the model.

(e) Outlier removal: YES — A5 has mean=1200 but std=8500 (std >> mean), and extremes of 150,000 and -30,000. A2 also has max=210.5 which is ~13.6 standard deviations above the mean (suspicious). These suggest outliers that could distort model training.

Question 2: Bias-Variance and Design Choices [4 marks]

(a) Diagnosis [1 mark]: The model displays high variance (overfitting). The training accuracy (98%) is significantly higher than the validation accuracy (52%), indicating the model has learned to fit the training data very closely but fails to generalise to unseen data. The model is too complex for the amount of data (8 layers, 128 neurons, no regularisation).

(b) Evaluate suggestions [1 mark each]:

Dropout (rate=0.5): YES, likely to improve validation accuracy. Since the model is overfitting, dropout will randomly deactivate 50% of neurons during each training step. This prevents co-adaptation of neurons and forces the network to learn more robust, distributed representations. It acts as a regularisation technique, reducing the gap between training and validation performance.

Reducing layers from 8 to 2: YES, likely to improve validation accuracy. The current model (8 layers, 128 neurons) has very high capacity, which contributes to overfitting. Reducing to 2 layers decreases the model's capacity to memorise training noise. However, it might also reduce the model's ability to learn complex patterns — the optimal size depends on the problem complexity. Given the severe overfitting, reducing complexity is a reasonable first step.

Training for 1000 epochs: NO, this will likely make the validation accuracy worse. The model is already overfitting at 500 epochs. Training for longer will allow the model to memorise the training data even more closely, further increasing the gap between training and validation accuracy. If anything, earlier stopping would be more beneficial.

Question 3: Evaluation Metrics [3 marks]

(a) Calculations [1.5 marks]:

$$\text{Accuracy} = \frac{TP + TN}{Total} = \frac{180 + 1440}{2000} = \frac{1620}{2000} = 0.81 \text{ (81%)}$$

$$\text{Precision} = \frac{TP}{TP + FP} = \frac{180}{180 + 360} = \frac{180}{540} = 0.333 \text{ (33.3%)}$$

$$\text{Recall} = \frac{TP}{TP + FN} = \frac{180}{180 + 20} = \frac{180}{200} = 0.90 \text{ (90%)}$$

(b) Suitability analysis [1.5 marks]:

Despite the seemingly good accuracy (81%), the model is not suitable for deployment as a spam filter in its current form.

The problem is precision (33.3%): Of all emails the model flags as spam, only 1 in 3 actually IS spam. This means 2 out of every 3 "spam" flags are legitimate emails being incorrectly blocked (360 false positives). For a spam filter, this is extremely disruptive — users would miss important emails regularly.

The recall (90%) is good: The model catches 90% of actual spam, missing only 20 out of 200 spam emails.

Recommendation: In a spam filter, precision is arguably more important than recall — it's better to let some spam through (lower recall) than to block legitimate emails (low precision). The classification threshold should be adjusted to increase precision, even at the cost of some recall. Alternatively, flagged emails could be moved to a "spam folder" rather than deleted, allowing users to review.

Question 4: Learning Rate and Batch Normalisation [4 marks]

(a) Momentum [2 marks]:

The momentum mechanism maintains an exponentially decaying running average of past gradients to determine the direction and magnitude of weight updates.

Without momentum, each gradient update depends only on the current mini-batch gradient, which can be noisy — the optimisation path zigzags. With momentum, the update is:

$$v_t = \beta \cdot v_{t-1} + (1 - \beta) \cdot \nabla L$$

The parameter $\beta$ (typically 0.9) controls how much weight is given to past gradients.

Effects:

1. Smoother optimisation: By averaging past gradients, noisy fluctuations are dampened, leading to more consistent update directions
2. Faster convergence: When the gradient consistently points in the same direction, momentum builds up speed (like a ball rolling downhill), accelerating progress
3. Escaping local minima: The accumulated momentum can carry the optimisation through shallow local minima that would trap standard gradient descent

(b) Batch normalisation effects [2 marks]:

Effect 1: Speeds up training. Batch normalisation normalises the activations within each mini-batch to have zero mean and unit variance. This keeps the input distribution to each layer stable throughout training (reduces internal covariate shift), allowing the use of higher learning rates. The activations remain in a range where gradients are meaningful (not too small, not too large), leading to faster convergence.

Effect 2: Has a regularising effect. Because normalisation is computed over mini-batches rather than the full dataset, each sample's normalised value depends on which other samples happen to be in the same mini-batch. This introduces noise into the activations, similar to dropout. This noise acts as implicit regularisation, helping prevent overfitting without explicitly adding a regularisation term.

Question 5: Transformers and ViT [3 marks]

(a) Q, K, V roles [1.5 marks]:

In the self-attention mechanism:

• Query (Q): Represents what each position is "looking for" — it encodes the current position's request for relevant information from other positions. Think of it as a question: "What information do I need?"

• Key (K): Represents what each position "contains" or "offers" — it encodes features that other positions can match against. Think of it as a label on a file: "This is what I'm about."

• Value (V): Represents the actual information content at each position — once the attention weights are computed (by matching Q with K), the values are combined according to these weights to produce the output.

The attention score between two positions is computed as the dot product of Q and K (how well the query matches the key), scaled by $\sqrt{d_k}$, then softmaxed to get weights. These weights are applied to V to get the final attended output.

(b) Why multi-head is preferred [1.5 marks]:

Multi-head attention is preferred because it allows the model to attend to information from different representation subspaces simultaneously.

A single attention head computes one set of attention weights, which tends to have an averaging effect — it tries to capture all types of relationships (syntactic, semantic, positional) in a single attention distribution. This limits its expressiveness.

Multi-head attention runs $h$ parallel attention operations, each with independently learned $W^Q$, $W^K$, $W^V$ projections. Each head can specialise in different aspects: one head might learn to attend to nearby positions (local syntax), another to semantically related words far away, another to co-reference relationships. The outputs are concatenated and projected, giving the model a much richer representation of the relationships in the input.

Question 6: CNN Architecture [3 marks]

(a) Answer: The number of inputs to the fully connected layer is 784.

(b) Step-by-step calculations:

Layer: Conv1 (padding = 0, valid)
  Input:  [32, 32, 3]
  Calc:   (32 + 2×0 - 5) / 1 + 1 = 28
  Output: [28, 28, 8]       ← 8 filters

Layer: MaxPool1 (k=2, s=2)
  Input:  [28, 28, 8]
  Calc:   (28 - 2) / 2 + 1 = 14
  Output: [14, 14, 8]       ← depth unchanged

Layer: Conv2 (padding = 1, same)
  Input:  [14, 14, 8]
  Calc:   (14 + 2×1 - 3) / 1 + 1 = 14
  Output: [14, 14, 16]      ← 16 filters

Layer: MaxPool2 (k=2, s=2)
  Input:  [14, 14, 16]
  Calc:   (14 - 2) / 2 + 1 = 7
  Output: [7, 7, 16]        ← depth unchanged

Flatten: 7 × 7 × 16 = 784

Answer: 784 inputs to the fully connected layer.

Scoring Summary:

Mock Exam 2 — 30 Marks (2024 Format)

Instructions: 60 minutes total. 5 minutes reading + 55 minutes writing. Double-sided page of notes allowed. No calculators. Attempt ALL 7 questions. Be concise and clear.

Question 1: Data Preprocessing Pipeline [4 marks]

Consider the following 2 preprocessing pipelines. For each pipeline, describe the likely characteristics of the input raw data and explain how you can derive them from the preprocessing steps. [2 marks per pipeline]

Pipeline 1:

Raw data → Min-Max Normalisation → PCA (dimensionality reduction) → Pre-processed data

Pipeline 2:

Raw data → Imputer (median value) → Standardisation → Polynomial Feature Expansion → Pre-processed data

Question 2: Design Choices [6 marks]

You trained a neural network with the following settings: 3 hidden layers with ReLU units, 64 neurons per layer, Xavier initialisation, trained for 300 epochs with a learning rate of 0.01. The training accuracy is 55% and the validation accuracy is 53%. You know that state-of-the-art models achieve 92% on the same problem.

For each of the following suggestions, explain if it is likely to improve the validation accuracy and why. [2 marks per suggestion]

• (a) Using L2 regularisation with $\lambda = 0.01$.
• (b) Increasing the model to 6 layers with 256 neurons per layer.
• (c) Using data augmentation techniques relevant to the problem domain.

Question 3: Evaluation [4 marks]

A medical diagnostic model is evaluated on 500 patients (100 have the disease, 400 do not). The results are:

(a) Calculate the accuracy, precision, recall, and F1 score. [2 marks]

(b) In a medical context, discuss whether this model's performance is acceptable. Consider the implications of each type of error. [2 marks]

Question 4: Learning Rate [4 marks]

(a) Give an example of a learning rate schedule. Explain how it works and why it might be beneficial when training a neural network. [2 marks]

(b) The Adam optimiser is often described as combining the benefits of two other optimisation techniques. Name these two techniques and briefly explain what each contributes to Adam. [2 marks]

Question 5: RNN and Transformer [4 marks]

(a) Explain the vanishing gradient problem in the context of RNNs. Why does it make learning long-range dependencies difficult? [2 marks]

(b) Describe two mechanisms that the Transformer architecture uses to handle sequential data, and explain the purpose of each. [2 marks]

Question 6: CNN [4 marks]

Consider the following CNN architecture:

• Input: [64, 64, 1] (grayscale images)
• Conv1: 32 filters, kernel size = 3, stride = 1, padding = 1 (same)
• MaxPool1: kernel size = 2, stride = 2
• Conv2: 64 filters, kernel size = 3, stride = 2, padding = 0 (valid)
• Conv3: 128 filters, kernel size = 3, stride = 1, padding = 1 (same)
• MaxPool2: kernel size = 2, stride = 2
• Fully connected layer: ? inputs → 256 → 10 outputs

(a) Calculate the number of inputs to the fully connected layer. Show your workings for each layer. [3 marks]

(b) If you were to visualise what the filters in Conv1 have learned, what types of patterns would you expect to see? What about Conv3? Explain the difference. [1 mark]

Question 7: Activation Functions and DNN [4 marks]

(a) Explain the difference between sigmoid and softmax activation functions. Give a scenario where each would be the most appropriate choice for the output layer. [2 marks]

(b) Explain two different strategies that help mitigate the challenges of training very deep neural networks (e.g., 50+ layers). For each strategy, explain the mechanism and why it helps. [2 marks]

END OF MOCK EXAM 2

Mock Exam 2 — Answer Key & Detailed Explanations

Question 1: Data Preprocessing Pipeline [4 marks]

Pipeline 1: Raw data → Min-Max Normalisation → PCA → Pre-processed data

• The data is numerical because Min-Max normalisation operates on numerical values, scaling them to a fixed range (typically [0,1]).
• The features may have different scales, which is why normalisation is applied first — PCA is sensitive to feature scaling, so normalisation ensures all features contribute equally.
• The data likely has a large number of features (high dimensionality) because PCA (Principal Component Analysis) is used to reduce dimensionality. This suggests the dataset has many correlated features, and PCA extracts the most informative combinations while reducing noise and computational cost.

Pipeline 2: Raw data → Imputer (median) → Standardisation → Polynomial Feature Expansion

• The data is numerical with missing values since median imputation is used. The choice of median over mean suggests the data may have outliers or a skewed distribution, as the median is more robust to extreme values.
• Features are on different scales, hence standardisation (z-score normalisation to mean=0, std=1) is applied.
• The relationship between features and the target may be non-linear, which is why polynomial feature expansion is used. This creates new features by combining existing ones (e.g., $x_1^2$, $x_1 \cdot x_2$), allowing a linear model to capture non-linear patterns.

Question 2: Design Choices [6 marks]

Diagnosis: The model is underfitting (high bias). Both training (55%) and validation (53%) accuracies are low and close together, while 92% is achievable. The model is not complex enough to capture the patterns in the data.

(a) L2 regularisation: NO, this is unlikely to improve validation accuracy. L2 regularisation constrains the model by penalising large weights, effectively reducing the model's effective capacity. Since the model is already underfitting (not fitting even the training data well), adding regularisation would further constrain it, potentially making the underfitting worse. Regularisation is a remedy for overfitting, not underfitting.

(b) Increasing to 6 layers, 256 neurons: YES, this is likely to improve validation accuracy. Since the model is underfitting, it likely lacks the capacity to represent the complexity of the data. Increasing the number of layers (from 3 to 6) and neurons per layer (from 64 to 256) gives the model more representational power. With more parameters, the model can learn more complex decision boundaries. However, this should be done carefully — too much capacity without enough data could lead to overfitting in subsequent iterations.

(c) Data augmentation: YES, this could help improve validation accuracy. Data augmentation increases the effective size and diversity of the training set by applying transformations (rotations, flips, noise, etc.). For an underfitting model, more diverse data can help the model learn more general and robust patterns. Additionally, if the model is underfitting partly due to insufficient training data (not enough examples to learn from), augmentation addresses this directly. Note: the augmentation must be relevant to the domain — irrelevant augmentation could hurt performance.

Question 3: Evaluation [4 marks]

(a) Metric calculations [2 marks]:

$$\text{Accuracy} = \frac{TP + TN}{Total} = \frac{70 + 370}{500} = \frac{440}{500} = 0.88 \text{ (88%)}$$

$$\text{Precision} = \frac{TP}{TP + FP} = \frac{70}{70 + 30} = \frac{70}{100} = 0.70 \text{ (70%)}$$

$$\text{Recall} = \frac{TP}{TP + FN} = \frac{70}{70 + 30} = \frac{70}{100} = 0.70 \text{ (70%)}$$

$$\text{F1} = \frac{2 \times 0.70 \times 0.70}{0.70 + 0.70} = \frac{0.98}{1.40} = 0.70 \text{ (70%)}$$

(b) Medical context analysis [2 marks]:

In a medical diagnostic context, this model's 88% accuracy appears good, but the critical concern is the 30 false negatives — these are patients who HAVE the disease but the model says they don't. In medicine, missing a real disease (false negative) can be life-threatening, as these patients won't receive treatment.

The recall of 70% means the model only detects 70% of actual disease cases, missing 30% of sick patients. In a screening context, this is often unacceptable — a good screening test should have high recall (>95%) to avoid missing cases.

The precision of 70% means 30% of positive predictions are false alarms. While false positives are inconvenient (unnecessary follow-up tests), they are generally less dangerous than false negatives in medical contexts.

Conclusion: For a high-stakes medical application, this model needs improvement, particularly in recall. The threshold could be lowered to catch more true positives (increasing recall), accepting more false positives (lower precision) — because the cost of missing a disease is much higher than the cost of an extra test.

Question 4: Learning Rate [4 marks]

(a) Learning rate schedule [2 marks]:

Example: Step Decay — the learning rate is reduced by a factor (e.g., halved) every N epochs.

$$lr_t = lr_0 \times \gamma^{\lfloor t / N \rfloor}$$

For example, starting with lr=0.01, reducing by factor 0.5 every 50 epochs:

• Epochs 1-50: lr = 0.01
• Epochs 51-100: lr = 0.005
• Epochs 101-150: lr = 0.0025

Why beneficial: A high initial learning rate allows the optimiser to make large updates and quickly approach a good region of the loss landscape. As training progresses and the model approaches an optimum, the large learning rate would cause overshooting — oscillating around the minimum without converging precisely. By reducing the learning rate over time, the optimiser can make finer adjustments near the optimum, leading to better final performance.

(b) Adam's components [2 marks]:

Adam combines:

1. Momentum (from SGD with Momentum): Adam maintains an exponentially decaying average of past gradients (first moment estimate). This smooths the optimisation trajectory, reduces oscillations, and accelerates convergence in consistent gradient directions.

2. RMSProp: Adam also maintains an exponentially decaying average of past squared gradients (second moment estimate). This provides per-parameter adaptive learning rates — parameters with large recent gradients get smaller learning rates (prevents overshooting), while parameters with small gradients get larger learning rates (speeds up learning in flat regions).

By combining both, Adam provides momentum-like acceleration AND per-parameter adaptation, making it robust across a wide range of architectures and problems, which is why it's the most popular default optimiser.

Question 5: RNN and Transformer [4 marks]

(a) Vanishing gradient problem [2 marks]:

In RNNs, during backpropagation through time (BPTT), the gradient of the loss with respect to earlier time steps requires multiplying gradients across many time steps:

$$\frac{\partial L}{\partial h_1} = \frac{\partial L}{\partial h_T} \cdot \prod_{t=2}^{T} \frac{\partial h_t}{\partial h_{t-1}}$$

Each factor $\frac{\partial h_t}{\partial h_{t-1}}$ involves the recurrent weight matrix and the derivative of the activation function. If these factors are consistently less than 1, their product decreases exponentially with sequence length. For a sequence of length 100, the gradient at step 1 might be $0.9^{100} \approx 0.00003$ of the gradient at step 100.

This makes it extremely difficult for the model to learn that early inputs in a sequence are relevant to later outputs — the gradient signal is too weak for the optimiser to update the weights meaningfully. For example, in a sentence like "The man who lived in the big house near the river that runs through the valley was happy", the RNN struggles to connect "man" (position 2) with "was" (position 17) because the gradient vanishes over those 15 steps.

(b) Two Transformer mechanisms [2 marks]:

1. Self-Attention Mechanism: The Transformer computes attention scores between ALL pairs of positions in the sequence simultaneously. Each position creates Query, Key, and Value vectors; the attention weight between any two positions is the softmax-normalised dot product of their Q and K vectors. This creates direct connections between any two positions regardless of distance — solving the long-range dependency problem without sequential processing. Crucially, all attention computations are matrix multiplications that can be done in parallel.

2. Positional Encoding: Since the Transformer processes all positions simultaneously (no sequential structure), it has no inherent notion of order — "dog bites man" and "man bites dog" would produce the same attention scores. Positional encoding adds a unique signal to each position's embedding using sinusoidal functions of different frequencies (or learned embeddings). This allows the model to distinguish and reason about the order of elements in the sequence while still processing them in parallel.

Question 6: CNN [4 marks]

(a) Calculation [3 marks]:

Layer: Conv1 (padding = 1, same)
  Input:  [64, 64, 1]
  Calc:   (64 + 2×1 - 3) / 1 + 1 = 64
  Output: [64, 64, 32]       ← 32 filters

Layer: MaxPool1 (k=2, s=2)
  Input:  [64, 64, 32]
  Calc:   (64 - 2) / 2 + 1 = 32
  Output: [32, 32, 32]       ← depth unchanged

Layer: Conv2 (padding = 0, valid)
  Input:  [32, 32, 32]
  Calc:   (32 + 2×0 - 3) / 2 + 1 = 29/2 + 1 = 15.5 → floor = 15
  Output: [15, 15, 64]       ← 64 filters

Layer: Conv3 (padding = 1, same)
  Input:  [15, 15, 64]
  Calc:   (15 + 2×1 - 3) / 1 + 1 = 15
  Output: [15, 15, 128]      ← 128 filters

Layer: MaxPool2 (k=2, s=2)
  Input:  [15, 15, 128]
  Calc:   (15 - 2) / 2 + 1 = 13/2 + 1 = 7.5 → floor = 7
  Output: [7, 7, 128]        ← depth unchanged

Flatten: 7 × 7 × 128 = 6272

Answer: The fully connected layer has 6,272 inputs.

(b) Filter visualisation [1 mark]:

Conv1 filters (close to input): Would detect low-level features such as edges (horizontal, vertical, diagonal), simple textures, and basic colour gradients. These are the building blocks of visual recognition.

Conv3 filters (deeper): Would detect higher-level features that combine the patterns from earlier layers — such as corners, shapes, parts of objects, or more complex textures. These represent more abstract, semantically meaningful features.

The difference is due to hierarchical feature learning: each layer builds increasingly complex representations by combining patterns detected by the previous layer. Early layers = simple edges → middle layers = shapes and patterns → deep layers = object parts and complex structures.

Question 7: Activation Functions and DNN [4 marks]

(a) Sigmoid vs Softmax [2 marks]:

Sigmoid: Outputs a value between 0 and 1 for each neuron independently. Each output can be interpreted as the probability of a binary event, without any relationship to other outputs.

• Best for: Multi-label classification (e.g., image tagging where a photo can be both "sunset" AND "beach" AND "ocean"). Each label is predicted independently.

Softmax: Outputs values between 0 and 1 that sum to 1 across all output neurons. Creates a probability distribution over classes.

• Best for: Multi-class classification (e.g., classifying an image as exactly one of "cat", "dog", or "bird"). The classes are mutually exclusive.

Key difference: Sigmoid treats each output independently; softmax creates competition between outputs (increasing one probability decreases others). Using softmax for multi-label tasks would be wrong because detecting one label would suppress detection of other labels.

(b) Two strategies for very deep networks [2 marks]:

1. Skip/Residual Connections (as in ResNet): Instead of learning the full mapping $H(x) = F(x)$, the network learns the residual $F(x) = H(x) - x$, and the output becomes $F(x) + x$. The $+x$ shortcut connection creates a direct gradient path from later layers to earlier layers, bypassing the many intermediate multiplications that cause gradients to vanish. Even if $F(x)$ has vanishing gradients, the gradient flows through the skip connection unimpeded. This allows training of networks with 100+ layers, whereas without skip connections, training degrades beyond ~20 layers.

2. Batch Normalisation: Normalises activations at each layer to have zero mean and unit variance (computed per mini-batch), then applies learned scale and shift parameters. This addresses two key deep learning challenges:

• Vanishing/exploding gradients: By keeping activations in a controlled range, gradients remain well-scaled throughout the network
• Internal covariate shift: Each layer's input distribution remains stable during training, allowing higher learning rates and faster convergence

Together, these techniques have enabled the training of networks with hundreds of layers (e.g., ResNet-152, ResNet-1001) that were previously impossible to optimise.

Scoring Summary:

Mock Exam 3 — 20 Marks (2025 Format)

Instructions: 60 minutes total. 5 minutes reading + 55 minutes writing. Double-sided handwritten cheat sheet allowed. No calculators. Attempt ALL questions. Be concise and clear.

Question 1: Dataset cleaning [2 marks]

Consider the following summary of a dataset containing 5 attributes and 6,000 samples. The table gives the type and number of missing values for each attribute, as well as the mean, standard deviation, minimum and maximum value for each numerical attribute.

For each of the following cleaning steps, explain if it makes sense to apply it to this dataset. Briefly justify your answers. [0.5 mark per step]

• (a) Missing value replacement based on most frequent value.
• (b) Missing value replacement based on median value.
• (c) Removing an attribute.
• (d) Outlier removal.

Question 2: Design choices [3 marks]

You trained a neural network with the following settings: 2 hidden layers with ReLU units, 16 neurons per layer, Xavier initialisation, trained for 200 epochs with a learning rate of 0.001 and L2 regularisation. The training accuracy is 58% and the validation accuracy is 56%. You know that state-of-the-art models achieve 94% on the same problem.

For each of the following suggestions, explain if it is likely to improve the validation accuracy and why. [1 mark per suggestion]

• (a) Adding dropout (rate=0.3) to each hidden layer.
• (b) Increasing the model to 5 layers with 128 neurons per layer and removing the L2 regularisation.
• (c) Using a learning rate schedule that starts at 0.01 and decays over time.

Question 3: Evaluation & Confusion Matrix [4 marks]

Part A [2 marks]

A fraud detection model is evaluated on a dataset of 10,000 transactions (100 fraudulent, 9,900 legitimate). The confusion matrix is:

(a) Calculate the accuracy, precision, and recall. [1 mark]

(b) The bank is considering deploying this model to automatically block fraudulent transactions. Discuss whether this model is suitable for this purpose. Consider what each type of error means in this context. [1 mark]

Part B [2 marks]

A different team builds a second fraud detection model, evaluated on the same dataset. The confusion matrix is:

(c) Calculate the accuracy, precision, and recall for Model B. [1 mark]

(d) Compare Model A and Model B. Which model would you recommend for the bank's fraud detection system? Justify your answer. [1 mark]

Question 4: Activation functions [3 marks]

(a) ReLU is the most commonly used activation function in hidden layers of deep neural networks. Explain one problem that can arise when using ReLU, and how LeakyReLU mitigates it. [1 mark]

(b) A hospital is building a neural network to screen X-ray images. The model should detect whether each of the following conditions is present: fracture, pneumonia, tumour. A single X-ray image may show multiple conditions at the same time.

What activation function would you choose for the output layer? Justify your choice by explaining why another common choice would not work here. [2 marks]

Question 5: Learning rate & optimisers [4 marks]

(a) Explain in your own words what batch normalisation does and give two effects it has on the training of a neural network. [2 marks]

(b) The Adam optimiser is often described as combining the benefits of two other optimisation techniques. Name these two techniques and briefly explain what each contributes to Adam. [2 marks]

Question 6: CNNs [4 marks]

Consider the following CNN architecture:

• Input images' shape: [40, 40, 3]
• Convolutional layer 1: 16 filters with kernel size = 5, stride = 2 and padding = 0 (valid).
• Max pooling layer 1: kernel size = 2, stride = 2.
• Convolutional layer 2: 32 filters with kernel size = 3, stride = 1 and padding = 1 (same).
• Max pooling layer 2: kernel size = 2, stride = 2.
• Fully connected layer 1: ? inputs and 10 outputs.

(a) What is the number of inputs to the fully connected layer? Choose 1 answer. [1 mark]

• i. 128
• ii. 512
• iii. 800
• iv. 1152
• v. 2048

(b) Show your workings for each layer. [3 marks]

END OF MOCK EXAM 3

Mock Exam 3 — Answer Key & Detailed Explanations

Question 1: Dataset cleaning [2 marks]

(0.5 mark per step)

(a) Most frequent value imputation:

Yes, for Attribute 2 (categorical, 5 missing values). Most frequent value is the standard strategy for categorical data with missing values. It is not applicable to numerical or binary attributes here (Attribute 5 has no missing values).

(b) Median value imputation:

Yes, for Attribute 1 (numerical, 40 missing values out of 6,000 — a reasonable amount). Median is preferred over mean because the max (620) is relatively far from the mean (320), and Attribute 4's extreme values (std=4200 >> mean=50) also suggest the data may contain outliers. Median is more robust to outliers than mean.

(c) Removing an attribute:

Yes, for Attribute 3 — it has 5,980 out of 6,000 values missing (99.7%). Imputing this many values would create almost entirely fabricated data, which is misleading and unlikely to help the model. It makes sense to remove this attribute completely.

(d) Outlier removal:

Yes, for Attribute 4 — the standard deviation (4,200) is much larger than the mean (50), and the range is extreme (from -85,000 to 90,000). This strongly suggests the presence of outliers that could distort model training. Outlier removal or clipping should be considered.

Question 2: Design choices [3 marks]

Diagnosis: The model is underfitting (high bias). Both training (58%) and validation (56%) accuracies are low and close together, while 94% is achievable. The model lacks the capacity to capture the patterns in the data.

(a) Adding dropout (rate=0.3): [1 mark]

No, this is unlikely to improve validation accuracy. Dropout is a regularisation technique that reduces overfitting by randomly deactivating neurons during training. Since the model is already underfitting (not fitting even the training data well), adding more regularisation would further constrain the model's capacity, potentially making the underfitting worse. The model already has L2 regularisation — adding dropout on top of that would make it even harder to learn.

(b) Increasing to 5 layers with 128 neurons and removing L2: [1 mark]

Yes, this is likely to improve validation accuracy. The current model (2 layers, 16 neurons) has very limited capacity, which is the likely cause of underfitting. Increasing the number of layers and neurons gives the model more representational power to learn complex patterns. Removing L2 regularisation also makes sense because the model needs more freedom to fit the data, not less. However, after increasing capacity, overfitting should be monitored — regularisation may need to be re-introduced later if overfitting occurs.

(c) Learning rate schedule starting at 0.01 with decay: [1 mark]

Yes, this could help improve validation accuracy. The current learning rate (0.001) may be too small, causing the optimisation to converge slowly or get stuck in a poor local minimum. Starting with a higher learning rate (0.01) allows the optimiser to make larger updates and explore the loss landscape more effectively. The decay then helps the optimiser settle into a good minimum as training progresses, avoiding overshooting. This is a reasonable change for an underfitting model where slow convergence may be part of the problem.

Question 3: Evaluation & Confusion Matrix [4 marks]

Part A [2 marks]

(a) Calculations for Model A: [1 mark]

$$Accuracy = \frac{TP + TN}{Total} = \frac{80 + 8910}{10000} = \frac{8990}{10000} = 0.899 \text{ (89.9%)}$$

$$Precision = \frac{TP}{TP + FP} = \frac{80}{80 + 990} = \frac{80}{1070} \approx 0.075 \text{ (7.5%)}$$

$$Recall = \frac{TP}{TP + FN} = \frac{80}{80 + 20} = \frac{80}{100} = 0.80 \text{ (80%)}$$

(b) Suitability analysis for Model A: [1 mark]

Despite the seemingly good accuracy (89.9%), this model is not suitable for automatically blocking fraudulent transactions.

The critical problem is the extremely low precision (7.5%). Of every 1,070 transactions the model flags as fraud, only 80 are actually fraudulent — meaning 92.5% of blocked transactions are legitimate. This would cause massive disruption: customers would have their legitimate purchases blocked constantly, leading to poor user experience and potential loss of business.

The recall (80%) is reasonable — the model catches 80% of actual fraud. But the cost of the low precision (blocking 990 legitimate transactions to catch 80 fraudulent ones) is too high for automatic blocking. The model could be used for flagging transactions for human review instead.

Part B [2 marks]

(c) Calculations for Model B: [1 mark]

$$Accuracy = \frac{TP + TN}{Total} = \frac{95 + 9850}{10000} = \frac{9945}{10000} = 0.9945 \text{ (99.45%)}$$

$$Precision = \frac{TP}{TP + FP} = \frac{95}{95 + 50} = \frac{95}{145} \approx 0.655 \text{ (65.5%)}$$

$$Recall = \frac{TP}{TP + FN} = \frac{95}{95 + 5} = \frac{95}{100} = 0.95 \text{ (95%)}$$

(d) Comparison and recommendation: [1 mark]

Model B is clearly better and should be recommended for the bank's fraud detection system.

Model B is superior in every metric:

• Higher recall (95% vs 80%): Model B catches 95 out of 100 fraudulent transactions, missing only 5. In fraud detection, high recall is critical because each missed fraud can mean significant financial loss.
• Much higher precision (65.5% vs 7.5%): Model B produces far fewer false alarms. Only about 1 in 3 flagged transactions is a false positive, compared to Model A where 12 out of 13 flags are false positives. This means fewer legitimate customers are impacted.
• Higher accuracy (99.45% vs 89.9%): Model B correctly classifies almost all transactions.

Model A's low precision makes it unusable for automatic blocking. Model B, while not perfect (its precision could still be improved), strikes a much better balance and is a viable candidate for deployment, potentially with human review for flagged transactions.

Question 4: Activation functions [3 marks]

(a) ReLU problem and LeakyReLU [1 mark]:

A potential problem with ReLU is the dying ReLU problem. When a neuron's input is negative, ReLU outputs exactly 0, and the gradient is also 0. If a neuron consistently receives negative inputs (e.g., due to a large negative bias or an unfortunate weight update), it will always output 0 and its weights will never be updated — the neuron effectively "dies" and stops contributing to the network. (0.5 mark)

LeakyReLU mitigates this by replacing the zero output for negative inputs with a small slope ($\alpha x$, where $\alpha$ is typically 0.01–0.3). This ensures that even when the input is negative, there is still a small, non-zero gradient flowing back. The neuron can still receive gradient signal and recover, preventing it from dying permanently. (0.5 mark)

(b) Output activation function for multi-label X-ray classification [2 marks]:

The appropriate activation function is sigmoid. (1 mark)

Since a single X-ray can show multiple conditions simultaneously (e.g., both a fracture AND pneumonia), this is a multi-label classification problem. Each condition must be predicted independently — the presence of one condition does not affect the prediction of another.

Sigmoid outputs a probability between 0 and 1 for each output node independently. Each output can be interpreted as "the probability that this specific condition is present", and multiple outputs can be high at the same time.

Softmax would not work because softmax creates a probability distribution that sums to 1 across all outputs. This means increasing the probability of one condition would decrease the probabilities of others. If a patient has both fracture and pneumonia, softmax would suppress one to boost the other — making it impossible to correctly predict both conditions simultaneously. Softmax is designed for mutually exclusive classes, but medical conditions are not mutually exclusive. (1 mark)

Question 5: Learning rate & optimisers [4 marks]

(a) Batch normalisation [2 marks]:

Batch normalisation normalises the activations at each layer within a mini-batch to have zero mean and unit variance, then applies learned scale ($\gamma$) and shift ($\beta$) parameters.

Effect 1: Speeds up training / allows higher learning rates. (1 mark) By keeping the input distribution to each layer stable throughout training, batch normalisation reduces internal covariate shift. The activations remain in a range where gradients are meaningful (not too small for sigmoid/tanh, not too large to explode). This allows the use of higher learning rates, leading to faster convergence.

Effect 2: Has a regularising effect. (1 mark) Because normalisation statistics (mean and variance) are computed per mini-batch rather than over the full dataset, each sample's normalised value depends on which other samples happen to be in the same batch. This introduces noise into the activations, similar to dropout. This noise acts as implicit regularisation, helping to prevent overfitting without explicitly adding a regularisation term.

(b) Adam's two components [2 marks]:

Adam combines:

1. Momentum (from SGD with Momentum): (1 mark) Adam maintains an exponentially decaying average of past gradients (first moment estimate, $m_t$). This smooths the optimisation trajectory by averaging out noisy gradient fluctuations, and accelerates convergence in directions where the gradient is consistent. It helps the optimiser build up speed in consistent directions, like a ball rolling downhill.

2. RMSProp: (1 mark) Adam maintains an exponentially decaying average of past squared gradients (second moment estimate, $v_t$). This provides per-parameter adaptive learning rates. Parameters that have had large recent gradients get smaller learning rates (prevents overshooting), while parameters with small gradients get larger learning rates (speeds up learning in flat regions of the loss landscape). This adaptation makes Adam robust across different architectures.

Question 6: CNNs [4 marks]

(a) Answer: ii. 512 [1 mark]

(b) Step-by-step calculations: [3 marks]

Output size of convolution: $\left\lfloor\frac{n_H + 2p - f}{s} + 1\right\rfloor \times \left\lfloor\frac{n_W + 2p - f}{s} + 1\right\rfloor \times n'_C$

Output size of pooling: $\left\lfloor\frac{n_H - f}{s} + 1\right\rfloor \times \left\lfloor\frac{n_W - f}{s} + 1\right\rfloor \times n_C$

Layer: Conv1 (padding = 0, stride = 2, kernel = 5)
  Input:  [40, 40, 3]
  Calc:   floor((40 + 2×0 - 5) / 2) + 1 = floor(35/2) + 1 = 17 + 1 = 18
  Output: [18, 18, 16]       ← 16 filters

Layer: MaxPool1 (kernel = 2, stride = 2)
  Input:  [18, 18, 16]
  Calc:   floor((18 - 2) / 2) + 1 = floor(16/2) + 1 = 8 + 1 = 9
  Output: [9, 9, 16]         ← depth unchanged

Layer: Conv2 (padding = 1, stride = 1, kernel = 3, same)
  Input:  [9, 9, 16]
  Calc:   floor((9 + 2×1 - 3) / 1) + 1 = floor(8) + 1 = 9
  Output: [9, 9, 32]         ← 32 filters

Layer: MaxPool2 (kernel = 2, stride = 2)
  Input:  [9, 9, 32]
  Calc:   floor((9 - 2) / 2) + 1 = floor(7/2) + 1 = 3 + 1 = 4
  Output: [4, 4, 32]         ← depth unchanged

Flatten: 4 × 4 × 32 = 512

Answer: The fully connected layer has 512 inputs.

Scoring Summary:

📝 Confusion Matrix 完整答题手册

零、拿到题目怎么分析（思维流程）

拿到一道 Confusion Matrix 题，脑子里按这个顺序跑一遍：

Step 0：读题，搞清楚"Positive 是什么"

这是最容易搞混的地方。题目说的 Positive 到底是谁？

确认好 Positive 之后，再去对应 Confusion Matrix 的四个格子。

Step 1：算完数字之后，先看三个信号

信号 1 — 数据是否 imbalanced？

看 Positive 和 Negative 的总数比例：

• Positive 总数 = TP + FN（实际为正的那一列加起来）
• Negative 总数 = FP + TN（实际为负的那一列加起来）

如果比例严重失衡（比如 1:99），accuracy 就不可信了。 判断方法：算一下"如果模型全猜 Negative，accuracy 是多少？" 如果这个 baseline accuracy 已经很高（比如 99%），说明 accuracy 没有参考价值。

信号 2 — Precision 和 Recall 谁高谁低？

考试英文：

• 激进型："The model appears to be very sensitive (low threshold), predicting most instances as positive. This results in high recall but very low precision."
• 保守型："The model appears to be very conservative, only predicting positive when highly confident. This results in high precision but low recall."

信号 3 — FP 和 FN 各有多少？哪个更致命？

这一步要结合题目场景：

• 看 FP 的数量 → 想象这些"冤枉"的后果
• 看 FN 的数量 → 想象这些"放过"的后果
• 哪个后果更严重 → 那个对应的 metric 就更重要

口诀：FP 多 → Precision 差 → 误报严重；FN 多 → Recall 差 → 漏报严重

Step 2：用真题示例走一遍完整分析

示例：2024 真题 Q3（1000 instances）

                Positive    Negative
Predicted +       500         400
Predicted -         0         100

分析流程：

1. Positive 是什么？ 题目没给具体场景，只需要通用分析
2. 数据 imbalanced 吗？ Positive 总数 = 500+0 = 500，Negative 总数 = 400+100 = 500。比例 1:1，不 imbalanced
3. 算数：
   • Accuracy = (500+100)/1000 = 60%
   • Precision = 500/(500+400) = 500/900 ≈ 55.6%
   • Recall = 500/(500+0) = 100%
4. 信号判断： Recall=100% 但 Precision=55.6% → 模型是"激进型"
5. 具体分析： FN=0（没有漏报），FP=400（大量误报）。模型把几乎所有东西都预测成 Positive，所以当然不会漏掉任何真正的 Positive（recall=100%），但代价是把 400 个 Negative 也错误地标成了 Positive
6. 结论（真题标答原文）： "The model is very good at classifying true examples, but it does poorly at classifying negative examples. The model seems to be very sensitive (positive prediction threshold very low) and ends up predicting most examples as positive."

示例：Practice Test Q4（100 instances）

                Positive    Negative
Predicted +         5          20
Predicted -        10          65

分析流程：

1. 数据 imbalanced 吗？ Positive=15，Negative=85。有一定 imbalance（15:85）
2. 算数：
   • Accuracy = (5+65)/100 = 70%
   • Recall = 5/(5+10) = 5/15 = 33.3%
3. 信号判断： Accuracy 70% 看起来还行，但 Recall 只有 33.3%
4. 分析（标答原文）： "The accuracy is relatively high (70%) but the recall is quite low (33%). This shows that even if the model seems to perform relatively well globally, it is not very good at predicting the positive class. This comes from class imbalance, with the positive class being the minority class here. If we care about maximising the true positives, we should consider that this model does not perform well."

Step 3：判断"适不适合部署"的决策树

题目问 "Is this model suitable?"
           │
           ▼
   有没有 class imbalance？
      │              │
     有              没有
      │              │
      ▼              ▼
  Accuracy 可信吗？   直接看 Accuracy
  （算 baseline）    是否达标
      │
      ▼
  看 Precision 和 Recall
      │
      ▼
  场景中 FP 和 FN 哪个更致命？
      │              │
   FN 更致命         FP 更致命
   (医学/欺诈)       (垃圾邮件/封号)
      │              │
      ▼              ▼
  Recall 够高吗？   Precision 够高吗？
      │              │
   够 → 可以部署     够 → 可以部署
   不够 → 不适合     不够 → 不适合

一、定位四个格子

                   Actually Positive      Actually Negative
Predicted +            TP (真阳性)             FP (假阳性 / 误报)
Predicted -            FN (假阴性 / 漏报)       TN (真阴性)

记忆口诀：对角线是对的（TP, TN），反对角线是错的（FP, FN） 第二个字母看"实际"：P = 实际是 Positive，N = 实际是 Negative 第一个字母看"对错"：T = 猜对了，F = 猜错了

二、公式速查表

三、场景分析万能句式

A. 判断模型是否适合部署（开头句）

"Despite the seemingly high accuracy of X%, the model is not suitable / suitable for deployment because..."

"The accuracy alone is misleading in this context. The more informative metrics are precision and recall."

"To evaluate this model, we need to consider what each type of error means in practice."

B. 解释 False Positive 的后果（误报）

通用模板：

"A false positive means the model incorrectly predicts [事件] when it is actually [正常]. In this context, this would result in [具体后果]."

C. 解释 False Negative 的后果（漏报）

通用模板：

"A false negative means the model fails to detect [事件] when it is actually present. In this context, this could lead to [具体后果]."

D. 说明哪个 metric 更重要

Recall 更重要时（漏报代价 >> 误报代价）：

"In this scenario, the cost of a false negative is much higher than the cost of a false positive. Therefore, recall is the more critical metric — we need to ensure that as few positive cases as possible are missed, even if it means accepting more false alarms."

"A high recall is essential because missing a [positive case] could result in [严重后果], whereas a false positive only leads to [轻微后果]."

适用场景：疾病筛查、欺诈检测、安全系统、自动驾驶障碍检测

Precision 更重要时（误报代价 >> 漏报代价）：

"In this scenario, the cost of a false positive is higher than the cost of a false negative. Therefore, precision is the more critical metric — we need to ensure that when the model predicts positive, it is very likely to be correct."

"A high precision is essential because incorrectly flagging a [negative case] as positive would result in [严重后果], whereas missing a [positive case] only leads to [可接受后果]."

适用场景：垃圾邮件过滤、自动封号/封禁、推荐系统（宁缺毋滥）

两者都重要时：

"In this application, both false positives and false negatives have significant consequences. Therefore, the F1 score, which balances precision and recall, is the most appropriate metric."

E. Accuracy 陷阱（Class Imbalance）

"Although the accuracy appears high at X%, this is misleading due to class imbalance. The dataset contains [N positive] out of [Total] instances ([比例]). A naive model that simply predicts every instance as negative would achieve [Y%] accuracy without detecting any positive cases at all. Therefore, accuracy is not a reliable metric here — precision and recall provide a much more meaningful evaluation."

具体套用示例（以本题 Q3 Part A 为例）：

"The accuracy is 89.9%, but this is misleading because the dataset is heavily imbalanced (only 100 fraud cases out of 10,000 transactions, or 1%). A model that predicts every transaction as legitimate would already achieve 99% accuracy. The precision of 7.5% reveals the real problem: the model flags too many legitimate transactions as fraud."

F. 模型对比句式

"Model B outperforms Model A across all metrics: higher accuracy (X% vs Y%), higher precision (X% vs Y%), and higher recall (X% vs Y%)."

"While Model A has higher recall, Model B has significantly better precision. Given that [场景分析], Model B is the better choice because..."

"There is a trade-off between the two models: Model A prioritises recall ([X%]) at the expense of precision ([Y%]), while Model B achieves better precision ([X%]) with slightly lower recall ([Y%]). For this application, [选择] is more appropriate because..."

G. 提出改进建议

"To improve this model, the classification threshold could be adjusted. Lowering the threshold would increase recall (catching more positive cases) but decrease precision (more false alarms). Raising the threshold would have the opposite effect."

"One practical solution is to use the model for flagging rather than automatic action. Flagged cases can be reviewed by a human, combining the model's detection ability with human judgment to reduce false positives."

"The model could be retrained with class weighting or oversampling of the minority class to improve its sensitivity to positive cases."

四、答题模板（完整结构）

遇到 Confusion Matrix 题，按这个顺序写：

Step 1: 算数（写公式 + 代入 + 结果）
  Accuracy = (TP + TN) / Total = ... = X%
  Precision = TP / (TP + FP) = ... = X%
  Recall = TP / (TP + FN) = ... = X%

Step 2: 判断整体表现（一句话）
  "The model has high/low accuracy/precision/recall."

Step 3: 场景分析（这才是得分关键）
  "In this context, a false [positive/negative] means..."
  "Therefore, [precision/recall] is the more critical metric."

Step 4: 结论
  "This model is / is not suitable for deployment because..."
  "To improve, we could..."