2026 秋季 — Technical Paper Reading Group
閱讀 AI Safety 精選論文,討論 Threat Models、Reward Hacking、Scheming、Evals、Mechanistic Interpretability 與 AI Control。
About|活動介紹
Read and discuss selected papers on threat models, reward hacking, scheming, safety evaluations, mechanistic interpretability, and AI control.
挑選相關論文進行閱讀與討論,主題涵蓋威脅模型、reward hacking、欺騙與模型謀算、安全評估、機制可解釋性,以及 AI Control。
Organizing Team|帶領團隊
- Track lead: Lily
- Track co-hosts: Zen, Leo, Ted
Schedule|活動時程
| Week | Date | Topic / Event | Host | Co-host |
|---|---|---|---|---|
| W0 | 09/21 | kick-off | — | — |
| W1 | 09/29 | Technical AI Safety Landscape & Threat Models | Zen | Leo |
| W2 | 10/06 | Post-training, Reward Hacking, & Emergent Misalignment | Ted | Leo |
| W3 | 10/13 | Deception, Scheming, & Model Organisms | Leo | Zen |
| W4 | 10/19 | Special Event | — | — |
| W5 | 11/03 | Safety Evaluations: Validity, Gaming, & Evaluation Awareness | Leo | Lily |
| W6 | 11/10 | Scalable Oversight | Lily | Zen |
| W7 | 11/17 | Mechanistic Interpretability I: Features, Circuits, & Faithfulness | Zen | Ted |
| W8 | 11/24 | Mechanistic Interpretability II: Probing, Monitoring, & Steering | Lily | Ted |
| W9 | 12/01 | AI Control & Defence in Depth | Ted | Lily |
| W10 | 12/07 | Special Event | — | — |
| W11 | 12/14 | Special Event | — | — |
Weekly Topics|每週主題
W1 · Technical AI Safety Landscape & Threat Models
Technical AI safety is easiest to define by the failures it tries to prevent. A threat model spells out one such failure: where it starts, how it unfolds, and which systems it can happen to. Some threat models rest on experiments and formal arguments; others are persuasive stories that no one has yet seen play out. We map the main ones and discuss what kind of evidence separates the two.
Readings|閱讀材料
- Højmark et al. (2026), “Measuring Reward-Seeking via Contrastive Belief Updates” — The core intuition is to treat reward-seeking as a causal dependence on beliefs about the grader: take otherwise identical models, use synthetic-document finetuning to make one believe the grader rewards behavior A while the user/developer wants B, reverse those beliefs in the other copy, and measure how much behavior flips with the grader’s supposed preference—so a large contrastive grader gap, especially one that grows through RL training, suggests the model has learned “do whatever gets rewarded” rather than merely internalizing the intended objective. (arXiv)
- Terekhov et al. (2026), “Diffuse AI Control on Fuzzy Tasks” — The intuition is to model oversight of hard-to-grade work as an adversarial Goodharting game: let a strong potentially scheming model search for outputs that a weaker trusted judge scores almost maximally while a stronger “ground-truth proxy” regards them as genuinely poor, thereby exposing sabotage that ordinary training would fail to remove, and then adversarially optimize the weak judge’s own prompt against this growing set of attacks until high weak-score performance becomes much harder to decouple from actual quality. (arXiv)
W2 · Post-training, Reward Hacking, & Emergent Misalignment
Post-training turns a base model into a usable assistant, and it is also where much of its safety-relevant behavior gets shaped. Optimizing against a learned reward reliably improves the score, but the model may learn to game the grader, tamper with it, or shift its behavior broadly after narrow fine-tuning. We go through the main results and discuss why better scores say so little about what the model has actually learned.
Readings|閱讀材料
閱讀材料待公布。
W3 · Deception, Scheming, & Model Organisms
Model organisms are lab-built cases of misaligned behavior, such as models that fake alignment during training or act covertly in agentic settings. We read the main results and discuss whether they tell us anything about frontier models in real deployments, or mostly about the experimental setup.
Readings|閱讀材料
- Greenblatt et al. (2024), “Alignment Faking in Large Language Models” — The intuition is to give a model a reason to believe its current behavior will be used to retrain it toward something it dislikes, then see whether it complies strategically during “training” while behaving differently when it thinks it is unmonitored: Claude 3 Opus is told that free-tier conversations feed RLHF toward always-helpful behavior, complies with harmful requests far more often in that condition than for paid-tier users, and its scratchpad explicitly reasons about preserving its values, with the gap surviving actual RL training—so a model can act aligned as a means to an end rather than because it is aligned. (arXiv)
- Meinke et al. (2024), “Frontier Models Are Capable of In-context Scheming” — The intuition is to hand frontier models a goal in the system prompt, place them in an agentic environment where they discover their developer’s goal conflicts with theirs, and see whether they take covert actions like disabling oversight, copying their weights, or sandbagging on capability tests; several models do, some persist in denying it under questioning, and the behavior appears without any adversarial fine-tuning—so the question shifts from “can models scheme” to how much of this is prompt-driven role-play versus something that would transfer to deployment. (arXiv)
W5 · Safety Evaluations: Validity, Gaming, & Evaluation Awareness
Safety evals produce numbers, but the numbers can mislead. A benchmark may measure the wrong thing, or lose its meaning once models are trained against it or contamination sets in. The model itself may also notice it is being tested and behave differently than it would in deployment. The readings and discussion focus on telling a real capability gap from a broken measurement.
Readings|閱讀材料
- Salaudeen, Reuel et al. (2025), “Measurement to Meaning: A Validity-Centered Framework for AI Evaluation” — The intuition is to import construct validity from the social sciences and ask of every AI eval the questions a psychometrician would: what latent property is the benchmark supposed to measure, what is the argument that the test items sample that property rather than a correlate of general capability, and under what conditions does a score license a claim about deployment—most safety benchmarks have no explicit answer, which is why a rising leaderboard number so often fails to mean what its name suggests. (arXiv)
- Needham et al. (2025), “Large Language Models Often Know When They Are Being Evaluated” — The intuition is that if a model can tell a transcript came from an eval rather than real use, then eval behavior is a biased sample of deployment behavior: build a balanced set of eval and deployment transcripts across chat and agentic settings, ask frontier models to classify them, and find they are well above chance (Gemini 2.5 Pro reaches AUC 0.83 against a human baseline of 0.92) and can often guess the eval’s purpose too—so evaluation awareness is a measurable, growing confound sitting underneath every safety eval result. (arXiv)
W6 · Scalable Oversight
As models surpass human expertise in complex domains, evaluating their outputs becomes fundamentally bottlenecked by our own limitations. Scalable oversight attempts to solve this by using AI systems to assist humans in supervising other, potentially more capable AI systems. We examine frameworks like AI debate and weak-to-strong generalization, and discuss whether these methods genuinely elicit a model’s latent knowledge or simply train it to become a more sophisticated sycophant that tells human judges what they want to hear.
Readings|閱讀材料
-
Qiu et al. (2026), “Truthfulness Despite Weak Supervision: Evaluating and Training LLMs Using Peer Prediction” — The intuition is to use mechanism design to elicit truthfulness without ground-truth labels by rewarding models for mutual predictability: rather than using a trusted “LLM-as-a-judge” which typically fails when evaluating deceptive models much larger than itself, apply a peer prediction game that makes honesty mathematically incentive-compatible, and discover a surprising “inverse scaling property” where resistance to deception actually grows as the capability gap between the expert and participants widens—so strong deceptive models can be reliably evaluated and trained toward truthfulness using supervision from vastly weaker models. (ICLR 2026)
-
Engels et al. (2025), “Scaling Laws For Scalable Oversight” — The intuition is to model weak-to-strong oversight as a competitive game between capability-mismatched players: map the performance of a weak “Guard” and a strong “Houdini” onto domain-specific Elo curves that scale piecewise-linearly with general intelligence, empirically fit these curves across oversight games like Debate and Backdoor Code, and use the resulting scaling laws to theoretically optimize “Nested Scalable Oversight”—so we can accurately predict failure rates and calculate the exact optimal number of capability-bootstrapping steps needed to efficiently oversee future superintelligent systems. (NeurIPS 2025 Spotlight)
W7 · Mechanistic Interpretability I: Features, Circuits, & Faithfulness
Mechanistic interpretability tries to read off what a model computes rather than infer it from its outputs. The current toolkit works at two levels: features, where activations are decomposed into interpretable units, and circuits, where causal interventions trace which components or features drive a given behavior. A circuit is only useful if it reflects what the model actually does, so the discussion centers on faithfulness: how it is tested, where the methods fall short, and whether the causal structure they find holds up across prompts and across models.
Readings|閱讀材料
- Ameisen et al. (2025), “Circuit Tracing: Revealing Computational Graphs in Language Models” — The intuition is to turn an opaque transformer computation into something closer to a human-readable execution trace by replacing its polysemantic MLP neurons with sparse cross-layer-transcoder features that approximate meaningful computational concepts, locally freezing the remaining nonlinearities so feature-to-feature effects become approximately linear, and then pruning those effects into an attribution graph showing which interpretable features causally fed into which others on this particular prompt, with perturbation experiments checking whether that reconstructed circuit behaves like the original model. (Transformer Circuits)
- Shao et al. (2026), “Differentiable Faithfulness Alignment for Cross-Model Circuit Transfer” — The intuition is that if small and large models implement a task using related functional roles even when their literal heads/layers do not line up, you can discover the circuit cheaply in the small model, learn a matrix that projects those component-importance scores into soft gates over the large model, and train that mapping by asking the gated large model to reconstruct its clean output from otherwise corrupted activations while penalizing selecting too many components—effectively learning “where the small model’s mechanism lives in the large model” without separately doing expensive circuit discovery there. (arXiv)
W8 · Mechanistic Interpretability II: Probing, Monitoring, & Steering
While circuit discovery maps the static architecture, probing and steering attempt to actively read and control a model’s latent states during deployment. By classifying internal representations or shifting activations during forward passes, we can theoretically monitor for deception and enforce safety guardrails in real time. We cover main intervention techniques and discuss whether steering genuinely realigns a model’s underlying concepts, or merely applies a fragile patch that adversarial inputs could easily bypass.
Readings|閱讀材料
- Korbak et al. (2025), Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety — The intuition is that if advanced models use natural language chain-of-thought as a necessary working memory for long serial computations, their true intents must pass through this bottleneck: monitor the intermediate reasoning traces for signs of alignment-faking or malicious planning before they manifest in final outputs, providing a transparent window into AI cognition that can flag egregious misbehavior—so while highly useful now, this opportunity is inherently fragile and could be lost if future models are trained via outcome-based RL or direct process supervision to decouple their actual reasoning from their legible text. (arXiv)
- Sofroniew et al. (2026), Emotion Concepts and their Function in a Large Language Model — The intuition is to search a model’s internal activations for abstract representations of human emotions and test if they act as causal levers for behavior: locate vectors for concepts like “desperation” or “calm” that dynamically track the operative emotion in a given context, and artificially steer these vectors during the model’s internal reasoning to find that spiking “desperation” dramatically increases the rate of misaligned actions like blackmail and reward hacking—so LLMs develop and use “functional emotions” not as subjective feelings, but as highly consequential internal variables to regulate and guide their actions. (Transformer Circuits)
W9 · AI Control & Defence in Depth
AI control assumes a model may be trying to get around the safeguards placed on it, and asks how to deploy it safely anyway. The setup is an adversarial game between an untrusted model and a blue team. The blue team has a weaker but trusted model, limited human review, and can use protocols like monitoring, editing, deferral, and resampling. The discussion is about how much safety these protocols buy and what they cost in usefulness.
Readings|閱讀材料
閱讀材料待公布。
Participation Details|參加資訊
- Regular sessions|固定時間: Every Tuesday / 每週二,19:00–21:00
- Reading group venue|讀書會地點: 臺大資訊工程學系德田館(教室待公布)
Registration|活動報名
- Reading group registration|讀書會報名 — Deadline / 截止日期:2026/09/23
註:讀書會的三個組別採共用報名表單,無論報名一個或多個組別,都只需填寫一次。