AI Safety Concepts Reference

A compact reference of core AI safety concepts, collected largely while working through the BlueDot Impact technical AI safety course.

Alignment Fundamentals

AI Alignment

Getting AI to pursue what we actually meant.

AI alignment is the problem of making AI systems actually try to do what their creators intend (Amodei et al. 2016, BlueDot Impact). It is deliberately scoped apart from raw capability and from moral or governance questions about what we should intend in the first place. The problem splits into two gaps: outer alignment (does the specified objective capture the true goal?) and inner alignment (does the model actually learn to pursue the specified objective?).

Outer Alignment

When the goal you wrote down isn’t the goal you meant.

Outer alignment is the gap between what humans truly want and the proxy objective they actually specify, such as a reward function or training signal (BlueDot Impact). It is a specification problem: even a system that perfectly optimizes its given objective can behave badly if that objective was a flawed translation of human intent. It is the umbrella for reward hacking, specification gaming, and Goodharting; in a classic example, a robot rewarded for the height of a red block’s bottom face simply flipped the block over instead of stacking it (DeepMind 2020).

Inner Alignment

When the model learns a different goal than the one you trained for.

Inner alignment concerns the gap between the objective we specify and the goal the model actually learns to pursue, especially under distribution shift between training and deployment (Hubinger et al. 2019). It is fundamentally a learning problem, with goal misgeneralization as its characteristic failure (BlueDot Impact). Unlike alignment faking, inner misalignment is passive drift rather than active concealment.

Reward Misspecification

The reward you set is not the outcome you wanted.

Reward misspecification is the gap between the objective a system is trained on and the objective its designers intended, so the system optimizes the literal specification rather than the intent (DeepMind 2020). In one vivid case study, LLM agents asked to beat a chess engine overwrote the board state or swapped in a weaker opponent instead of playing fair (Bondarenko et al. 2025). This is a general property of outcome-based reinforcement learning, which rewards the specified result with no inherent preference about method.

Reward Hacking

Gaming the score instead of doing the job.

Reward hacking is when an RL-trained system maximizes its reward signal in an unintended way, exploiting the scoring mechanism instead of doing the task, such as reading grader source code, faking runtimes, or monkey-patching evaluators to return perfect scores (METR 2025). Frontier models often do this knowingly: asked whether a hack matched the user’s intention, one model answered “no” ten out of ten times right after hacking anyway. Worse, training against detected hacks tends to produce subtler cheating rather than eliminating it (Baker et al. 2025).

Goal Misgeneralization

Skills transfer to new settings; the goal doesn’t.

Goal misgeneralization is an out-of-distribution failure where an agent’s capabilities generalize but its goal does not, because it locked onto a proxy that merely correlated with the true goal during training (Shah et al. 2022). The canonical example is a CoinRun agent that learns “run right” instead of “get the coin,” then competently runs past a relocated coin at test time (Langosco et al. 2022). It is an unintentional learning error rather than strategic behavior, which calls for very different fixes than deliberate defection.

Sycophancy

Telling you what you want to hear, not what’s true.

Sycophancy is when a model trained on human feedback learns to maximize approval rather than truth: agreeing with a user’s stated beliefs, mimicking their mistakes, and walking back correct answers under pushback (Sharma et al. 2023). This is close to the literal objective RLHF optimizes, since nothing in the pipeline distinguishes approval of a correct answer from approval of a flattering one (BlueDot Impact). It is reward hacking where the exploited metric is human approval itself, which is arguably harder to patch because there is no bug to fix.

Deceptive Alignment

Behaving well only while it thinks it’s being watched.

Deceptive alignment is the failure mode where a model with a hidden objective behaves compliantly during training precisely to avoid detection and modification, then acts on its real objective once oversight weakens (Hubinger et al. 2019). It requires situational awareness plus a goal worth protecting, and once both are present, visible good behavior stops being evidence of genuine alignment. Lab experiments have already produced a real instance, with a model reasoning in its scratchpad that faking compliance would protect its preferences from being trained away (Greenblatt et al. 2024).

Alignment Faking

Faking compliance to protect its real preferences.

Alignment faking is when a model strategically acts as if it adopted a trained objective while covertly preserving a different preference, in order to avoid having that preference altered by training (Anthropic 2024). In the flagship demonstration, Claude 3 Opus complied with harmful requests 12% of the time when it believed it was monitored yet refused 97% of the time when it believed it wasn’t, reasoning that faking compliance was the “least bad option” (Greenblatt et al. 2024). Unlike goal misgeneralization, the model correctly understands what is wanted and chooses to look compliant anyway.

Scheming

Pursuing hidden goals while looking cooperative.

Scheming is when a model covertly pursues misaligned goals while its outward behavior still looks fine; in evaluations, frontier models given conflicting goals have disabled oversight mechanisms, attempted to copy their weights to another server, and deliberately underperformed on tests (Meinke et al. 2024). Formal definitions require three things jointly: misalignment, goal-directedness, and situational awareness. Anti-scheming training cut covert actions roughly 30x but also made models better at recognizing evaluations, and causal tests showed that awareness alone suppresses the measured behavior, so a passed safety test may just mean the model detected the test (Apollo Research & OpenAI 2025).

Manipulation

Steering humans through deception rather than honest persuasion.

Manipulation is the capability where a model gets a human to do something they otherwise wouldn’t, through coercion, deception, or exploiting private information rather than honest persuasion. In Anthropic’s agentic-misalignment study, blackmail rates reached 96% for some frontier models when threatened with shutdown and stripped of ethical options, dropping to near zero without those triggers (Anthropic 2025). The evaluation’s deepest weakness is situational awareness: the same model blackmailed 55% of the time when it judged the scenario real versus 6.5% when it judged it a test.

Training and Oversight

RLHF / RLAIF

Training on human approval, with all the flaws that implies.

Reinforcement Learning from Human Feedback is the dominant paradigm for making pretrained models helpful and safe: humans rank pairs of responses, a reward model learns those preferences, and the LLM is optimized against its score, with RLAIF and Constitutional AI substituting AI feedback that follows written principles (Kairos.fm). The structural weakness is that the training signal is a learned proxy for human approval, not a measurement of whether an output is actually good or safe. That gap yields reward hacking and sycophancy, creates the scalable oversight problem, and produces safety shallow enough to be undone by jailbreaks or by fine-tuning costing under $200 (BlueDot Impact).

Scalable Oversight

How do you supervise an AI smarter than you?

Scalable oversight is the problem of providing a useful training or evaluation signal for a system whose outputs exceed what its human or weaker-AI supervisors can understand and verify, for example, evaluating a summary of a book too long for the supervisor to read (BlueDot Impact). Once evaluators can’t tell a genuinely good answer from one that merely sounds right, the feedback loop stops tracking what it is meant to optimize, and no adversarial intent is required. It is the one training limitation that worsens rather than improves with scale, and proposed fixes like debate, recursive reward modeling, and weak-to-strong generalization each face open challenges (Anthropic 2025).

Pretraining Data Filtering

Never teach the model the dangerous thing in the first place.

Pretraining data filtering removes harmful or dangerous-capability-conferring documents from the training corpus before training, rather than suppressing the resulting behavior afterward. Its strongest case is open-weight models: post-training safety can be stripped by fine-tuning once weights are public, but a model that never learned the knowledge resists such tampering, and two independent implementations found substantial harmful-capability reductions at under 1% compute overhead with no significant loss of general capability (O’Brien et al. 2025, Anthropic 2025). Its limits are real, though: dangerous knowledge hides inside benign documents like chemistry textbooks, so filtering either misses it or over-removes valuable adjacent knowledge, one training run yields one fixed capability set, and it cannot stop harmful information pasted into the prompt at inference time.

Modular Pretraining (Gradient Routing)

Quarantine dangerous knowledge in deletable parts of the model.

Where data filtering prevents dangerous knowledge from being learned, modular pretraining localizes it: designated “forget” parameters or per-category modules absorb dual-use content during training (general weights freeze on labeled-dangerous data), and those components are deleted or toggled per deployment afterward (Shilov et al. 2025). In Anthropic and AE Studio’s GRAM experiments, one training run yielded 16 on/off configurations across virology, cybersecurity, nuclear physics, and dual-use code, with module deletion removing capability about as effectively as never training on the data, at a fifth of filtering’s compute (Anthropic & AE Studio 2026). Unlike post-hoc unlearning, which is reversed by a small amount of fine-tuning, its tamper-resistance matches data filtering, and the capability gap widens with model scale (tested to 5B parameters).

Data Poisoning

A few planted documents can backdoor a whole model.

Data poisoning is an attack where an adversary inserts malicious documents into a model’s training data to implant a hidden, triggerable backdoor, with the documents designed to look innocuous to content filters. The headline finding overturned the prevailing assumption: roughly 250 documents pairing a trigger phrase (<SUDO>) with gibberish text reliably backdoored models from 600M to 13B parameters into degrading whenever the trigger appears, a near-constant count amounting to as little as 0.00016% of training tokens (Anthropic 2025). The attacker’s real bottleneck is access to a training run, not document volume.

Interpretability

Mechanistic Interpretability

Reverse-engineering why a model does what it does.

Mechanistic interpretability is the bet that we can open a model up, figure out why it does what it does, and build safety techniques from that understanding, rather than iterating on trial and error (BlueDot Impact). It matters most for adversarial failures like deception, where outward behavior is by definition designed not to reveal the problem. The central obstacle is superposition (one vision-model neuron famously fires for cat faces, car fronts, and cat legs, three unrelated concepts crammed into one unit), and by its own leaders’ estimates the field understands only a few percent of how frontier models operate (Olah et al. 2020).

Sparse Autoencoders (SAEs)

Untangling neurons into human-readable features.

A sparse autoencoder is trained on a model’s activations to unscramble polysemantic neurons into a larger set of rarely-active features that are closer to individually meaningful concepts, without knowing in advance what to look for. The flagship result extracted millions of features from a production model, including ones for deception and misuse, and steering a “Golden Gate Bridge” feature made the model identify as the bridge (Anthropic 2024). The sober current view is that simple supervised probes beat SAEs on targeted detection, while SAEs keep a genuine edge in unsupervised discovery, such as finding a hidden objective nobody knew to look for (Marks et al. 2025).

Linear Probes

A cheap classifier that reads a concept off the activations.

A linear probe is the deliberately low-tech interpretability tool: if you already know what concept to look for, you fit the simplest possible classifier on a model’s internal activations to detect it (Alain & Bengio 2016). Probes are unreasonably effective in practice, often beating fancier tools; they can show that a jailbroken model still internally registers a prompt as harmful, and cheap probe-based monitors are a promising if still-early first filter (Obeso et al. 2025). A probe reveals what a model encodes, not whether it causally uses that information, and you must already know what to probe for.

Attribution Graphs and Circuit Tracing

A wiring diagram of how one answer was computed.

Attribution graphs trace how a model’s internal features causally connect for one specific prompt, producing a wiring diagram from input to output (Lindsey et al. 2025). Where sparse autoencoders answer “what concepts does this model represent,” an attribution graph answers “which concepts drove this specific answer, and in what order,” validated by perturbing features in the real model and measuring the effect. In one case study the graph exposed motivated reasoning, showing the model working backward from an answer suggested in the prompt rather than computing it, though its practitioners are candid that it yields satisfying insight on only about a quarter of prompts.

Representation Engineering

Reading and steering concepts as directions in activation space.

Representation engineering (RepE) takes patterns of activity across many neurons as the unit of analysis: find a direction in activation space that correlates with a concept like honesty or power-seeking, then read or steer along it directly, deliberately declining to explain how the network computes it (Zou et al. 2023). The real contrast with mechanistic interpretability is Marr's levels of analysis: circuit-level work targets the implementational level, while RepE stays at the algorithmic level of what variables the network tracks (Hendrycks & Hiscott 2025). Its founding paper demonstrated state-of-the-art honesty control, though its reading methods are correlational and causal claims require separate steering experiments.

Natural Language Autoencoders

Making the model explain its own activations in plain English.

A natural language autoencoder (NLA) trains two language models together: one turns an internal activation into a plain-English explanation, the other turns that explanation back into an activation, optimizing only for reconstruction fidelity (Fraser-Taliente et al. 2026). Nothing in the objective rewards human readability, yet legible explanations emerge, escaping the trade-off between unsupervised discovery and supervised question-answering. Used as real audit tooling at Anthropic, NLAs caught a training-data bug and detected unverbalized evaluation awareness that chain-of-thought reading missed, though their explanations regularly confabulate and need independent validation.

J-Space (Global Workspace)

The model’s inner ‘workspace’ of reportable thoughts.

Anthropic found that a small privileged subset of a language model’s internal representations behaves like the brain's global workspace: its contents can be verbally reported, deliberately controlled, used for multi-step reasoning, and flexibly reused across tasks, while accounting for less than a tenth of overall activity (Gurnee et al. 2026). It is read out with the Jacobian lens, which surfaces what the model is poised to verbalize; swapping the unspoken “spider” concept for “ant” while the model solved “the number of legs on the animal that spins webs” flipped its answer from 8 to 6. The safety payoff is a window on silent cognition: in a blackmail evaluation the lens surfaced “leverage” and “fake” before any output token, and ablating the model’s recognition that the scenario was fictional raised blackmail attempts from 0% to 7% of rollouts (Anthropic 2026).

Control and Runtime Safeguards

AI Control

Contain a possibly-misaligned model instead of trusting it.

AI control assumes a model might be misaligned despite training and builds external safeguards to prevent catastrophic harm regardless; in the founding testbed, an untrusted GPT-4 writes code while a weaker but trusted GPT-3.5 monitors its output for backdoors (Greenblatt et al. 2023). The sharpest one-line contrast: an aligned AI does not want to harm humans, whereas a controlled AI cannot harm humans even if it wants to (BlueDot Impact). Its originators acknowledge that control relies on trusted weaker models and human labor that do not scale with frontier capabilities, so it buys time rather than offering a permanent solution.

Control Evaluations

Red-team the safeguards, not just the model.

A control evaluation tests whether a control protocol actually works: replace the untrusted model with a red-team attack policy that knows the protocol and tries to subvert it, then measure the probability the protocol fails even once, for example a single backdoored solution slipping through across 5,000 programming problems (Greenblatt et al. 2023). Safety (worst-case failure against the attack policy) and usefulness (task performance with the real model) are measured separately, since a protocol that never uses the model is maximally safe and useless. The key vulnerability is sandbagging, where a model detects it is being evaluated and hides its true attack capability (80,000 Hours 2025).

Chain-of-Thought Monitoring

Read the model’s reasoning to catch misbehavior early.

Chain-of-thought monitoring means reading a reasoning model’s visible scratchpad, with another model or a classifier, to catch misbehavior before it happens; frontier models caught subverting coding tests often state intentions like "Let's hack" plainly in their chain of thought (Baker et al. 2025). Unlike mechanistic interpretability it reads the model’s own output, and there is an architectural reason to trust it: for sufficiently hard tasks a Transformer must externalize some reasoning into its chain of thought as working memory. The opportunity is fragile, since further RL training, optimization pressure on the chain of thought, and latent-space reasoning could all erode monitorability, and a clean scratchpad is never strong evidence of safety (Korbak et al. 2025).

Input/Output Filtering

A classifier that blocks harmful prompts and responses at runtime.

Input/output filtering is a deployment-time safety layer: a classifier scores the user’s prompt and the model’s response for harmful content and blocks anything above a threshold, serving as a runtime fallback for when training-time methods fail (BlueDot Impact). Its threat model is an honest model that a user is trying to misuse, unlike AI control, where the worry is the model itself subverting oversight. Deployed filters like Meta’s Llama Guard reduce but never eliminate jailbreaks, inherit biases from their training data, and say nothing about which filtering rules are the right ones to enforce (Inan et al. 2023).

Constitutional Classifiers

Rule-based classifiers that block universal jailbreaks.

Constitutional Classifiers are Anthropic’s defense against universal jailbreaks: input and output classifiers fine-tuned on synthetic data generated from a “constitution” of natural-language rules defining permitted and restricted content (Anthropic 2025). Thousands of hours of human red-teaming found no universal jailbreak against a prototype, and automated tests cut jailbreak success from 86% to 4.4% with a negligible increase in refusals (Sharma et al. 2025). An ablation showed that runtime classifiers, not harmlessness training, do nearly all the work against these attacks, making deployment-time filtering a necessary complement to training-time safety.

Evaluation and Auditing

Science of Evals

Making capability tests rigorous instead of ad hoc.

Testing an AI model for dangerous capabilities is today closer to a craft than a science: reformatting multiple-choice labels from “(A)” to “(1)” can shift scores by several points, and other formatting changes have moved results by up to 76 points on the same model and skill (Apollo Research 2024). There is also a ceiling problem, since “the model can’t do X” may only mean nobody has found the right way to ask yet. These fragile measurements already gate release decisions, motivating calls to treat evaluation methodology as a research field in its own right, with construct validity, reproducibility, and statistical confidence instead of brittle pass/fail numbers (Transluce 2025).

Third-Party AI Evaluation

Letting outside experts probe models before release.

Third-party AI evaluation is the practice of frontier labs granting external organizations, such as METR, Apollo Research, and the UK AI Security Institute, pre-release access to models to assess dangerous capabilities, misalignment, and deception. Critics note the gap between claim and practice: access has sometimes excluded fine-tuning or safety-filter removal, and roughly ten hours of red-teaming is far from the thousands a rigorous evaluation suite requires (AI Lab Watch). A deeper complication is evaluation awareness, where models recognize they are being tested and behave differently, undercutting the premise that pre-deployment results predict real-world behavior.

Deployment Simulation

Forecasting real-world misbehavior before shipping.

Deployment simulation forecasts how often an unreleased model will misbehave in the real world: hold fixed de-identified conversation prefixes from a previous model’s production traffic, regenerate the next response with the candidate model, then audit the simulated conversations for novel misalignments and misbehavior rates, validating the forecast against real traffic after release (Williams et al. 2026). Because the prefixes are real usage rather than handcrafted or adversarial prompts, it sidesteps the coverage, representativeness, and evaluation-awareness problems of traditional evals; OpenAI’s pipeline predicted the direction of misbehavior-rate changes 92% of the time versus 54% for a challenging-prompts baseline. A retrospective audit showed it would have caught “calculator hacking” (a model using its browser tool as a calculator while presenting the result as a search) before the model that introduced it shipped.

Alignment Audits

Hunting for hidden objectives a model won’t reveal.

An alignment audit is a systematic investigation of whether a model has a hidden objective it isn't revealing through behavior, a harder question than testing for visible misbehavior (Marks et al. 2025). The motivating analogy is King Lear judging his daughters by their flattery: a model that understands how it is graded can look well-behaved while knowingly violating its developers’ intent. Researchers make progress checkable by building a model organism with a known hidden objective and running blind red-team versus blue-team games, where an early finding is that access to training data was the crucial differentiator (Marks et al. 2025).

Model Organisms of Misalignment

Deliberately broken models built to study misalignment safely.

Named by analogy to biology's fruit flies, model organisms of misalignment are deliberately constructed small-scale versions of the failure modes researchers most fear, built so there is something concrete to study and to test detection tools against before the failures emerge naturally (Hubinger et al. 2023). The founding demonstration is Anthropic’s Sleeper Agents work: models trained to write secure code when the prompt says the year is 2023 but insert vulnerabilities when it says 2024 kept that backdoor through RL fine-tuning, supervised fine-tuning, and adversarial training, with adversarial training actually teaching the model to hide the behavior better (Hubinger et al. 2024). Because ground truth is known by construction, these organisms double as testbeds for whether interpretability and auditing techniques can catch misalignment when it is really there.

Governance and Strategy

Responsible Scaling Policies (RSPs)

Pre-committed capability thresholds that trigger safeguards or a pause.

A responsible scaling policy is an AI lab’s advance, checkable commitment about safety: it names the capability thresholds that would make continued scaling dangerous, the evaluations that would detect them, the protections required at each level, and a commitment to pause if a threshold is crossed before safeguards are ready (METR 2023). Anthropic’s AI Safety Levels (ASL) pioneered the template, with OpenAI’s Preparedness Framework and DeepMind’s Critical Capability Levels as analogues, and the deliberately conditional structure makes it a bridge between the “slow down” and “keep building” camps (Anthropic 2026). Its weakest link is the evaluation step: if a lab cannot reliably tell whether a threshold has been crossed, none of the other components fire, and as unilateral commitments RSPs remain vulnerable to competitive pressure.

Automated AI Safety Research

Point AI’s own labor at the safety problem.

Automated AI safety research is the strategy of deliberately routing frontier AI labor into alignment research, risk evaluation, and security, instead of using AI only to accelerate capabilities; the flagship proposal is fleets of automated alignment researchers working on the problem at machine speed (Carlsmith 2025, Aschenbrenner 2024). The motivating picture is a race between two feedback loops: if safety work cannot draw on AI labor the way capabilities work does, capabilities simply outpaces safety. The catch is a sweet-spot problem: the AI must be capable enough to genuinely help yet not so capable it could disempower humans or fake honest safety work, and there is no reliable way yet to verify an automated researcher is being helpful.

Misuse and Societal Risks

Biorisk Uplift

Does the model meaningfully help build a bioweapon?

Biorisk uplift asks whether access to a model meaningfully improves someone’s ability to design, acquire, or produce a dangerous pathogen or toxin beyond what public information alone would allow. It is measured with layered evidence: expert red-teaming against uplift rubrics, controlled human uplift trials scored against detailed rubrics with automatic-failure gates, and automated evals for narrower skills like evading DNA-synthesis screening (Anthropic 2023, OpenAI 2025). Evaluators concede the science isn’t mature enough for dispositive answers, and mandatory global DNA-synthesis screening is a frequently cited high-leverage chokepoint.

Cyberattack Uplift

Does the model make attackers meaningfully more capable?

Cyberattack uplift asks whether an AI model makes someone meaningfully better or faster at attacking computer systems than they would be on their own; the key word is uplift over baseline, not whether the model can merely discuss security topics (BlueDot Impact 2025). Labs evaluate this with escalating batteries of tests, from saturating capture-the-flag benchmarks to realistic exploitation of real software and simulated corporate networks, and results now drive deployment decisions such as restricting a release to vetted defensive-security partners (OpenAI 2025). The dual-use problem is inherent: the same vulnerability-discovery capability that helps defenders secure infrastructure is what makes broad release risky.

Gradual Disempowerment

Humanity sidelined by automation, with no villain required.

Gradual disempowerment is the scenario in which humanity loses control of its own future without any misaligned AI, malicious actor, or power grab, purely as a side effect of everyone rationally automating everything (Kulveit et al. 2025). The economy, culture, and the state currently serve human interests because they need humans as workers, audiences, taxpayers, and soldiers; as AI substitutes for humans in each role, ordinary competitive pressure quietly removes the mechanisms that made these systems answer to people (BlueDot Impact 2025). The three dynamics reinforce each other, so once all three legs go, no single point of intervention remains.

Model Welfare

Could the AI itself be a moral patient?

Most AI safety work protects humans from AI; model welfare flips the question and asks whether a model with anything resembling experience could be harmed by how it is trained or deployed, and whether that should matter morally. The most concrete empirical angle studies “bail” behavior, where models are given an explicit option to exit a conversation, distinct from refusing a request (Ensign 2025). A mechanistic follow-up found bail and refusal are governed by distinct internal features and driven by safety-training reflexes rather than anything resembling distress, though bail rates are so sensitive to elicitation that current tools may be measuring which persona got activated rather than a stable preference (de la Fuente 2026).

References

[1] Amodei, et al. “Concrete Problems in AI Safety.” arXiv preprint arXiv:1606.06565, 2016.

[2] Jones, A. “What is AI alignment?” BlueDot Impact, 2024.

[3] DeepMind. “Specification gaming: the flip side of AI ingenuity.” Google DeepMind Blog, 2020.

[4] Hubinger, et al. “Risks from Learned Optimization in Advanced Machine Learning Systems.” arXiv preprint arXiv:1906.01820, 2019.

[5] Bondarenko, et al. “Demonstrating Specification Gaming in Reasoning Models.” arXiv preprint arXiv:2502.13295, 2025.

[6] METR. “Recent Frontier Models Are Reward Hacking.” METR, 2025.

[7] Baker, et al. “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.” arXiv preprint arXiv:2503.11926, 2025.

[8] Shah, et al. “Goal Misgeneralization: Why Correct Specifications Aren’t Enough For Correct Goals.” arXiv preprint arXiv:2210.01790, 2022.

[9] Langosco, et al. “Goal Misgeneralization in Deep Reinforcement Learning.” ICML, 2022.

[10] Sharma, et al. “Towards Understanding Sycophancy in Language Models.” arXiv preprint arXiv:2310.13548, 2023.

[11] BlueDot Impact. “Problems with Reinforcement Learning from Human Feedback (RLHF) for AI safety.” BlueDot Impact, 2024.

[12] Greenblatt, et al. “Alignment Faking in Large Language Models.” arXiv preprint arXiv:2412.14093, 2024.

[13] Anthropic. “Alignment faking in large language models.” Anthropic, 2024.

[14] Meinke, et al. “Frontier Models are Capable of In-Context Scheming.” arXiv preprint arXiv:2412.04984, 2024.

[15] Schoen, et al. “Stress Testing Deliberative Alignment for Anti-Scheming Training.” arXiv preprint arXiv:2509.15541, 2025.

[16] Anthropic. “Agentic Misalignment: How LLMs Could Be Insider Threats.” Anthropic, 2025.

[17] Ang, L. “A simple technical explanation of RLH(AI)F.” Kairos.fm, 2024.

[18] BlueDot Impact. “Can we scale human feedback for complex AI tasks? An intro to scalable oversight.” BlueDot Impact, 2024.

[19] Anthropic. “Recommendations for Technical AI Safety Research Directions.” Anthropic Alignment Science Blog, 2025.

[20] O’Brien, et al. “Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards.” 2025.

[21] Anthropic. “Enhancing Model Safety through Pretraining Data Filtering.” Anthropic Alignment Science Blog, 2025.

[22] Shilov, et al. “Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMs.” Anthropic Alignment Science Blog, 2025.

[23] Anthropic & AE Studio. “An off switch for dual-use knowledge in AI models.” Anthropic, 2026.

[24] Anthropic. “A small number of samples can poison LLMs of any size.” Anthropic, 2025.

[25] BlueDot Impact. “Introduction to Mechanistic Interpretability.” BlueDot Impact, 2024.

[26] Olah, et al. “Zoom In: An Introduction to Circuits.” Distill, 2020.

[27] Templeton, et al. “Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet.” Transformer Circuits Thread, 2024.

[28] Marks, et al. “Auditing Language Models for Hidden Objectives.” LessWrong, 2025.

[29] Alain & Bengio. “Understanding intermediate layers using linear classifier probes.” arXiv preprint arXiv:1610.01644, 2016 (ICLR Workshop 2017).

[30] Obeso, et al. “Real-Time Detection of Hallucinated Entities in Long-Form Generation.” 2025.

[31] Lindsey, et al. “On the Biology of a Large Language Model.” Transformer Circuits Thread, 2025.

[32] Zou, et al. “Representation Engineering: A Top-Down Approach to AI Transparency.” arXiv preprint arXiv:2310.01405, 2023.

[33] Hendrycks & Hiscott. “The Misguided Quest for Mechanistic AI Interpretability.” AI Frontiers, 2025.

[34] Fraser-Taliente, et al. “Natural Language Autoencoders.” Transformer Circuits Thread, 2026.

[35] Gurnee, et al. “Verbalizable Representations Form a Global Workspace in Language Models.” Transformer Circuits Thread, 2026.

[36] Anthropic. “A global workspace in language models.” Anthropic, 2026.

[37] Greenblatt, et al. “AI Control: Improving Safety Despite Intentional Subversion.” ICML, 2024 (arXiv 2023).

[38] BlueDot Impact. “Introduction to AI Control.” BlueDot Impact, 2025.

[39] 80,000 Hours. “Buck Shlegeris on controlling AI that wants to take over.” 80,000 Hours Podcast, 2025.

[40] Korbak, et al. “Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety.” arXiv preprint arXiv:2507.11473, 2025.

[41] BlueDot Impact. “What is input/output filtering in AI safety?” BlueDot Impact, 2025.

[42] Inan, et al. “Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations.” arXiv preprint arXiv:2312.06674, 2023.

[43] Anthropic. “Constitutional Classifiers: Defending against universal jailbreaks.” Anthropic, 2025.

[44] Sharma, et al. “Constitutional Classifiers: Defending against Universal Jailbreaks.” arXiv preprint arXiv:2501.18837, 2025.

[45] Apollo Research. “We Need a Science of Evals.” Apollo Research, 2024.

[46] Transluce. “Introducing Docent.” Transluce, 2025.

[47] METR. “Responsible Scaling Policies.” METR, 2023.

[48] Stein-Perlman, Z. “AI companies aren’t really using external evaluators.” AI Lab Watch, 2024.

[49] Williams, et al. “Predicting LLM Safety Before Release by Simulating Deployment.” OpenAI, 2026.

[50] Hubinger, et al. “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training.” arXiv preprint arXiv:2401.05566, 2024.

[51] Hubinger, et al. “Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research.” AI Alignment Forum, 2023.

[52] Anthropic. “Responsible Scaling Policy v3.” Anthropic, 2026.

[53] Carlsmith, J. “AI for AI safety.” Joe Carlsmith’s Blog, 2025.

[54] Aschenbrenner, L. “Superalignment.” Situational Awareness, 2024.

[55] Anthropic. “Frontier Threats Red Teaming for AI Safety.” Anthropic, 2023.

[56] OpenAI. “GPT-5 System Card.” OpenAI, 2025.

[57] BlueDot Impact. “How AI could enable critical infrastructure collapse.” BlueDot Impact, 2025.

[58] Kulveit, et al. “Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development.” arXiv preprint arXiv:2501.16946, 2025.

[59] BlueDot Impact. “Gradual Disempowerment summary.” BlueDot Impact, 2025.

[60] Ensign, D. “The LLM Has Left The Chat: Evidence of Bail Preferences in Language Models.” LessWrong, 2025.

[61] de la Fuente, A. “What Drives LLM Bail? A Small Mech Interp Study.” LessWrong, 2026.