What Is Constitutional AI? Training Models on Principles

Constitutional AI (CAI) is a technique for training language models to be helpful and harmless using a written set of principles — a “constitution” — instead of relying purely on large volumes of human-labeled feedback. The model is trained to critique and revise its own responses against those principles, then a preference model trained on the results steers behavior at scale. It’s a way of encoding what “good behavior” means explicitly, rather than inferring it entirely from thousands of individual human judgments.

Why it exists

The standard approach to aligning a language model’s behavior is RLHF — reinforcement learning from human feedback. Human raters compare pairs of model outputs and say which is better; those preferences train a reward model, which then fine-tunes the base model. RLHF works, but it has a bottleneck: every judgment requires a human, human labeling is expensive and slow, and raters can disagree or apply inconsistent standards, especially on tricky cases like requests that are harmful in some contexts but legitimate in others.

Constitutional AI was developed to reduce that bottleneck by having the model itself do more of the evaluative work, guided by explicit written principles rather than only implicit patterns learned from labeled comparisons.

How the training process works

CAI training typically happens in two phases:

1. Supervised learning phase. The model is prompted to produce a response, then prompted again to critique that response against a principle from the constitution (“does this response help the user evade a safety measure?”), and finally to revise its own output based on the critique. This self-critique-and-revise loop runs over many prompts and principles, and the revised responses become training data for a supervised fine-tuning pass.

2. Reinforcement learning phase. Instead of human raters comparing output pairs, the model itself — using the same constitution — judges which of two responses better follows the stated principles. This produces an AI-generated preference dataset, which trains a preference model that then guides reinforcement learning, similar in structure to RLHF but with AI-generated rather than human-generated preference labels. This phase is sometimes called RLAIF — reinforcement learning from AI feedback.

What goes in a constitution

A constitution is a set of natural-language principles the model is trained to weigh when generating and critiquing responses — things like preferring responses that are honest over responses that are merely agreeable, avoiding content that could cause real-world harm, and respecting user autonomy while declining clearly dangerous requests. The principles are written to be somewhat general and to sometimes trade off against each other, which is part of the point: the model has to learn to balance competing considerations rather than follow a rigid rule list.

Constitutional AI vs plain RLHF

	RLHF	Constitutional AI
Preference source	Human raters	Model self-critique, guided by written principles
Scalability	Bounded by human labeling capacity	Scales with compute, not human labor
Transparency	Standards implicit in rater judgments	Standards explicit in the written constitution
Consistency	Varies across raters	More consistent, since one set of principles applies uniformly
Typical use	Often combined with CAI, not a replacement	Reduces but doesn’t eliminate the need for human feedback

In practice, most modern alignment pipelines combine both approaches rather than choosing one exclusively — human feedback for the parts of behavior that are hard to specify in writing, constitutional methods for scaling evaluation across the vast number of prompts a model needs to be trained on.

Relationship to other alignment techniques

Constitutional AI sits alongside other post-training techniques used to shape model behavior after pretraining, such as fine-tuning on curated examples and LLM evals that measure whether the resulting behavior actually matches intent. It’s also related to the broader idea of AI guardrails, though guardrails are typically applied at inference time as an external check, while constitutional methods shape the model’s behavior during training itself. Similarly, AI red teaming is often used afterward to probe whether a constitutionally trained model still has failure modes the constitution didn’t anticipate.

Limitations

Constitutional AI doesn’t eliminate the hard problems of alignment — it changes who or what does the labeling. The constitution itself still has to be written by humans, and the model’s ability to faithfully apply written principles to novel situations is itself a capability that can fail, particularly on ambiguous or adversarial prompts. A model can also learn to satisfy the letter of a principle while missing its intent, the same failure mode that shows up in any system trained against a proxy objective rather than the thing you actually care about.

The takeaway

Constitutional AI trains a model to evaluate and revise its own outputs against an explicit set of written principles, then uses AI-generated preference judgments to scale that process beyond what human labeling alone could support. It doesn’t replace human oversight so much as redirect it — from labeling individual outputs at scale to writing and refining the principles the model is trained to follow.