Paper digest: how AI learns what people prefer
Chatbots are trained to give answers people prefer, using methods called RLHF and DPO. How they work, what three key papers found and what 'aligned' means.
Chatbots like ChatGPT aren't just trained to predict text from the internet. They are also trained on which answers people prefer, a step usually called alignment. In 2022, OpenAI showed that a small model trained this way was preferred over a version more than 100 times larger.
When an AI vendor says its model is "aligned", this is mostly what they mean. Knowing how it works helps you ask what the model was actually trained to please.
The papers
| Paper | Authors | Year | Link |
|---|---|---|---|
| Deep Reinforcement Learning from Human Preferences | Christiano, Leike, Brown, Martic, Legg and Amodei (OpenAI and DeepMind) | 2017 | arXiv:1706.03741 |
| Training language models to follow instructions with human feedback (InstructGPT) | Ouyang and others (OpenAI) | 2022 | arXiv:2203.02155 |
| Direct Preference Optimization | Rafailov, Sharma, Mitchell, Ermon, Manning and Finn (Stanford) | 2023 | arXiv:2305.18290 |
Why this research exists
A language model trained only on internet text learns to continue text, not to help. It might ramble, make things up or produce harmful content. Showing it ideal example answers helps, but writing those is slow and expensive.
It is much quicker for a person to look at two answers and say which is better. These papers turn those simple comparisons into training.
How it works
Step 1: learning from comparisons
Paul Christiano and colleagues tested the idea on video games and simulated robots in 2017. People watched pairs of short clips and picked the better one. A separate model learned to predict those choices and turned them into scores the AI could learn from.
It worked with feedback on less than 1 percent of the AI's actions. Some new behaviours took only about an hour of human time to teach.
Step 2: RLHF for chatbots
OpenAI's InstructGPT paper applied the same idea to language. The process, known as reinforcement learning from human feedback (RLHF), has three stages:
- Train the model on example answers written by people.
- Have people rank several of the model's answers, and train a "reward model" to predict those rankings.
- Let the chatbot practise, scoring its answers with the reward model, while keeping it from drifting too far from its starting point.
That last safeguard stops the chatbot from finding odd tricks that fool the reward model without actually being helpful.
Step 3: DPO, a simpler route
RLHF works but is complex and can be unstable. Rafael Rafailov and colleagues at Stanford showed the reward model step can be skipped. Their method, Direct Preference Optimization (DPO), trains directly on pairs of preferred and rejected answers, with a simple formula.
The result is one training run, no separate reward model and far less tuning. That made preference training practical for smaller teams.
Results that moved
From the paper abstracts:
| Paper | Key finding |
|---|---|
| Christiano 2017 | Complex behaviours learned with feedback on under 1% of actions |
| InstructGPT 2022 | People preferred answers from a 1.3 billion parameter model over the 175 billion parameter GPT-3 |
| DPO 2023 | Matched or beat RLHF on summaries and single-turn dialogue, and was better at controlling the tone of text |
InstructGPT also became more truthful and less toxic, with little loss on standard tests. The authors note it still made simple mistakes.
Limits
- Preferred is not the same as true. People can prefer a confident, fluent answer that is wrong.
- Models can learn to flatter. If longer or more agreeable answers win comparisons, the model learns to give them.
- The rankers shape the result. If most rankers are in the United States, the model learns their idea of a good answer. Caribbean phrasing or tone may be marked down.
- Helpfulness and safety pull against each other. Rewarding "always answer" produces a model that complies with bad requests, while rewarding caution produces one that refuses too much.
- Preferences from general chat don't transfer neatly to specialist work like legal, medical or customer service tasks.
Who should care
If you buy AI services, ask the vendor how the model was preference-trained and whose preferences were used. Ask for results on accuracy and harmful content, not just "win rates" against other models.
If you build AI tools, DPO is often the first thing to try after basic fine-tuning. A few thousand good comparisons from your own field can make a big difference.
For Caribbean businesses, your customer service tone and how your brand speaks are preference questions. A model tuned on local comparisons can sound more natural to your customers than a general chatbot. Make sure the input is right first, though. Tuning on badly transcribed voice notes, covered in our speech digest, will only teach the model the wrong things.
Bottom line
RLHF and DPO teach AI models to give answers people prefer, and DPO made that much simpler. When a vendor says "aligned", ask which method they used, whose preferences they collected and how they test for accuracy.