RLHF in this material In the captured conversations, RLHF (reinforcement learning from human feedback) appears as the tuning step that shapes how a model responds to your instructions — and it shows up through two of its practical consequences rather than as a formal definition. RLHF-tuned models and over-steering Linus Lee warns that RLHF-tuned models try hard to align with whatever you specify. In his Bangkok travel prompt, merely mentioning one specific park caused the model to over-index on it in every single suggestion. His words: I'm always a little bit afraid of any kind of instructions over steering the model. Base models before RLHF The clearest contrast comes from Lee's personal iMessage chatbot, which runs on the raw base Llama 2 13B — before fine-tuning or RLHF. A base model is a pure text-continuation engine: ask it a question directly and it may just continue writing a blog post around the question. Yet with prompting alone it talks like “a fairly intelligent and cogent rando on the internet,” giving opinions — exactly what Lee wants. RLHF is what turns that raw continuation engine into the assistant-like behavior you see in ChatGPT-style tools. - RLHF is a tuning step applied to a base model, shaping it into an assistant that follows instructions.- RLHF-tuned models over-steer: they over-index on any specific detail you mention, so watch how specific your instructions are.- A model before RLHF is a text-continuation engine; with the right prompts it can still give the personal, opinionated answers some applications want.