--- schema: 1 type: video title: 'From RL to IRL — Gaurav Mishra, Amazon AGI Lab' description: >- Asked to file an expense, the agent gets signed out mid task, reasons that it can infer the password, guesses twice, and locks the account. In a second run it clicks a sponsored button styled like the real submit button, lands on a different site, and begins typing personal details into it. Both are real trajectories from early browser training runs at the Amazon AGI Lab, and Gaurav Mishra's summary is that RL worked while the world was a game, and IRL starts when the game fights back. The talk catalogues what a reward function meets on contact with a real login screen. Observability is partial, since the DOM misses content baked into images and the screenshot misses whatever needs scrolling. Actions are irreversible, credentials expire mid trajectory, and done routinely does not mean successful. His answer is flight school rather than exams. Sandboxes train on layout shift, slow loads, pop ups, focus stealing, and stale tabs, and recovery becomes a native model action instead of an infra reset, so the agent refreshes, backtracks, waits, or escalates. A process reward model penalizes dangerous steps along the path instead of scoring only the outcome, and calibrated confidence teaches the agent to weigh whether an action is authorized, reversible, and visible before committing. The closing trajectory runs the same task correctly, including the agent refusing to guess the password and handing control back. Over time the model gets better and the harness gets thinner. Speaker info: - https://www.linkedin.com/in/gaurav-mishra-b307a437 Timestamps: 0:00 - RL to IRL, and a lightning review of RL for agents 3:26 - Why coding agents can do computer use at all 4:05 - The agent that guesses its own password 5:47 - The sponsored button that looks like submit 6:37 - Partial observability, irreversibility, expiring credentials 8:29 - Flight school, not exams 9:54 - Process rewards and calibrated confidence 11:11 - The pilot and the cockpit 14:07 - Assumption versus reality, point by point 15:11 - The same task, done right canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/from-rl-to-irl-gaurav-mishra-amazon-agi-lab-0f66b79ff68a markdown: >- https://liquidgold.cc/channel/aidotengineer/video/from-rl-to-irl-gaurav-mishra-amazon-agi-lab-0f66b79ff68a.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T05:24:14.146Z' language: null children: [] videoId: Cc0_nyxROBA durationSeconds: 1066 publishedAt: '2026-08-14T16:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 23 endSeconds: 217 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=23' heldReason: rights - ordinal: 1 text: null startSeconds: 23 endSeconds: 217 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=23' heldReason: rights - ordinal: 2 text: null startSeconds: 23 endSeconds: 217 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=23' heldReason: rights - ordinal: 3 text: null startSeconds: 217 endSeconds: 227 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=217' heldReason: rights - ordinal: 4 text: null startSeconds: 262 endSeconds: 349 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=262' heldReason: rights - ordinal: 5 text: null startSeconds: 349 endSeconds: 360 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=349' heldReason: rights - ordinal: 6 text: null startSeconds: 360 endSeconds: 404 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=360' heldReason: rights - ordinal: 7 text: null startSeconds: 404 endSeconds: 416 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=404' heldReason: rights - ordinal: 8 text: null startSeconds: 416 endSeconds: 520 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=416' heldReason: rights - ordinal: 9 text: null startSeconds: 416 endSeconds: 520 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=416' heldReason: rights - ordinal: 10 text: null startSeconds: 520 endSeconds: 531 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=520' heldReason: rights - ordinal: 11 text: null startSeconds: 531 endSeconds: 599 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=531' heldReason: rights - ordinal: 12 text: null startSeconds: 599 endSeconds: 609 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=599' heldReason: rights - ordinal: 13 text: null startSeconds: 609 endSeconds: 674 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=609' heldReason: rights - ordinal: 14 text: null startSeconds: 674 endSeconds: 685 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=674' heldReason: rights - ordinal: 15 text: null startSeconds: 685 endSeconds: 851 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=685' heldReason: rights - ordinal: 16 text: null startSeconds: 685 endSeconds: 851 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=685' heldReason: rights - ordinal: 17 text: null startSeconds: 851 endSeconds: 863 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=851' heldReason: rights - ordinal: 18 text: null startSeconds: 919 endSeconds: 932 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=919' heldReason: rights - ordinal: 19 text: null startSeconds: 932 endSeconds: 1066 watch: 'https://liquidgold.cc/watch/Cc0_nyxROBA?t=932' heldReason: rights --- # From RL to IRL — Gaurav Mishra, Amazon AGI Lab Asked to file an expense, the agent gets signed out mid task, reasons that it can infer the password, guesses twice, and locks the account. In a second run it clicks a sponsored button styled like the real submit button, lands on a different site, and begins typing personal details into it. Both are real trajectories from early browser training runs at the Amazon AGI Lab, and Gaurav Mishra's summary is that RL worked while the world was a game, and IRL starts when the game fights back. The talk catalogues what a reward function meets on contact with a real login screen. Observability is partial, since the DOM misses content baked into images and the screenshot misses whatever needs scrolling. Actions are irreversible, credentials expire mid trajectory, and done routinely does not mean successful. His answer is flight school rather than exams. Sandboxes train on layout shift, slow loads, pop ups, focus stealing, and stale tabs, and recovery becomes a native model action instead of an infra reset, so the agent refreshes, backtracks, waits, or escalates. A process reward model penalizes dangerous steps along the path instead of scoring only the outcome, and calibrated confidence teaches the agent to weigh whether an action is authorized, reversible, and visible before committing. The closing trajectory runs the same task correctly, including the agent refusing to guess the password and handing control back. Over time the model gets better and the harness gets thinner. Speaker info: - https://www.linkedin.com/in/gaurav-mishra-b307a437 Timestamps: 0:00 - RL to IRL, and a lightning review of RL for agents 3:26 - Why coding agents can do computer use at all 4:05 - The agent that guesses its own password 5:47 - The sponsored button that looks like submit 6:37 - Partial observability, irreversibility, expiring credentials 8:29 - Flight school, not exams 9:54 - Process rewards and calibrated confidence 11:11 - The pilot and the cockpit 14:07 - Assumption versus reality, point by point 15:11 - The same task, done right Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/from-rl-to-irl-gaurav-mishra-amazon-agi-lab-0f66b79ff68a · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.