--- schema: 1 type: video title: 'Let LLMs Wander: Engineering RL Environments — Stefano Fiorucci' description: >- Reasoning models like DeepSeek R1 have demonstrated that learning from interaction is just as critical as learning from examples. To build these capabilities ourselves, we need to move beyond static datasets and start building Reinforcement Learning Environments: little worlds where models can act, get rewards, and learn. In this talk, I will walk you through my journey exploring this space from a practical software engineering perspective. We will cover: - How classic Reinforcement Learning concepts translate to Language Models - Verifiers, an open-source library to build Environments as software artifacts - Concrete examples of environments, from single-turn tasks to multi-turn games and tool-using agents - How to use these environments for both evaluating and training Small Language Models. Join me to learn how to move from prompting models to building the gyms where they learn. Stefano Fiorucci - AI/SW Engineer/Explorer, deepset Stefano is an AI/Software Engineer and explorer. He currently works on AI Orchestration at Deepset, where he contributes to and maintains Haystack, a widely used open-source framework for building LLM applications. He loves experimenting with Small Language Models, Post-Training and Reinforcement Learning, and shares his learning through code, writing, and talks. LLM RL Environments Lil Course: https://github.com/anakin87/llm-rl-environments-lil-course Socials: https://twitter.com/theanakin87 https://www.linkedin.com/in/stefano-fiorucci/ https://github.com/anakin87 https://huggingface.co/anakin87 Slides: https://drive.google.com/file/d/116PKThwtyTxeH1GmZQ7bL3HPYM6KCgHa/view?usp=drive_link --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/71V3fTaUp2Q-let-llms-wander-engineering-rl-environments-stefano-fiorucci Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/let-llms-wander-engineering-rl-environments-stefano-887b7751cc4c markdown: >- https://liquidgold.cc/channel/aidotengineer/video/let-llms-wander-engineering-rl-environments-stefano-887b7751cc4c.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T15:33:00.934Z' language: null children: [] videoId: 71V3fTaUp2Q durationSeconds: 2435 publishedAt: '2026-04-08T06:15:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 16 endSeconds: 149 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=16' heldReason: rights - ordinal: 1 text: null startSeconds: 159 endSeconds: 266 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=159' heldReason: rights - ordinal: 2 text: null startSeconds: 159 endSeconds: 266 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=159' heldReason: rights - ordinal: 3 text: null startSeconds: 278 endSeconds: 395 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=278' heldReason: rights - ordinal: 4 text: null startSeconds: 278 endSeconds: 395 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=278' heldReason: rights - ordinal: 5 text: null startSeconds: 407 endSeconds: 501 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=407' heldReason: rights - ordinal: 6 text: null startSeconds: 511 endSeconds: 591 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=511' heldReason: rights - ordinal: 7 text: null startSeconds: 602 endSeconds: 727 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=602' heldReason: rights - ordinal: 8 text: null startSeconds: 602 endSeconds: 727 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=602' heldReason: rights - ordinal: 9 text: null startSeconds: 737 endSeconds: 850 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=737' heldReason: rights - ordinal: 10 text: null startSeconds: 860 endSeconds: 961 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=860' heldReason: rights - ordinal: 11 text: null startSeconds: 972 endSeconds: 1086 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=972' heldReason: rights - ordinal: 12 text: null startSeconds: 1098 endSeconds: 1215 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=1098' heldReason: rights - ordinal: 13 text: null startSeconds: 1226 endSeconds: 1320 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=1226' heldReason: rights - ordinal: 14 text: null startSeconds: 1332 endSeconds: 1439 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=1332' heldReason: rights - ordinal: 15 text: null startSeconds: 1450 endSeconds: 1530 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=1450' heldReason: rights - ordinal: 16 text: null startSeconds: 1542 endSeconds: 1650 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=1542' heldReason: rights - ordinal: 17 text: null startSeconds: 1661 endSeconds: 1774 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=1661' heldReason: rights - ordinal: 18 text: null startSeconds: 1786 endSeconds: 1899 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=1786' heldReason: rights - ordinal: 19 text: null startSeconds: 1910 endSeconds: 1990 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=1910' heldReason: rights - ordinal: 20 text: null startSeconds: 2000 endSeconds: 2130 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=2000' heldReason: rights - ordinal: 21 text: null startSeconds: 2142 endSeconds: 2265 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=2142' heldReason: rights - ordinal: 22 text: null startSeconds: 2142 endSeconds: 2265 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=2142' heldReason: rights - ordinal: 23 text: null startSeconds: 2275 endSeconds: 2365 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=2275' heldReason: rights - ordinal: 24 text: null startSeconds: 2365 endSeconds: 2377 watch: 'https://liquidgold.cc/watch/71V3fTaUp2Q?t=2365' heldReason: rights --- # Let LLMs Wander: Engineering RL Environments — Stefano Fiorucci Reasoning models like DeepSeek R1 have demonstrated that learning from interaction is just as critical as learning from examples. To build these capabilities ourselves, we need to move beyond static datasets and start building Reinforcement Learning Environments: little worlds where models can act, get rewards, and learn. In this talk, I will walk you through my journey exploring this space from a practical software engineering perspective. We will cover: - How classic Reinforcement Learning concepts translate to Language Models - Verifiers, an open-source library to build Environments as software artifacts - Concrete examples of environments, from single-turn tasks to multi-turn games and tool-using agents - How to use these environments for both evaluating and training Small Language Models. Join me to learn how to move from prompting models to building the gyms where they learn. Stefano Fiorucci - AI/SW Engineer/Explorer, deepset Stefano is an AI/Software Engineer and explorer. He currently works on AI Orchestration at Deepset, where he contributes to and maintains Haystack, a widely used open-source framework for building LLM applications. He loves experimenting with Small Language Models, Post-Training and Reinforcement Learning, and shares his learning through code, writing, and talks. LLM RL Environments Lil Course: https://github.com/anakin87/llm-rl-environments-lil-course Socials: https://twitter.com/theanakin87 https://www.linkedin.com/in/stefano-fiorucci/ https://github.com/anakin87 https://huggingface.co/anakin87 Slides: https://drive.google.com/file/d/116PKThwtyTxeH1GmZQ7bL3HPYM6KCgHa/view?usp=drive_link --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/71V3fTaUp2Q-let-llms-wander-engineering-rl-environments-stefano-fiorucci Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/let-llms-wander-engineering-rl-environments-stefano-887b7751cc4c · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:06 to 0:16) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (0:16 to 2:29) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (2:29 to 4:26) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (4:26 to 6:35) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (6:35 to 8:21) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (8:21 to 9:51) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (9:51 to 12:07) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 7: Document excerpt (12:07 to 14:10) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 8: Document excerpt (14:10 to 16:01) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 9: Document excerpt (16:01 to 18:06) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 10: Document excerpt (18:06 to 20:15) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 11: Document excerpt (20:15 to 22:00) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 12: Document excerpt (22:00 to 23:59) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 13: Document excerpt (23:59 to 25:30) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 14: Document excerpt (25:30 to 27:30) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 15: Document excerpt (27:30 to 29:34) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 16: Document excerpt (29:34 to 31:39) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 17: Document excerpt (31:39 to 33:10) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 18: Document excerpt (33:10 to 35:30) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 19: Document excerpt (35:30 to 35:42) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 20: Document excerpt (35:42 to 37:45) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 21: Document excerpt (37:45 to 39:25) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 22: Document excerpt (39:25 to 40:35) > Held: transcript rights do not permit this chunk's text.