A workshop by Angelos Perivolaropoulos (ElevenLabs) on training a small LLM from scratch on a local machine. Inspired by Andrej Karpathy's NanoGPT, it covers the four fundamental building blocks of an LLM, setup and prerequisites (Python, UV, hardware requirements), character-level tokenization, model architecture (GPT class structure, transformer blocks, vocab size, embeddings), parameter count and model sizing, the training loop with next-token prediction, optimization strategies (warm-up and cosine decay), validation and loss monitoring, and text generation and inference. It concludes with project file structure, debugging common training issues, a workshop challenge, and Q&A on base vs. reasoning models and applying the concepts to audio and multimodal models. # Channel Presentation Observations — Video: "Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs" ## Video Format & Context - **Workshop format with live audience**: The speaker opens with "Thank you very much for joining for this workshop. Uh this is going to be a hands-on workshop," and later invites audience interaction ("Sorry, go ahead," repeated question handoffs to a microphone). - **In-person event with a shared repository**: Attendees scan a QR code to access a GitHub repo; the speaker offers to "go around and help people out." - **Prize element**: A competition is run — "the winner is going to get some uh free swag from from Level Ups, maybe like a hoodie or like some free credits." ## Speaker Self-Description (Explicit Claims) - **Name and role**: "my name is Angelos. I lead the speech-to-text team at Eleven Labs. I'm a research engineer." - **Responsibilities claimed**: "I spend the most of my time training new models, working on inference, and also I'm on the product side." - **Current work claimed**: "Currently, I'm working on training real-time models for transcription. Uh specifically for agents." - **Product claim**: "we have this scribe V2 model that my team trained. It's currently the best transcription model in the market in terms of like popular public benchmarks." — Note: this is a creator claim about a product, not verifiable from the video. ## Intended Audience - **Aspiring/beginner ML practitioners with basic Python**: "I expect that most of you have like some idea of how to write Python. Uh like if you don't nothing is still quite you can just copy paste things until something works." - **People with modest hardware**: "any laptop will do uh that has at least six 16 GB of RAM"; Google Colab offered as fallback with free GPUs. - **Mixed experience levels**: the speaker acknowledges experienced attendees may find it simple — "this might be like seem a bit simple for you" — and offers extension challenges for them. - **Curious generalists**: extended audience Q&A covers reasoning models, multimodal models, audio tokenization, music generation — treated at an explanatory, accessible level. ## Claimed Scope of Content - **Train an LLM from scratch, no pre-trained weights**: "no pre-trained weights, no nothing that you can just grab online from like a Transformers library. We're going to work purely on torch." - **Complete pipeline covered**: four building blocks explicitly enumerated — tokenizer, model architecture, training loop, inference. - **Concrete deliverable**: a character-level GPT-2-style model (~1.8M parameters) trained on a Shakespeare dataset, ~15 minutes on Colab, resulting in model.py, train.py, generate.py. ## Demonstrated Presentation Patterns - **Educational, first-principles tone**: explains basics like what a tokenizer is ("LLMs don't see text") and loss-curve interpretation. - **Anecdotes from industry experience**: e.g., "we're going to spend maybe 6 months thinking about the tokenizer, and then we're going to spend 2 months on the architecture"; references Scale AI labeling pipelines and QA labeler hierarchies: "a big part of my job is actually like doing this organization." - **Cited inspiration**: Andrej Karpathy's nanoGPT is named as the project's inspiration. - **Honest hedging on detail**: "I don't remember exactly what other models do"; "I'm not sure" on cross-modal embedding similarity — claims are flagged as uncertain rather than asserted. - **Candid persona**: "I'm responsible for unfortunately talking to clients sometimes, which I don't fully enjoy." ## Observable Evidence of Affiliation - The **title attributes the speaker to ElevenLabs**, and the transcript confirms: "I lead the speech-to-text team at Eleven Labs." No other channel-level branding (e.g., channel name, series identity) is present in the supplied material. ## Not Present / Cannot Infer - No channel-wide self-description (e.g., "this channel is about X"), no upload cadence, no subscriber/community references, and no curated metadata were supplied. - No private motivations inferred; all observations above trace to quoted transcript passages or the title. # Delivery Observations: Angelos Perivolaropoulos — "Training an LLM from Scratch, Locally" ## Overall Tone & Register - **Casual, conversational workshop tone** throughout: filled pauses ("Uh", "Um") appear constantly, plus hedges like "I think," "maybe," "kind of," "quite." - Friendly and self-deprecating humor, e.g.: > "We could go a level below and not use torch, but I don't want to to torture you that much." > "I'm responsible for unfortunately talking to clients sometimes, which I don't fully enjoy." ## Vocabulary & Technical Framing - Uses **conversational analogies and informal glosses** for technical terms: "it will pass out" (model exceeding context length), "makes it go crazy" (high learning rate), "gibbles" (gibberish tokens). - Recurring vocabulary clusters: **"building blocks"**, **"from scratch"**, **"big labs"**, **"trade-off"**, **"optimize/improvement"**, **"converge"**, **"post-train"**. - Repeated market references to ground concepts: GPT-2/3/4/5, Gemini 3/3.1, ChatGPT, Scale AI, Andrej Karpathy's nanoGPT. ## Recurring Phrases & Verbal Habits - **"As I mentioned before" / "As I said before"** — used many times to re-anchor earlier points (e.g., "As I mentioned before like this is generally the first thing you think about when you create a new... transformer model"). - **"I don't want to go too deep" / "not going to go too deep into the details"** — a repeated disclaimer limiting depth: > "I'm I'm not going to go too deep on how transformers work." > "I I'm not going to go again to to too deep into the details." - **"Feel free to..."** as a recurring affordance phrase: "feel free to use it," "feel free to just copy-paste these these blocks," "feel free to train." - **"Like"** as an extremely frequent filler (dozens of instances per minute in dense passages). - **Word/stutter repetition** under load: "the the the," "we we we," "is is is" — noticeably concentrated when explaining complex material (e.g., the layer-norm and forward-pass sections). ## Teaching Habits & Structure - **Explicit enumerated structure**: announces "The four building blocks you need to to train a model" (tokenizer, architecture, training loop, inference) and walks through them in order. - **"Big picture first, details deferred"**: repeatedly gives high-level explanations, then tells attendees details are available later — "you can you have to like copy paste the the the parts or figure them out yourself." - **Concrete numeric anchors** to make abstractions tangible: 65 tokens, 4,225 bigrams, 384 embedding size, loss milestones ("it will start at around 4.17... slowly we're going to start seeing the loss going down to 3.3"). - **Loss-value-to-behavior mapping** as a teaching device: connecting loss numbers to observable model behavior ("at around 1.0 to 1.2, that's when the model is going to start being decent"). - **Honest admissions of uncertainty** rather than bluffing: > "I don't remember exactly what other models do the exact numbers." > "I I don't know. I imagine there probably is some uh connection." > "I don't I don't really remember if it did or not." - **Diagnostic heuristics delivered as rules of thumb**: > "if your train loss is not decreasing, it means your model is not learning. Uh the problem is you have a bug in your code." ## Audience Interaction Patterns - **Welcomes interruptions**: "Yeah, yeah, no, you can interrupt." - Handles a multi-round clarification patiently (the variable-names-vs-tokenizer exchange), restating the question before answering: "I I see that's what you mean." - Frequently **checks comprehension/closure**: "I hope that was a decent explanation," "I hope this answers your question," "I hope this answered your question." - Closes with hands-on facilitation: "I'll go around and like help people out." ## Communication Patterns - **Long multi-clause run-on sentences** with nested self-corrections, typical of live technical improvisation (e.g., the layer-norm explanation). - Minor live-delivery slips acknowledged with brief apologies: "Oh, oops. Yeah, I was showing... I apologize for that," "oops. This is what I was saying. Sorry. I have two different screens." - Encouraging, low-pressure framing for beginners: > "you can just copy paste things until something works or like ask for some help." > "Copy pasting will probably like allow you to go get 90% there." ## Uncertainty Notes - Filler frequency and stutter-repetitions are strongly observable across the transcript; exact per-minute rates cannot be quantified without timestamps beyond the bracketed markers. - Patterns like "I hope this answers your question" appear at least three times; whether this reflects a habitual closer or only Q&A-section usage is uncertain given the single-session sample. # Position Record — "Training an LLM from Scratch, Locally" **Speaker:** Angelos Perivolaropoulos, Research Engineer / Speech-to-Text Team Lead, ElevenLabs (per title and self-introduction) ## Workshop Framing & Scope - The workshop trains an LLM **from scratch** — no pre-trained weights, no Hugging Face Transformers — using PyTorch and basic libraries only. - Angelos states this covers "pretty much 80% of the way there" to creating a model from scratch; beyond that is "optimizations and making the scale be better." - Inspiration: Andrej Karpathy's **nanoGPT** ("that was like a very special project... it's a good blueprint"). ## Claims About Model Training ### Four Building Blocks 1. **Tokenizer** — use case dependent; large multilingual models need large tokenizers and much more data. 2. **Architecture** — most models are "very similar" causal decoder-only transformers with self-attention, MLP, layer norms. 3. **Training loop** — "the most important part." 4. **Inference** — trivial for small models. ### Key Positions - **Training/post-training drives performance more than architecture:** Citing GPT-4/4o/5 and Gemini 3 → 3.1, he claims models are "very similar" but "the way you train it actually makes a big difference" — e.g., Gemini 3.1 showing "double the performance in some benchmarks." - **Transformers are commoditized:** differences across labs are "more optimizations than necessarily reinventing the wheel"; the base GPT-2 architecture is fundamentally unchanged. - **You don't need deep theory to train models:** "When I first did this project myself I had no clue how transformers worked" — understanding comes with practice. ## Tokenizer Positions - **Character-level tokenization** chosen for the workshop: 65 tokens → ~4,225 bigrams, learnable with ~1M-character Shakespeare dataset. With a full tokenizer, "this will never converge." - **Rule of thumb (tentative):** for a 200k-token vocabulary, you need roughly vocab² tokens of data "at least... this is the magnitude." - **Character-level doesn't scale:** tokens like "S K Y" are harder for the model to attend to meaningfully; also embedding cost — GPT-2's 50k vocab at 384 dims = 19M parameters, "more than three times the model." - **BPE explanation** (in Q&A): common patterns in training data become tokens (e.g., `for`, `enumerate`); rare variable names fall back to character/byte-level tokenization — "a bit of a pain for inference." ## Architecture Positions - **Building blocks:** multi-head self-attention (token relationships), MLP/FFN (combines relationships toward logits), residual connections (stability, each layer changes input only slightly), layer norm (prevents exploding activations). - **Attention heads specialize:** e.g., one may track punctuation, another grammar. - **On which parameters to increase** (Q&A, stated tentatively): increase all except perhaps embedding size; "I don't remember exactly what other models do... I think that seems reasonable." - **Long context requires architectural change:** "You can't just change this number" — scaling 256 to 1M context needs architecture changes or training goes out of memory. - **Each transformer block has its own weights** (MLP, attention, layer norms). - Does **not remember** whether Karpathy's nanoGPT used residual connections: "It's been a while... I built everything from scratch." ## Training Loop Positions - **Learning rate:** warm-up (100 steps) → cosine decay to 5,000 steps; AdamW most common/starting optimizer. - **Loss milestones** on Shakespeare data: start ~4.17 (ln 65, knows nothing) → 3.3 (character frequencies) → ~2.5 (words like "the") → 1.5–2 (words form) → 1.0–1.2 (decent) → below 1.0 **signals overfitting**. - **Val loss** guards against overfitting; optimal checkpoint observed at ~2,400 steps; beyond that, models become "less creative." - **Diagnostics:** train loss not decreasing = bug; val loss rising = overfit; loss spikes = bug; plateau = need more data or bigger model. - Caveat: val loss "is not the best metric" for serious LLM training — benchmarks are preferred. ## Inference Positions - **Greedy decoding:** "pretty much never" for LLMs (boring, uncreative) but **best for transcription** — "you don't want it to be creative in transcription." - **Temperature ~0.7** is "the best middle ground"; randomness empirically improves performance. - **Top-K sampling** blocks very unlikely tokens. - **Seeds** make generation reproducible; he recommends seeding even with greedy decoding since implementations may secretly use small nonzero temperature. ## Q&A Positions (Audience-Driven) - **Reasoning models:** same base architecture; reasoning comes via post-training with high-quality chain-of-thought data. Claims OpenAI uses PhD-level labelers writing reasonings; "you can't just go on Reddit." Reasoning tokens are attended to at generation time. People have made Llama 1B-class non-reasoning models into reasoning models with the same architecture. - **Fundamental innovation?** Base attention largely the same; labs use efficiency tricks for long reasoning sequences. "You can take GPT-2 and make it into a reasoning model" given data and scale. - **Golden datasets:** labs use vendors like Scale AI ("an army of people") plus in-house labeling teams; datasets are expensive and "make or break" models. QA hierarchies of labelers maintain quality — human review, not just LLM evaluation. - **Audio vs. text:** "surprisingly very similar" fundamentals; audio tokens trained on patterns in mel spectrograms; specialized losses (e.g., L2 on mel spectrograms for TTS) instead of cross-entropy; KL divergence for distillation. Mixing voice + music tokenization is "very hard." - **Multimodal models:** video/audio encoders produce embedding vectors that override token embeddings at the model input — the transformer "just cares about these vectors." On whether video and text embeddings align for "horse": "Maybe there is, maybe there's not... that would be a good research paper." *(Stated uncertainty preserved.)* - **Music generation:** both autoregressive and diffusion approaches work; diffusion is "generally a bit easier to get working" for music/images. ## Workshop Logistics (as stated) - Runs on MPS, CUDA, or CPU; 16 GB RAM sufficient; Google Colab T4 GPU recommended; his run took ~15 minutes. - Competition: train today, submit seeded reproducible generations; winner judged by vote; swag prize. Model tweaks allowed (dropout, context length, custom BPE tokenizer, larger models). # Speaker Attribution Record — "Training an LLM from Scratch, Locally" ## Video Overview - **Title:** Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs - **Format:** Hands-on workshop (recorded live, with audience interaction) ## Speakers & Roles ### Angelos Perivolaropoulos — **Primary Speaker / Workshop Leader** Self-identified: leads the **speech-to-text team at ElevenLabs**, research engineer. Roles cited: model training, inference, product/client work, training real-time transcription models for agents. > "My name is Angelos. I lead the speech-to-text team at Eleven Labs. I'm a research engineer." ### Audience Members — **Unidentified Guests / Q&A Participants** Multiple unnamed attendees ask questions on tokenization of code, model parameters, reasoning models, golden datasets, seeds/determinism, audio vs. text training, multimodal encoders, music generation, and audio tokenization. One exchange involves an off-stage or production voice: > "Are you showing me some slides that we haven't seen yet?" ## Promotional Segments — Flagged - **ElevenLabs Scribe V2** promotion: > "We have this scribe V2 model that my team trained. It's currently the best transcription model in the market... feel free to use it." - Note: "best in the market" is the speaker's claim, not independently verified. - **Competition swag** promotion: winner receives "free swag from Level Ups, maybe like a hoodie or some free credits" — sponsor/organizer-provided prize. ## Referenced Sources & Third-Party Claims - **Andrej Karpathy / nanoGPT** — cited as inspiration; Karpathy identified as "one of the co-founders of OpenAI." Speaker states he does not remember whether nanoGPT included residual connections. - **GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-5 (OpenAI)** — referenced architectures; claim that OpenAI initially withheld GPT-2 over safety concerns. - **Gemini 3 / 3.1 (Google)** — speaker's claims about benchmark improvements and long-context research. - **Scale AI** — cited as data-labeling provider for labs including Anthropic (per speaker). - **OpenAI PhD labelers** — speaker's claim about reasoning data quality practices. - **Llama 1B** — cited example of older models converted to reasoning models. - **Third-party tools/libraries:** torch, numpy, tqdm, tiktoken, UV, Google Colab, matplotlib, Apple MPS/CUDA. ## Technical Provenance - **Dataset:** Shakespeare text (~1 million characters), included in the workshop GitHub repo (accessed via QR code). - **Model:** GPT-2-based architecture, ~10M parameter class (speaker states 1.8M at one point), character-level tokenizer (65 tokens), block size 256. - **Training environment:** local laptop (≥16 GB RAM), Google Colab free T4 GPU, or CPU; Python 3.12. ## Availability & Uncertainty - No curated metadata supplied; no comment availability information. - Speaker explicitly unsure on several points (nanoGPT residuals, an MLP parameter value, video/text encoder embedding similarity) — uncertainty is his own, preserved here without resolution. - Audience speakers are unnamed and unattributable to specific individuals or organizations. Thank you very much for joining for this workshop. Uh this is going to be a hands-on workshop. So, hopefully you can get your hands dirty doing some very interesting very interesting project. Uh to start with a little bit about myself. Uh my name is Angelos. I lead the speech-to-text team at Eleven Labs. I'm a research engineer. So, I spend the most of my time training new models, working on inference, and also I'm on the product side. So, I'm responsible for unfortunately talking to clients sometimes, which I don't fully enjoy. I'm the kind of person that really likes to go straight into research and do train models and make things that are set of the art and very powerful. Uh Currently, I'm working on training real-time models for transcription. Uh specifically for agents. And if you have if you don't know already, we have this scribe V2 model that my team trained. It's currently the best transcription model in the market in terms of like popular public benchmarks. Uh so, if you have any any transcription uh use cases, feel free to use it. I think it's quite good. Uh now as part of the workshop itself, uh today we're going to be training an LLM from scratch. So, no pre-trained weights, no uh nothing that you can just grab online from like a Transformers library. We're going to work purely on torch and some like very basic libraries. We could go a level below and not use torch, but I don't want to to torture you that much. Uh so, I think torch is like a good level and this will be like a good indication of how like actual like research engineers like in in big labs like design their models and whatever is further than that is more like optimizations and making the scale be better and bigger and improving for specific use cases. So, this what you're going to be doing today is like pretty much like 80% of of the way there to to create like a model from scratch. Um if you can go on this uh QR code, uh you're going to find this uh GitHub repo. Uh I'm going to switch to that now. I'll leave it for a little bit longer. Uh there's you have two options. Uh one is to either train the model locally in your laptop. Like if you have like a 16 GB of memory, you should be able to do it. It's a small, very tiny model that you can train fast. Uh if not, and I'm guessing because not many people have outlets, maybe a Google Colab would be a another option. Uh Google Colab gives you free GPUs that you can use for for training such a small model. So, either it will be your choice. Um well, I'm going to leave this on for a little bit longer. And then we can go in the actual repo itself. Um so, to give you some idea of the inspiration of this um uh of this project, uh my first exposure to to transformers and LLMs in general was through this video from from Andrej Karpathy, who was one of the co-founders of of OpenAI. Uh which was this code nanoGPT, and for me that was like a very special project. This is essentially like what inspires like this this workshop as well. Uh it's a bit lower level than what we're going to be doing now. So, it goes a bit deeper like into how you're using NumPy for like a lot of calculations. But I think it's a good uh blueprint that we can follow for for us to be able to create a model from scratch. Um let me put this to the screen. But yeah, if you if you don't if you if you if you don't know this project, I think it's that's it's a great introduction. If you do know this project and you've trained the elements like that before, this might be like seem a bit simple for you, but I think that there are ways we can expand this workshop to towards doing something that's a little more of a competition, which I would see I would like to see what you can come up with. For the specific For the specific workshop, we're going to work with a very small model that's based on the GP2 GPT-2 architecture. It's a bit of an older architecture, but the the the base the the fundamental parts of it are basically haven't changed too much. And we're going to go over over those in in a bit. The four building blocks you need to to train a model is First one is going to be the tokenizer. Uh And depending on what your use case is, you'd want to use a different tokenizer for that specific use case. If you want to train, for example, a very big model that can that wants you want to generate text for like multiple languages, you'll need a huge tokenizer, which means you're going to need a huge amount of data as well to train it. But for a for a smaller model, a smaller tokenizer with with with smaller embeddings is what would work best and train the fastest when your when your data limited, which is what we are right now. Next will be a model architecture. To be honest, like at most models like have at least at at the at the period that we're going to be working in were very similar. Like in in they were just decoder only causal decoder only models that had a very similar uh way of of using causal self-attention and the same like MLP layers and the same like layer norms and all kind of stuff, which we'll we'll get we'll get to it. Um So, if you know how to do this this like small models, it's very easy to go and do the same process for like bigger like newer models, but of course the newer models are going to be way more specialized for longer context and essentially essentially being able to scale this train the they're architected in a way to be able to scale training to as many tokens as possible, which you won't need in this case. And lastly it's the it's the training loop, which this is generally the most important part when you're training a new model. Like if you check the difference between like GPT-4, GPT-4o, and GPT-5 or even even if you go before that, what you'll see mostly that usually the pre-training is very similar. It's the fine-tuning and post-training and essentially what you use with a with the same base model or like very similar base model. And the way you train it actually makes a big difference in performances. And now we see for example uh Gemini 3 comes out and then has this this this many this good benchmarks and then 3.1 comes out that has like double the performance in some benchmarks, which is crazy. Like obviously it's very sim- the models are very similar, but actually during training the train the the train the new model in smarter ways to improve performance like very substantially. Now as for the end and lastly of course the inference part like that's going to be very easy for us like because we're just it's going to be a small model that can run everywhere. Uh so like without that's going to be very simple uh part of the the building blocks. Uh then the prerequisites, any laptop will do uh that has at least six 16 GB of RAM. You can work with with with smaller laptops as well, it'll just be a bit slower. Bigger laptops you can crank up the batch sizes higher, so it will train faster. Uh Python 3.12 and I expect uh I expect that most of you have like some idea of how to write Python. Uh like if you don't nothing is still quite you can just copy paste things until something works or like ask for some help. Uh this training can uses uh Apple Silicon, so the the MPS architect the MPS architecture, CUDA or CPU, so it can basically support everything. Um and as for getting started, uh this just makes things easier. Like I'm using UV for this project. Uh so if you have your laptop out, you can install UV on your machine if you haven't. Uh it's it's quite straightforward and there's no UV is quite simple. You can just um UV sync and it creates a virtual environment for you and makes your life easier. And we're going to write the code in a scratchpad or if you're using Google Colab, which depends if you're into those very good, maybe that's a better idea. Uh if you just go to um if you just go to and create a new um let me actually create a new on a separate one so we can we can follow along. Yeah. So if you open Google Colab and you create a new um Colab project, you can run this command which just installs the what we need, which is just torch, numpy, tqdm, and tick token. Tick token is mostly for uh for testing things out. And yeah, we can move on to the first part of this um uh of this workshop, which is going to be the tokenizer. As I mentioned, like this is generally the first thing you think about when you create a new a new a new transform model is what tokenizer you're using. Uh I come from the voice world, so this is where that that's the one of the most important things. Like we think, "Okay, we need to train this new TTS model and we're going to spend maybe 6 months thinking about the tokenizer, and then we're going to spend 2 months on the architecture." So that's that's like generally one of the most important things of deciding how to create a new transformer. For those who don't know what a tokenizer is, like LLMs don't see text. They they work with embeddings or like vectors. Uh so we need like some kind of representation of those vectors for the model to be able to process. Uh what we're going to be using here is a character level alignment. Uh a character level tokenization, sorry. Just because it's it has the lowest number of possible tokens. Um in the in our case for on our uh data set is going to be only 65 embeddings essentially cuz there's the 65 different characters that will appear in the in our training data. And the way it works uh like this this going to be using the Shakespeare data set there's a few works of Shakespeare. It's part of the um it it it's part of the repo itself but you can if you're using um if you're using Colab there's going to be a link on how to download it on Colab a little bit later. And essentially we're going to use this this story library which basically just converts uh will be converting strings to integers and that these integers will then be turned to embeddings through the embedding layer when we train the the LLM. And it's a very very simple straightforward tokenizer uses this enumerate uh function from from uh from Python and just then selects that specific item that has selected with a uh this dictionary. So yeah as I said before like we use we use character level because it's much easier to train. Uh because we have 65 65 uh tokens only it means that we the bigram combinations will be like 65 \* 65 so 4,225 possible bigrams. Bigrams are essentially like when you have one token and then you predict the next token after it. And this concept of of of bigram is very is a very important concept when when you're training transformers cuz you want your model to see as many possible bigrams as possible. So, if you have a model with let's say 200,000 tokens, you need 200,000 tokens squared at least data to be able to to train like a from scratch in a in a in a in a very good way. Uh so, the or at least like this is the magnitude that you're looking for. In our case, 4,000 bigrams is like very doable. Like this this data set should be included all bigrams like multiple times very likely. Um if we did try to train using a full tokenizer, this will never converge. We can just be training it for hours and hours and then our model will never be able to to get good results. Now, the problem with with character level uh tokenizers is that they don't really scale very well. Uh because the models they the way they work is that they need to understand correlation between different tokens. Um so, you can very easily have a correlation saying "The sky is blue." Like these tokens combined together like make a lot of sense. But S K Y and then I S and then B L U E that it's a bit harder for them for the model to be able to make like good uh to attend to these tokens in in in a good way. Uh so, this would this would work quite well for our example. But if you want like a very very good model, uh one it will be expensive to train because of course like you have to create like a ton of tokens when during inference and during training. Uh but also it will just never converge to something good cuz the model the the tokens combined don't make too much sense. Um so, that's our trade-off, but this is a trade-off that we're willing to take because of course we're running a a small model. But in the future if you want to expand this to something better, if you want to train like a proper LLM and you're happy to train for like a week or using bigger GPUs, uh using a proper tokenizer, that's used like byte pair encoding is like the most common uh way of like doing tokenizers these days, which essentially takes all You can the way you would train a BP tokenizer is that you take all your training data and you define all common patterns and combine those common those common like character patterns into specific tokens that then you can reuse and the model can understand relationships of. So, the the way that this connects to the model itself is we have this embedding table which is size vocab size and then it's what I said before, it takes uh a vector of integers and then returns uh sorry, a a list of integers integers and then returns a list of vectors which is going to be their embeddings. Uh and like like I said before, like that's if we did use uh such a big such a big tokenizer, that would also be like more than twice the size of our models cuz if you just multiply this, our uh embedding size is 384 for this model that we're going to be training. So, that's already like 25,000 parameters and in that case like if we used GP the tools vocab which is 50,000, that would be 19 million parameters which is like more than three times the model which wouldn't make sense in our case. Uh so, now moving on to the next part of this workshop which is going to be the transformer itself. And as as I mentioned before like the transformers, they have been kind of commoditized now. The way transformers work like there are different labs that find different optimizations, but the optimizations like in principle more about we have this like base like idea that works really well. How can we make it train faster and like have bigger context and and people find ways that are more optimizations than than than necessarily reinventing the wheel. Uh for this at least there are some of course hybrid approaches that are more complicated. Uh so I'm I'm not going to go too deep on how transformers work. Uh And also to prove a point that you don't necessarily need to know in a very deep level how transformers work to to be able to train something like this. Uh when I first like did this project myself I didn't I had no clue how transformers worked and I still didn't that much at the end of the the work the previous project I worked on. But it once the more you work on it and the more you like you have motivation to continue pushing through you can understand all the different concepts together and how they they they stack together and the reasons why they ended up the way they are. Uh to go back to the big picture uh transformers are are are using like these four different building blocks. Uh one is multi-head self-attention. Uh attention is like what the difference between that makes it transformers different than other neural networks is that they can actually attend to previous tokens and understand the relationship between tokens that I was I was mentioning before and that's where attention comes in. And of course the bigger your attention is the more the model understands those relationships. And going back to what I said that's the what like big labs like Gemini they're trying to do 1 million context and they're finding ways because if you just try to use a 1 million context for the for a model like this it would just break very easily like the the math wouldn't work. So then that's when like the engineers from the researchers from Gemini found ways to to make it work. And that's what that that makes a difference but again fundamentally it's like the same the same architecture. Uh next one is the is the MLP uh or like the feed forward network uh which essentially takes this different uh uh relationships between those tokens and while taking them while the the the relationship themselves are uh arranged all of the relationship between the tokens themselves, it it combines them together to to be able to generate the logits that will then be uh so it's essentially kind of a it takes the context and then organizes it in ways that the model can then generate the logits and then generate the tokens. I hope that was a decent explanation. And then you have the residual connections which are basically there for the model to to not have to reinvent itself after every layer. Uh residual basically means that every layer that that passes through the uh cuz as as as we'll talk about later transformer is is is is built on multiple different layers that we pass through the activations one after the other. And residuals are there so that each activation doesn't completely uh restart things from scratch. It just changes them slightly. So it just takes the previous input and makes a small and not just small difference. The next layer does the same and next layer does the same and this will continue on. And this way the model doesn't the each layer doesn't make a huge change on the inputs themselves uh and the model can be more stable during training. And lastly with the layer normalization has a very similar uh has a very similar role in uh being able to the the layer normalization is able is is allowing you to scale down those activations in ways that uh that allows that those activations not be to not be exploding into very big values. So if one layer like multiplies your activation by 10x, let's say, the layer norm is going to push it back back to like normal values, so it doesn't go 10x 10x 10x, and then you can have like millions of values that started like 0.5 and then end at 10 million. That's what the layer norm is for. Um but again, these are just the building blocks. You don't necessarily need to know like why they're there and what the what the purpose of them is. Like you you learn this as you start working on these models more and understand how does the why these decisions were made. Cuz all of them are as like as I'm talking to you, of course, were done by We have a certain idea. Oh, this didn't work. So, let's add this to make it work. Sorry, go ahead. Are you showing me some slides that we haven't seen yet? Oh, oops. Yeah, I was showing. Where is it? I apologize for that. Yes. So, I I was I was going through the tokenization. Uh that was the previous uh what what my previous point. Uh and that's if you if you can go back and you'll have to go back through and forth through the slides when you're working on your the model yourself, cuz you can you have to like copy paste the the the parts or figure them out yourself. But it it was explaining why we're going with character level tokenization and um and what the other options we have. Then for the transformer, I will explain the different in the big picture, the different blocks of of architecture. Um and then now going to how does this look like in code? Uh cuz all the things I described to you actually are very cool There's very little code you need to to to implement them. Uh first we start with the basis of Sorry, go ahead. Uh when it comes to code, does it apply and because at least in English, you know, you have words, vocabularies kind of fixed. But in Python, you have your own variables and so on. So, how does tokenization work in that setting? What would be a token? Sorry. So, in a programming language, of course, you have programming syntax, keywords, but then you have also normal variables, function names, and so on. So, when it comes to tokenization, how does that work? So, yeah, as I mentioned to you before, uh we're going to be using like a character-level tokenizer because it's going to be easy for this project, but most big labs, they don't use character-level tokenization. They They use what I was mentioning earlier, the BPE tokenizer or like byte push Essentially, they they what they do is that they You You look at your training data. Let's say you have like this many trillions of tokens, and your training data is going to be including like code, as you mentioned yourself. Uh and the way it works, it will just look at common patterns. Uh so, if you have like a lot of training data that is that is code itself, it will it will see, and then it will realize, "Okay, for loops seem to be like a good candidate for for that to be a token." So, the for is going to be for sure a token. And then you might have some like enumerate. Enumerate you Do you see quite commonly in the um in the training data, so that will be another token. So, it will look at all these different tokens, and then create this tokenizer based on the common uh relationship between them. Uh of course, like maybe some languages that are like maybe Pascal that might not be very common in the language, so this the keywords of Pascal might not be like very common. Although, I do think that there probably is like good representation in the tokenizers, too. But some other of those like crazy like white space-only like languages that this these ones probably are not going to be work very well. My question was more on the So, one thing is the keywords of the programming language, but then you have your own variable names. Which are not common among different programs. \>\> I I see that's what you mean. Making them as tokens, I'm not sure how does that help. So again like there's no there's no specific way there's no like human in the loop in this process. It depends on your training data that you use to train this tokenizer. Uh if your code like there's like foo and bar like that's like common variables these were probably are going to be tokens. Like if your your if your variable names are like very strange like random characters and yeah probably that's not going to be in the tokenizer. And when this happens uh the tokenizer will fall back to character level or like bit by byte code level tokenization. So if your if your token is like just random like gibbles it will just be each character or like some maybe some combinations might be together uh as different tokens but most of them are going to be separate tokens and that's going to be a bit of a pain to for during inference and uh maybe not not not not the best if you want uh efficient inference. Thank you. Um but yeah going back to to the transformer side of the things um as I said the the models are generally quite uh quite quite similar um and the code for actually implementing these transformers is also quite simple and like easy to write. It's like maybe maybe 100 lines actually usually more than that less than that. Um and the first thing we have to to to accept is what the parameters of this transformer should look like. As I mentioned before vocab size is the size of your tokenizer in our case it will be 65. Block size is essentially the sequence length or like the the the context window. Um in our case that would be 256 which is very very small for these models but because we're training a model locally like that's kind kind of what we have to do. Uh bigger labs would use like 1 million context size for something like that. Uh, but generally like 16,000 is like a common middle ground. Uh, then we have the layers. How many as I mentioned before like uh, transformers have multiple layers and then you run the activations through each one of those layers. We're going to go with a modest number of six. And then the attention heads like the the way that uh, attention works is that you might you'll have going to have different heads for attention that are going to be attending for different things. Like one of the attention heads might be looking at punctuation. Maybe another attention head might be looking at the grammar. Uh, so all these different attention heads are attending to like a specific uh, uh, feature of the text or if you're using audio a specific feature of the audio etc. Uh, and lastly that's the embedding dimension. Uh, uh, that's how big the actual vectors of those tokens that you uh, create are. Uh, in our case we're going to start with a 384. That's like the standard for GPT-2. But a bigger value of course would have more information per token. A smaller value would would have less, but that's like a pretty standard value you can start with. Um, as for the code itself, uh, feel free to copy paste part of this uh, if you want to spend more time understanding it. I don't want to go too deep into like all the little details of how things work. Uh, but essentially like you're generally have like this overarching module and I guess we're going to call that GPT and that overarching uh, top level module is going to include all the other modules that I described earlier. Uh, in this case it will this will take the config. Uh, in this case this this will take the config that we have we we had above and then we'll create this uh, using we're using torch uh, module dict here just because it's like the easiest way to implement this. But this is all just math like you can everything you see here would just be like calculations like two matrix multiplications that torch allows us to abstract and make things easier. And pretty much all the things you see here you see here are like neural networks like smaller or bigger that are combined together. Uh Yes. Yes. Um not necessarily. Sorry, maybe you can repeat the question with the microphone. So the the the size of the input sequence is related to the number of parameters in your model. Uh not necessarily. You can have like large sequences with like small models or like small sequences with with with bigger models. The The main thing that this shows is essentially there are two ways you can train a model. You can train a model like that looks at that's a full attention model let's say that looks at the full attention. So it looks at the whole past uh essentially uh which is what we're building here today. Or you can have a windowed model. That's where most of the like bigger models that are out right now that only look up to like this much in the past. And this parameter is the size of that window that Okay. \>\> Yes, it's what we're going to be using to train. So the model will will not have seen anything greater than 256 56 tokens in the sequence. If you try to go above that, it will it will pass out. It will have issues. But so let's say we have like computational capacity to increase the number of parameters to make the help the model reason better, right? Which number would you crank up here to say it? Like would you increase the number of layers, attention heads, or um I I would increase all of the all all of what they see here except of maybe the the embedding size and maybe dimension. I don't remember exactly what other models do the exact numbers. I think that seems reasonable to me. But everything else like it works well for like a small token small a small model. Um 256 block size is tiny. Like that's if you try to run this on chat GPT like you would just forget things that you wrote like 10 sentences before. Um but the the bigger you make this the harder it is to do the training. And that's what I was talking about scaling laws is that you can't just say he go here and say okay actually I don't want a 256 I want 2 million context length. You can't train this like that at least with this current architecture. So that's what like like researchers after like GPT 3.5 came out they were like okay people are complaining that we only have 16k context size. We want 1 million. How do we do it? You can't just change this number. You have to change the architecture to allow training to be able otherwise you're just going to go like out of memory like instantly. Uh so that's like one of the things that researchers now try to to to figure out how we're going to be able to increase these numbers here while keeping training stable. Um Or how how do you increase these numbers here while keeping training stable and still able to to have enough compute to be able to do this? I hope this answers your question. Awesome. Thank you. Um So yeah the the first thing we would have to do is as I mentioned before like create the the top level representation of our model that you can see here as the GPT class. Uh I'm not going to go again to to too much into the details but like you want a one would be the how the your model like understands embeddings and also how does it understand embedding position. Uh I don't want to go into details but essentially you need the tokens need to be able to understand both. Um then you'd have essentially here all the layers that I was mentioning earlier and each of them is configured as as a block, which we're going to go at the definition of block later as a little bit later. But essentially a block is like a layer of the transformer that includes its own attention and its own like layer norms and it continues the it feeds into each other the the one you do the forward pass. And lastly, you have the LM head and this is the LM head takes all the outputs of the above and connects them together into what we call like logits, which is the distribution of what the the next token should be that should be generated. Uh, cuz I as I mentioned earlier like maybe I did maybe I didn't, but the way transformers work is that they predict next token. So, you take the previous the previous context, you predict the next token and you have to sample this token on on this distribution. And the LM head creates this distribution that we we sample from. Then the the the next important bit that uh like is again like quite quite straightforward like it is it's quite similar in most of these models is the forward pass uh which essentially allows the model to um to go to to take the input, which is going to be the tokens that we mentioned and push them through all these different components that that that that I mentioned earlier. Uh, and just does some um uh does some some some pre-processing, turns tokens into embeddings and positional embeddings uh I I I add them together, then goes through all the blocks of the transformer, all the different layers of the transformer and just runs the the forward pass of that transformer. And then does the linear norms and then passes through the LM head to get a distribution of those logits that we mentioned earlier. Um and this is if you're doing training, this would also do cross-entropy loss um in this case and then at the end you're going to get logits and loss as the outputs of your of uh of your transform model. And I do have some diagrams here that you can see. Uh like they're quite they're quite basic, but essentially they show the flow of take token IDs, you pass them to make them into embeddings, token embeddings and positional embeddings, you add them together, you pass them through the transformer, then through the layer norm that just makes the outputs in a space that the LM head can comprehend easier. And then you return this distribution of like what the next token should be that has that has this this size and as I mentioned before, like 65 is the size of like your voca- our vocab- our our our token size. And now self-attention is like a little bit more complicated. I don't want to go again to go too deep of how how that works, but essentially at- attention is there to to understand the relationships between the tokens. Uh Uh essentially what what is the important like if I say the sky is blue, blue and sky have a a very big correlation. Uh so that's what attention does. Based on the how you've trained your weights, you will understand what tokens should be attending to each other and put higher emphasis on those specific relationships. Now again going back to what I was say- saying before about the what that's why the tokenizer matters a lot cuz sky and blue are very easy to very easy for the model to make this relationship. While in our case you'll have to like combine different groups of tokens together the different characters together, that's quite a bit harder. But this that's that's what attention does. It's It's the what what each token should be attending to in the past and what has most importance for it. And it has also a forward pass as all of these different blocks do. And in your implementation, feel free to just copy-paste these these blocks. Uh maybe you can spend some more time on understanding the diagrams and and how things work. Uh but as you can see, even something as complicated as attention has a very small It It's just a few lines of code to to implement. And yeah, lastly the the MLP block uh is again as I mentioned earlier uh also like again I already talked about the multi-head why attention has multiple heads cuz each head attends to different parts of what makes language what it is. And then the MLP block takes the outputs of attention and and makes sense of all those different relationships into something that the model can then understand a bit better. Uh which again is just essentially just a a neural network that just combines things into something that's a bit more understandable for the LM head to then be able to to generate distribution for logits. Is Yeah, yeah, no, you can interrupt. Do we have the same MLP for individual transformer blocks? Uh Like is it the same architecture and the same weights between the different transformer blocks? No, each each block has its own weight. Each block The the way each layer essentially you'd have each layer that we'd call blocks. Uh they usually have like a different section for like a different prefix for the weights of like MLP would be a different like maybe usually they're called FFN blocks instead of MLP, but it would still be the same. So, each block has its own weights. And the MLP each each layer Yes, each block has its own MLP. So, it's is you have its block has its own um uh as you can see here, it's on attention, its own uh linear lay linear norms, and its own MLP. That's what basically makes a layer of a of a transformer. And and that's what I what my point was going to be later. Everything comes together into this block that we can see here, which is basically a layer of a transformer that has some normalization, running the attention to get the relationship between the models, and then having the MLP that combines the relationship of the sorry, the relation between tokens, and then the MLP that takes those relationships and turns them into like a representation that's easy for the model to uh to create the logits. And uh this uh oops. This is what I was saying. Sorry. I have two different screens. I shouldn't be doing that. But yes, this is this is the transformer block. This is the the building block essentially of of a transformer. Uh It has layer norm, the attention, and another layer norm. Like not all of the models have this this configuration, but this specific one has this. And then the MLP that takes everything and creates it into a a representation that makes sense for us to be able to make the generations. And uh you can see here like a bit more of a like a simple diagram of how this this works. Yes? The original uh nano GPT had the residual connection as well. Do you remember? Uh so, the residual should be there. Um No, no. What I mean on the on Andre Andre Karpathy's nano GPT, did it have the I I don't I don't remember. Uh I It's been a while since I I Okay. Like I got I I I had the idea but from there, but I built everything from scratch. So, I don't I don't really remember if it did or not. Uh but yeah, residual as you can what, the idea of residual uh would be, as you can see here, instead of doing X equals the attention, you do X equals X plus attention. So, you you get the difference. The the the values of the activations don't change that much. Uh, that's the idea of like doing the residuals. Um, uh, and yeah. So, we have this like if you wanted to implement this yourself, you can copy paste all these different classes into one file that you can call model.py. Um, and this is essentially all the maths of how the transformers works. Um, and as I mentioned before, then we have to decide how big the transform is, like the parameter count that you can see here, uh, is just a 10 million parameter based on what I was saying above. Uh, you can just find yourself by just summing all the all the different parameters of all the different parts of the different blocks that we had above. Um, the token embeddings is 65 \* 384. We said this is the vector for each embedding, so it's 25k. The positional embeddings is 256. 256, remember, is the sequence length, the max sequence length of the model. So, that's another 98k parameters. And then where the the biggest part of the logic is is in the actual transformer blocks themselves, where most of the parameters live. So, you have um, you have 4x. Uh, so, the four there comes of the way attention works, where you have key value key query key and value pair. So, it's it's part of the potential has like four different parameters. For for um, for 384, which is the amount of tokens times the relationships between those tokens. So, 204 per vector times 384. So, that's 590 that we have for the for the attention per layer. And then for the MLP, you have again the similar logic of I don't remember what this 1,536 is, but yeah, in the end it turns out into being 1.2 million. So, total amount of parameter size we're going to be training is going to be 1.8 million parameters, which should be good for which should be easy to train for like most devices. Uh so, yeah, you can go back to this. It has a lot of details that that makes um uh that that will that will make sense for you to go back and understand a bit deeper, do your own research on on this. But, this is like the general like very very high-level idea of how these transformers work. And uh we're going to go now to the the training loop. And that's where most of the meat of this project is going to be. Um Well, we have this this transformer, uh how we're going to train it into do what we want it to do. The objective of this training we're going to be working today is something that one has to be very easily recognizable. Uh but, you know, when the model starts working and and groks, which means that it it understood what it's trying to do. It's going to be very easy to understand that the model actually works now. Uh and secondly, it should be something that's like quite easy for the model to learn. Uh if you if the model had to to write uh Python uh like a competitive programming for you, then yeah, that's that's that's a that's a very hard task, so we're not going to be able to do it here. But, in in our case, like the the objective is going to be to create a Shakespearean-like LLM that can create uh verses from from Shakespeare. Um And as we mentioned before, that these models like are learning as next token prediction. So, the way that cross entropy works is that you take your current the current tokens that you want to train. In this case, let's say that it's T0, T1 to TN. And then you wanted to predict T1 T2 to T plus one TN plus one. Uh so, the way this cross entropy works is that you take your sequence and you just offset it by one as like what the model needs to learn what to calculate. And of course the last one you don't have it so you have to like cut it off. Uh and then the model learns how to predict the next token based on based on this logic. Uh now for the actual training code itself, um first we need to load the data and we have this function that that loads the data. The data is already in the uh repo itself. It's in this data directory. It's a It's a collection of different uh different uh like lines and verses from from Shakespeare. It's It's about 1 million tokens or 1 million characters. And we first load the data. The data loading is is is is very simple, too simple. So, that's one of the things that I think you can optimize. Um essentially just takes all the tokens, uh splits into a validation and a and a training set, and then just essentially shuffles it and takes part 256 batches uh of sequences of of of of tokens that are from from the text itself. And uh and just essentially use it for training. Batch size going to be 64. So, it takes 256 uh token sequences, 64 of them, stacks them together, and passes it to the model to to to teach it how to do how to train. So, a very simple data loader. Usually these data loaders are can be quite complex uh because of course like if you have especially when you have higher context lengths, like the way you load data is like a very big part of how the model like lis- to learn. But in our case, we have a very very simple implementation. Um next up is the way that this how the code works for um we need to get to use a device. Uh this code works for MPS, it works for CUDA, and it works for CPU. Depends on what your your laptop supports, or like if you run things on Google Colab. Uh like if you run Google Colab, it will detect CUDA and it will be quite fast. MPS is also quite fast. CPU will be the slowest. But that's like it that's low it work decently well. Uh next up is the learning the learning rate, and that's like a big part of how the the models are able to learn. Uh the way the models work is that you first start generally with as high of a learning rate as as you can, as you can afford without making the model become unstable. Uh so it basically makes it go crazy. Uh and the way it works you start at a very high learning rate, which is essentially the the amount of the model is able to learn per step. How much the the weights need to move into the direction that you want. If you have a very high learning rate, you would offset your target, so it can go off the rails very fast your model. So you want to use a very appropriate learning rate. Um for your model to be able to train well. And usually you have this this concept of a warm-up where you start with a very small learning rate so that all your optimizations uh the the model weights are able to to to stick to places that are uh that again that they don't go into like they they can they can start in places that are appropriate for the training to begin. So you start at a very low learning rate, increase it slightly until until you reach the peak, and then at the peak you continue you start reducing the learning rate, and that's what we call weight decay. Um and uh until you reach a point that you're you're satisfied with it. Some people For some people that is zero. Like I prefer to not go to zero cuz then it's hard to restart the training. Um but that's the idea. You want the learning rate to be less when the model is like close to being perfect. So small you you want it to start with very big changes to find like good local minima look look global minima and then it for it to calibrate us training goes goes further. Um so we'll start with like a small warm-up of 100 steps and then we're going to use a cosine decay until the max steps that we're going to be in this case is going to be 5,000. So you're going to start from like very low peak at 100 steps and then start going down down to uh to 5,000 steps. And that's what the AdamW normalizer does. Uh it's essentially allows this um the the the this concept of like controlling the learning rate um using this cosine decay. And that's the most common normalizer people use. At least used to use. Now there's better normalizers, but this is the most simple to to start with. And here's how the full training loop would look like. Again, like not not not that much code. You just uh initialize your config. Uh like in this case it would be like a six uh six layers, six attention heads, 384 um embedding size, and then 256. That's the sequence length. Uh you initialize your model. And then you create your um you you start your optimizer. You start your steps. You take a DM helps with like tracking like the losses and all and all that kind of stuff. You want your loss to start high and then keep going down until you reach levels that are acceptable. Um and then another important part is evaluations to make sure that your model actually works well. That's what the val loss is here. So, it's very easy for models to like overfit, especially like this this this small models, cuz you don't have that much data. So, the loss might keep going down, which means that the what the model predicts and what the training data is are very similar. So, your loss can keep going down a lot. Uh but actually at one point maybe your model actually overfits in this case. And when it overfits, even though the loss goes down, actually the performance of the model is worse. That's why we have this val loss, which is a is a part of the data the model has never seen, and we run a forward pass to get the loss of that specific part of the data set. And if this is very low, it means the model can is is performing well, cuz it had the model has never seen this this data, so it can't memorize things it had never seen. So, that's what that's what we have here with the val loss. And uh Now, I'm not going to explain the concept of of of of of uh backwards losses, but essentially that's the way the model the model weights move towards the direction of being optimized. Uh so, for for each step, we have a batch size of in our case 256, um 256 tokens, batch size 64. So, we we push this matrix through our model, and then it learns, and then the optimizer does an extra step, and now the learning rates are adjusted depending on what the steps you're in right now. And lastly, just to have an extra way of uh uh every every 1,000 steps, you save your checkpoint to be able to restart your training if you need it from that point. And we're also running uh inference on the current checkpoint to see what the model actually predicts at this point. Uh so, what we're we're going to start seeing is when we start from from big the beginning because it's a model we're training from scratch, the loss is going to be essentially random. And in that case, that would be natural log of 65, so it will start at around 4.17. Uh that basically means the model like knows nothing. It has no clue of what this data is. Uh and slowly we're going to start seeing the loss going down to 3.3. That's when the model is going to understand character frequencies. Uh it will still not be able to do words yet, but it might understand things like th as part of the, which is a common word. Like th is going to be part of the things that it starts generating. Then at around 2.5, it's going to it was going to get a little bit better about this th, and then it will understand the word in and stuff like that. Then at about 1.5 to two losses, it will start actually creating words. And then at around one 1.0 to 1.2, that's when the model is going to start being decent at at at this task. You will actually be able to understand names from the text. It will start creating things that start making sense. But then when the loss starts going below 1.0 for this specific data set, that's where we're going to start seeing overfitting. The model will still be like producing like reasonable things, but it will no longer start be getting better at it. Uh this is like an example at 200 steps when I was testing this. Uh it was just producing like complete nonsense. Uh and the val loss was around 3.5. Uh then at about 800 steps, it started producing like decent things. Uh still not not not great things, but it was it was starting to get there. And at 1,000 steps, it got better. And there was one point that the val loss actually started increasing instead of decreasing, and that's where we know the model overfit. So at around 2,400 steps is where the that was the optimal performance of this model. And then if we kept going, the the performance was actually maybe not decreasing, but the models are becoming less creative. So, that's one of the things to to keep in mind. Now, val loss is like not the best metric like for if you're actually being serious about training LLMs, usually you might have like some benchmarks that are running as part of your training and you can see like if the benchmarks are getting worse or not. Uh but for us that's like a very easy and cheap way of like understanding how the model is doing. Um So, yeah, that what um next steps now like uh probably to actually train this model yourself. Um Considering the internet is already good, I would suggest probably using Google Colab for this. Um you can you can copy paste stuff from from from from here. You can glue things together. Uh Copy pasting will probably like allow you to go get 90% there. Uh there is a lot of room for for improvement for what I did in here. I on purpose I did it like super simple and there are things that you can improve yourselves. Uh but the idea is to uh to get something working. I hope everybody will be able to get something working if if you're interested in do in do in do work on this. And have something that starting from nothing and getting like a model that can actually produce like a result that seems reasonable. Um after we have trained a model, the next part is text generation which is like the inference side of things. Uh we're just going to be using uh there's a multiple ways to to do to do inference. One way is like greedy decoding. It's what I was mentioning earlier. You have all the logits which is as a distribution of tokens and you just take the most likely token and that's what the greedy decoding means. Uh so, let's say that you have like your the token T and the token H are are both in the distribution. One has like 80% probability, the other one has like 15% probability. You always be taking the the top the top one. Uh That's greedy decoding. Greedy decoding doesn't work very well for LLMs. It can work well for other models, but for LLMs, like this essentially makes them very boring and not very creative in what they generate. So, you pretty much never want to use greedy decoding for LLMs. You would want to use it for for other models like transcription, for example, greedy decoding is the best cuz there's usually like only one way you can you can transcribe something. You don't want it to be creative in transcription. Like that's not that's not a good idea. Um so, that's what greedy decoding is. In our case, we're not going to use that. We're going to use temperature. So, essentially what temperature is is that you're not always choosing the highest probability token. Sometimes you might choose the second highest probability token or like the third highest. And uh even though it doesn't make any sense, why would you choose like a worse token for the situation? It has proven that this actually makes the model perform better. Uh It like sometimes it might like go into like these weird loops, so there aren't like techniques you can make sure that it doesn't go like crazy by generating some nonsense. Uh the worst thing is if if it predicts like an end of transcript token and end of text token and then it's like stops the generation, which maybe you have seen sometimes when using ChatGPT where suddenly it just like stops for no reason. You sometimes it's because of that, but there are ways you can you can you can um prevent this. Um Generally like a 0.7 uh temperature is like the best. The best middle ground to make to to inference work well. And then you have top K sampling. Uh which essentially like it prevents the model for if you have like, let's say, five tokens that are very likely and then the sixth token is like completely unlikely, top K sampling sampling prevents the model from predicting this like sixth very unlikely token, even though the temperature might you might get unlucky and the temperature might hit it. Um so, that's what top K sampling does. And this is like our inference function is like it's very straightforward. It just does it just runs uh takes all the tokens as as the input, passes it through the model, takes the the logits out of the model, and runs uh soft this is that what I was describing before is called soft max. It takes using a temperature the probabilities and then uh it it decides what the the next token should be based on the on those probabilities. Um and one thing you can do you can use seeds and essentially seeds is that it like right now everything will be random if you just keep returning it. But if you if you keep returning it but if you use a a set seed then your inference will always be returning the same value. Uh which will be uh relevant in later. Uh then putting it all together if we put all together we should have three different files. One is like the model.py that includes our model architecture. One is train.py which includes the data data set loading and the training loop. And lastly the generate.py that includes our inference. And in total this should be like maybe a few hundred lines of code. Uh very straightforward and that's with even with this much code like if we have a lot big hardware we could train a good LLM using this architecture if we have enough data and enough resources. That's that's all you need essentially. And that's what basically like GPT-3 and GPT-2 like when OpenAI released it I remember when OpenAI was about to release uh GPT-2 and they were saying we're not going to release it because it's too dangerous for for humanity. And that was like back then. It the that was the code that they were working on and like a lot of data. And of course a bigger model. Uh now this seems to be like kind of kind of funny but for them like that was like a very warm moment that we did this and then the model like actually performs really well on tasks. Um, but in the end it was just it was just what you see here. Um, so yeah, putting all together, uh, you can uh, if you use Google Colab, you can use this this, uh, snippet of code to download the the data set and you can use this pip install command to install the different dependencies. Uh, and then it should look uh, like something like this where you install your dependencies, you copy your code. Uh, and you might need to do some, um, connection some connections together, but essentially like you can run a a train that you can use a train command to run to to and use the data set that you want to use and that this will start the training and then you can see how the performance of the model improves step by step. Like for me this took about like 15 minutes to train on on Google Colab until I started getting good results. Uh, with some improvements you can make it go faster and maybe you can make it go slower, but actually get better results. Um, but essentially like it's it's very simple to get to get working. One thing to remember is that you have to change your run time change run time type to T4 GPU uh, because this this this is free and it will it will run quite fast. Um, hey yeah, feel free to train. Like you can try like maybe starting with a very tiny model the 0.5 million parameter uh, with a whole that has two layers and only two attention heads and a smaller embedding size for for each token. And then you can try bigger and bigger models until you get some usable results. Um, and as I mentioned before you can try different context lengths. I started with 256, you can try bigger, 512. Um and then you can if you want to monitor and see how your losses went down like in a nice way, you can use this like uh uh this pipe plot lib uh to see like your the graphs. Uh the things that you want to look for is if your train loss is not decreasing, it means your model is not learning. Uh the problem is you have a bug in your code. Uh if your uh the you you also could be like a a low training loss. If your train loss is is decreasing, but your val loss is not it's actually increasing, that means you overfit. Uh and if you have very weird spikes in loss like the loss would be like very smooth in general. If you have very weird spikes in your loss, it means like again there's some some kind of bug either in your data or in your training. And uh when the model starts plateauing and not getting any better, it means you kind of you have pretty much exhausted the usefulness of your current data set. So either you going to use a bigger model or you going to need you have to need more data. Uh now one one like interesting part about what uh I would like us to do is uh I think it'd be cool if we had like some sort of competition of who can train the best model here if uh anybody is able to to get things working. Uh hopefully the data is good enough. Uh And also for further reading, you can see like a lot of the resources for this uh for this workshop. Uh but uh essentially what the challenge is is we can we can vote all together to find which which model actually produces the best verse of Shakespeare. Or it could be if you use a different data set, a best poem or in that kind of category. Um The rules are you have to train the model yourself like here and today. Like it it can't be like some you can't ask ChatGPT to give you like a good verse. You have to like use your own model. And to prove this, you'd use like a seed with a specific prompt and see what output it gives you. You're free to like regenerate as many times until you you get the best results. Um and uh you I'll have like a QR code that you can submit your results and I can go around and like help people out that uh if you need any help getting uh training running. And uh of course uh this training is like super bare-bones. There are many ways you can optimize this and make this better. So, I guess for people that are a bit more experienced, they can they can implement this this uh improvements and maybe get a better outputs of your other model. Uh the winner is going to get some uh free swag from from Level Ups, maybe like a hoodie or like some some free credits. Uh I'll see what I can actually give. So, yeah, this is the submission. Uh you need to be creative and then I'm and this will be essentially like a good verse. And we can maybe use like a kind of um a kind of bracket. Oh, what happened? Oh, cool. We can use a bracket and then we can vote which verse like sounded best. That could be funny, that could be like like well-made, like up to you. And uh the reproducibility should look something like this. Uh where you run python generate on your best checkpoint, the prompt that you decide that you can it's up to you, temperature and then a a seed that that proves that you can actually produce your results. Uh you can try different model sizes. Uh like I guess like 85 million parameters is maybe too big for this, but if you have resources, you can try it. Uh you can try better tokenizers, like this one of course is like character-based, but perhaps you can train your own BP tokenizer uh based on that data set. And there's also other tweaks that you can do, like bigger context, uh like there are some some training optimizations, like using a dropout value. Uh you can stop whenever you want, when you feel like the model is good enough, change the learning rates. Um and essentially get like a make the model as good as possible. Uh yeah, that that is the idea. Uh I'm going to go around and help you out. Sorry, go ahead. So, reasoning models act to to repeat the uh uh You're asking like uh if there's any models that are quite a bit different training. The base the building blocks are very similar. Like you can train the same exact model, you can post train it which is how like usually reasoning is is uh is being taught to this model. You have a good base instruct model and then you post train it to be a reasoning model. This is very data set driven. So, you need very very high quality data. And you're going to use a loss that's like good enough to be able to uh to learn this data in a in a very good sense. Uh the complication of reasoning models is finding this good chain of thought data. Uh that's why like OpenAI has like all these labelers that are like PhD students that they write down the reasonings of like how to how they solve problems. Uh because this data needs to be very high quality cuz it's it teaches the model how to think. So, you can't just go on like Reddit and just get random posts. You're not going to learn how to think this way for sure. Uh you need like some very high value good quality data to teach this reasoning process. But in the end, reasoning is is essentially just adding to the context of the model like this to to the attention essentially. Uh this like logic that then the model can when it generates a response, it can go back and attend to those reasoning tokens and get a better response out. So, it could be like describing the model a bit better then it goes back, sees those tokens that it described, and say, "Oh, actually yeah, I already figured this out. I'm going to write it down now." Okay. Uh the microphone is not working. Is it? Yeah, perfect. Uh so, reasoning and non-reasoning models share the same base often, and then one is post-trained in a different way and add this uh you know, in a way different post-training. Yes. So, a a lot of the labs, like let's say the current model that was released like 1 3, usually they release like a base version, an instruct version, and the usually the base version doesn't have a chain of thought reasoning. Uh it's usually like the same model that they first pre-train to be like quite good, maybe do some fine-tuning as well. And then the next step is doing post-training to teach it this kind of like this this uh this knowledge. Uh so, that's why like you see a lot of a lot of uh big improvements that happen very fast in the industry, like Gemini 3 to 3.1, it's essentially giving it like better reasoning data and like like better fine-tuning post-training data for like specific problems. It's a little bit like benchmarking, like usually this data is like very similar to what the benchmarks, the current good benchmarks are. But yeah, essentially just taking a base model, using this new data to improve it to the next level. Okay. And second question is compared to what you are showing us here, like this very bare-bone model, has there like has there been fundamental innovation in the main training of the model like powering nowadays models or is it mostly the same, but just with smarter tricks, small tweaks, better data, and so on. So, I like are everyone still using the same attention layer and so on or are there like some fundamental shifts that have been happening on the latest, and maybe you know, maybe you don't, I don't know. No, yeah, no, yeah. So, you're you're correct. That that So, they a lot of this base is the same. They might do some changes in terms of how attention attends to those different tokens because sometimes reasoning can be quite large so you need big sequence lengths to be able to get good results. So there's a lot of like tricks that the new labs that the labs do to make this to make the attention more efficient. But overall you can take like a very you can take this model GPT-2 and make it into a reasoning model. Like if you have the the data and like a big enough model to actually learn from that reasoning as well cuz like this time models reasoning won't help them too much. But if you had a big enough model that can get stuff out of the reasoning there are people that have taken like older models like let's say llama 1B that small and weren't trained for reasoning and they made them into reasoning models using the exact same architecture. Yes. How much effort do you put in you know getting your golden data set? What is that process that you follow? You mean for post training? Yeah. Yeah so like as I mentioned before like usually what most labs do is that they go to companies like Scale AI that's probably the biggest one. And Scale AI has an army of like people that you can you can say I want physicists. Give me data from physicists and Scale AI is going to find contract physicists pay them like as much money as they need. And then these physicists might like write things down they might be contacted to do different things. But essentially like a lot of these companies they like Scale AI provides data for like Anthropic and another good one by Meta so probably not that much anymore. But essentially they they these data sets are provided by these like labeling companies. Scale AI is one of them but a lot of the the big labs have their own labeling teams too that they hire contractors to generate these data sets for them. Uh but as I as I said before like you want you if this data set has like even some small issues it can literally make or break your model. So, these data sets are like kind of the most expensive, like they will cost tons and tons of money, but they actually are the ones that make the models like as good as they are. Just a quick follow-up. So, even on that I mean, you still have to evaluate, but the answer wouldn't be exactly the same. So, our people are really reviewing this, or it's still you're relying on LLM to evaluate? Uh you you you rely on people for this kind of stuff. Usually, the way it works, uh a big part of my job is actually like doing this uh organization. Uh usually, the way it works is you have you might have like one person that generates the label this like very high-quality label data, and then you have maybe some labeler that have graduated into like a QA position, where their job is to essentially make sure that all the like other labelers, the more entry-level, their outputs are correct. And it's it's a it's quite a the tough job because like if your QA is not good, you get fired. So, it's like it's that that's the way they keep the level quite high. Yes. So, LLMs are non-deterministic, but um how does that seed parameter work? So, the the way the LLMs are non-deterministic is because they they essentially use random number generators for of like the of of of your machine. Uh or like of the the the the GPU that you're currently using of the the system. The seed essentially like makes all those different calculations to be always return always the same value. So, things are no longer random. Like you're you this is not only for uh for for this uh these seeds can be used for any random generation. Uh so, it could be like for password hashes. Yeah, it it works the same exact way. Yes. But, this is like you mentioned pretty deterministic, right? So, if we use that that they are deterministic, right? \>\> Yes. Yeah, if you use greedy decoding uh there might be still some things that like like it's good to use a seed even if you use greedy decoding cuz there might be some other stuff that maybe you don't you don't you don't control. Like maybe you're using some like sometimes greedy decoding might not be actually greedy decoding might be 0.01 like a temperature like for VLM for example that they do some electrics like that to be able to get uh good outputs. Uh so in in general it's good to use seed even if you're using greedy decoding. Yes? Um this is obviously all in text. Eleven Labs is mostly audio, right? Yes. How different is this versus like doing this with audio? It's surprisingly very similar. Really? It's more complicated for sure, but parts of the the stack of like most audio models they also can do text cuz they you need the the models need to understand language itself. And the best way to teach something language is text. Uh of course you can train an an audio-only model like uh And again it's like what I was talking about the tokenizer. Like how do you tokenize audio? Like what what is the concept of like a sound? Like how how big should your tokenizer be for audio? What should it be only human speech or should it also be like a music? Like do you take the notes of different music instruments as like different tokens? These are the kind of problems that you have to solve like if you're creating a uh an audio model but you wouldn't necessarily need it as a text. This is a bit easier. Uh but the fundamentals are still like the same. Like if you want to generate audio you would you generate like an audio token and then that audio token would be using a tokenizer and then you you it's not exactly the same cuz the audio token then you have to process it in some different ways uh to make it into like actual audio. But uh a lot of the same uh general ideas still apply. But do you have a lot more loss or something cuz with text you know like this is the text the text can be down to very clear like a one is a one but in audio like particular pitch or frequency or sound but like cuz then the unity there is already lost straight away right? Yeah yeah so the way it it works is that you don't use like this cross entropy loss you use different types of losses and you can use cross entropy but usually it's it's losses that are more specialized for what you're trying to do. Like for example there's a loss called L2 loss which essentially takes like two mel spectrograms and tries to see the difference between those two mel spectrograms which is essentially like a sound waves in encoded in a certain way. And that's like a very common way for TTS models to be trained you train on this specific types of loss. And the same thing I was mentioning earlier cross entropy doesn't really work as well for for things like post training you might use different types of losses there or if you're distilling a model from a big model to a smaller model you don't use cross entropy loss you might use a KL divergence loss where you find the the token distributions of like the bigger model and you try to match the logits of the smaller model so there's different types of losses for different use cases not all of them work the same way. Yes. So the way multi modal models work usually you don't use tokens in the same sense. Um Like okay this is where like it becomes a little complicated cuz like these models are not really built for this like the GPT model like the bit the newer models they have also like an embedding input which essentially as As As mentioning earlier like you have each token then corresponds to like a vector. But these vectors, they don't have to correspond to specific token. You can take these vectors from other places, too. And what a lot of these labs do is that instead of having a tokenizer for like video, for example, what they do is they have another transformer that they call a video encoder. And they put the video first through that video encoder. And uh this video encoder will be taking let's say you have a a 30-second video, it will take one frame per second of this video. And it will take those frames and then put them through this like new transformer, this encoder transformer, that works quite a bit differently than this one. And what you do is that you take the final layer of this transformer, the hidden values, which are also vectors. You take those those vectors out of this encoder and you're going to input them in the embedding layer of the of the transformer model. So, what the model is going to see is going to usually is is like prefixed. So, you take the you take the video, put it through the encoder, get some vectors, and then put those vectors in the embedding input of your transformer that does text. And the way it would look like if you look at the sequence, it would probably be like a prompt and then it would be probably like a video token representation. And then actually but the embedding of that video token is going to be overridden by the output of the encoder. So, that's how like this this multimodal models work. It produces a vector which is the same length as like a vector for words. Yes, it's exactly the same vector. And that's it for audio. You have an audio encoder and do the same but but for audio. And the but for in in terms of how what the model cares about, it just cares about like these embeddings, right? They don't care if it's text or if it's audio or if it's or if it's video. It cares about these vectors and that's how you you represent them into the the same dimension as the model the transformer expects. So, there's there's no relation between like the video of the horse jumping and the word horse. Is there no similarity between those two like Um maybe there is. I I'm not It depends on how you train it. Like it's These are the kind of things that are a bit black boxes. Maybe there is, maybe there's not. Actually, that's a good idea to like That would be a good research paper to to see like uh for video encoders, do they actually match in the same dimension as the text encoders? I I don't know. I imagine there probably is some uh connection. Sorry, you had a question as well? Uh yeah, I was wondering about like since you do both uh normal speech, do you also do some music generation? Is that like a very different problem or will the architecture be like similar? And also like when you have these harmonics and stuff are you still able to do it with just like a basic like uh auto regressive transformer or do you have to like do diffusion or is like things depend more on each other? Uh when you might want to like generate everything at the same time. Y- Y- You can do both. There's music models that are auto regressive, there's music models that are diffusers. Uh like it depends on how you how you train it. Um Like I think some of the some of the Google models are like like transformer based. Some of uh open source models are like diffuser based. Like both kind of work kind of work very well. It's just that this it's a little bit As I said, like it's a little bit more difficult to uh to to put into perspective this the concept of like all music. If you tokenize and predict the next token, it's kind of hard cuz it's very abstract. Uh so usually diffusers work a bit better in this like image modalities for generation or like music or like even audio for some models. Like they have some kind of diffuser diffuser um diffuser part of the process. Um both can work. Diffusers are generally a bit easier to get it working. I hope this answered your question. \>\> sense. Yep. How do you tokenize audio? I mean like in text you get repetitions like repetitions happens quite a lot. It's easier to combine if they're tokens. Very big advantage of that. Audio different voices that are kind of like how long how long it takes to play. It's very hard. And it's not really something that you do by by just sitting down and thinking, "Okay, I'm going to tokenize this word to that." This This is not something that you like sit down and like decide. You use some kind of process in the you train like an audio tokenizer through You have a It's a very similar like case, I guess, how you train like a text tokenizer. You'd use your training data and you'd find like common patterns in the audio and then tokenize those patterns. Of course, like when we say audio, we don't don't always just mean like the actual like sample rate and like the the audio waves. Usually convert them to something that's a bit more easy to to tokenize, to to do this processing. The most common one is like is mel spectrograms. It's First, you convert your audio to mel spectrograms and then use this this this this essentially arrays of like of numbers to train your your tokenizers. And it will be very dependent on your training set. Like if you wanted to tokenize music and use a music data set, your audio tokens for music are going to be very different than if you had a voice data set. That's going to be like focusing on like human voice. Now, the hard part is what if you want to do both voice and music? That's where like That's That's very hard. Any more questions? Okay, awesome. Yeah, if you if you want, you can start working on training the model if you have a laptop out or if you have already started. You can follow this. You you can you you can follow the the workshop uh that gets something working. And uh if we do have enough submissions, we can do the the competition and and uh see who whoever wins will is going to get some some nice swag. Sorry? Uh So, uh because we don't have that much time left, let's just say 5:45. Uh if you have any any questions or you need any help, uh please call me and I'll I'll come over.