--- schema: 1 type: video title: 'Building makemore Part 5: Building a WaveNet' description: >- We take the 2-layer MLP from previous video and make it deeper with a tree-like structure, arriving at a convolutional neural network architecture similar to the WaveNet (2016) from DeepMind. In the WaveNet paper, the same hierarchical architecture is implemented more efficiently using causal dilated convolutions (not yet covered). Along the way we get a better sense of torch.nn and what it is and how it works under the hood, and what a typical deep learning development process looks like (a lot of reading of documentation, keeping track of multidimensional tensor shapes, moving between jupyter notebooks and repository code, ...). Links: - makemore on github: https://github.com/karpathy/makemore - jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part5_cnn1.ipynb - collab notebook: https://colab.research.google.com/drive/1CXVEmCO_7r7WYZGb5qnjfyxTvQa13g5X?usp=sharing - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - our Discord channel: https://discord.gg/3zy8kqD9Cp Supplementary links: - WaveNet 2016 from DeepMind https://arxiv.org/abs/1609.03499 - Bengio et al. 2003 MLP LM https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf Chapters: intro 00:00:00 intro 00:01:40 starter code walkthrough 00:06:56 let’s fix the learning rate plot 00:09:16 pytorchifying our code: layers, containers, torch.nn, fun bugs implementing wavenet 00:17:11 overview: WaveNet 00:19:33 dataset bump the context size to 8 00:19:55 re-running baseline code on block_size 8 00:21:36 implementing WaveNet 00:37:41 training the WaveNet: first pass 00:38:50 fixing batchnorm1d bug 00:45:21 re-training WaveNet with bug fix 00:46:07 scaling up our WaveNet conclusions 00:46:58 experimental harness 00:47:44 WaveNet but with “dilated causal convolutions” 00:51:34 torch.nn 00:52:28 the development process of building deep neural nets 00:54:17 going forward 00:55:26 improve on my loss! how far can we improve a WaveNet on this data? canonical: >- https://liquidgold.cc/channel/andrejkarpathy/explore/video/building-makemore-part-5-building-a-wavenet-910da705c2c3 markdown: >- https://liquidgold.cc/channel/andrejkarpathy/video/building-makemore-part-5-building-a-wavenet-910da705c2c3.md llms: null parent: 'https://liquidgold.cc/channel/andrejkarpathy.md' channel: andrejkarpathy updated: '2026-10-25T03:13:13.626Z' language: null children: [] videoId: t3YJ5hKiMQ0 durationSeconds: 3382 publishedAt: '2022-11-21T00:32:48.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 12 endSeconds: 101 watch: 'https://liquidgold.cc/watch/t3YJ5hKiMQ0?t=12' heldReason: rights - ordinal: 1 text: null startSeconds: 12 endSeconds: 101 watch: 'https://liquidgold.cc/watch/t3YJ5hKiMQ0?t=12' heldReason: rights - ordinal: 2 text: null startSeconds: 209 endSeconds: 289 watch: 'https://liquidgold.cc/watch/t3YJ5hKiMQ0?t=209' heldReason: rights - ordinal: 3 text: null startSeconds: 427 endSeconds: 559 watch: 'https://liquidgold.cc/watch/t3YJ5hKiMQ0?t=427' heldReason: rights - ordinal: 4 text: null startSeconds: 711 endSeconds: 844 watch: 'https://liquidgold.cc/watch/t3YJ5hKiMQ0?t=711' heldReason: rights - ordinal: 5 text: null startSeconds: 570 endSeconds: 700 watch: 'https://liquidgold.cc/watch/t3YJ5hKiMQ0?t=570' heldReason: rights - ordinal: 6 text: null startSeconds: 954 endSeconds: 1041 watch: 'https://liquidgold.cc/watch/t3YJ5hKiMQ0?t=954' heldReason: rights - ordinal: 7 text: null startSeconds: 1463 endSeconds: 1546 watch: 'https://liquidgold.cc/watch/t3YJ5hKiMQ0?t=1463' heldReason: rights - ordinal: 8 text: null startSeconds: 1860 endSeconds: 1992 watch: 'https://liquidgold.cc/watch/t3YJ5hKiMQ0?t=1860' heldReason: rights - ordinal: 9 text: null startSeconds: 2443 endSeconds: 2569 watch: 'https://liquidgold.cc/watch/t3YJ5hKiMQ0?t=2443' heldReason: rights - ordinal: 10 text: null startSeconds: 2581 endSeconds: 2725 watch: 'https://liquidgold.cc/watch/t3YJ5hKiMQ0?t=2581' heldReason: rights - ordinal: 11 text: null startSeconds: 2880 endSeconds: 3095 watch: 'https://liquidgold.cc/watch/t3YJ5hKiMQ0?t=2880' heldReason: rights - ordinal: 12 text: null startSeconds: 1206 endSeconds: 1307 watch: 'https://liquidgold.cc/watch/t3YJ5hKiMQ0?t=1206' heldReason: rights --- # Building makemore Part 5: Building a WaveNet We take the 2-layer MLP from previous video and make it deeper with a tree-like structure, arriving at a convolutional neural network architecture similar to the WaveNet (2016) from DeepMind. In the WaveNet paper, the same hierarchical architecture is implemented more efficiently using causal dilated convolutions (not yet covered). Along the way we get a better sense of torch.nn and what it is and how it works under the hood, and what a typical deep learning development process looks like (a lot of reading of documentation, keeping track of multidimensional tensor shapes, moving between jupyter notebooks and repository code, ...). Links: - makemore on github: https://github.com/karpathy/makemore - jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part5_cnn1.ipynb - collab notebook: https://colab.research.google.com/drive/1CXVEmCO_7r7WYZGb5qnjfyxTvQa13g5X?usp=sharing - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - our Discord channel: https://discord.gg/3zy8kqD9Cp Supplementary links: - WaveNet 2016 from DeepMind https://arxiv.org/abs/1609.03499 - Bengio et al. 2003 MLP LM https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf Chapters: intro 00:00:00 intro 00:01:40 starter code walkthrough 00:06:56 let’s fix the learning rate plot 00:09:16 pytorchifying our code: layers, containers, torch.nn, fun bugs implementing wavenet 00:17:11 overview: WaveNet 00:19:33 dataset bump the context size to 8 00:19:55 re-running baseline code on block_size 8 00:21:36 implementing WaveNet 00:37:41 training the WaveNet: first pass 00:38:50 fixing batchnorm1d bug 00:45:21 re-training WaveNet with bug fix 00:46:07 scaling up our WaveNet conclusions 00:46:58 experimental harness 00:47:44 WaveNet but with “dilated causal convolutions” 00:51:34 torch.nn 00:52:28 the development process of building deep neural nets 00:54:17 going forward 00:55:26 improve on my loss! how far can we improve a WaveNet on this data? Canonical: https://liquidgold.cc/channel/andrejkarpathy/explore/video/building-makemore-part-5-building-a-wavenet-910da705c2c3 · Channel: [Andrej Karpathy](https://liquidgold.cc/channel/andrejkarpathy.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.