---
schema: 1
type: video
title: 'Building makemore Part 2: MLP'
description: >-
  We implement a multilayer perceptron (MLP) character-level language model. In
  this video we also introduce many basics of machine learning (e.g. model
  training, learning rate tuning, hyperparameters, evaluation, train/dev/test
  splits, under/overfitting, etc.). Links: - makemore on github:
  https://github.com/karpathy/makemore - jupyter notebook I built in this video:
  https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part2_mlp.ipynb
  - collab notebook (new)!!!:
  https://colab.research.google.com/drive/1YIfmkftLrz6MPTOO9Vwqrop2Q5llHIGK?usp=sharing
  - Bengio et al. 2003 MLP language model paper (pdf):
  https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf - my website:
  https://karpathy.ai - my twitter: https://twitter.com/karpathy - (new) Neural
  Networks: Zero to Hero series Discord channel: https://discord.gg/3zy8kqD9Cp ,
  for people who'd like to chat more and go beyond youtube comments Useful
  links: - PyTorch internals ref
  http://blog.ezyang.com/2019/05/pytorch-internals/ Exercises: - E01: Tune the
  hyperparameters of the training to beat my best validation loss of 2.2 - E02:
  I was not careful with the intialization of the network in this video. (1)
  What is the loss you'd get if the predicted probabilities at initialization
  were perfectly uniform? What loss do we achieve? (2) Can you tune the
  initialization to get a starting loss that is much more similar to (1)? - E03:
  Read the Bengio et al 2003 paper (link above), implement and try any idea from
  the paper. Did it work? Chapters: 00:00:00 intro 00:01:48 Bengio et al. 2003
  (MLP language model) paper walkthrough 00:09:03 (re-)building our training
  dataset 00:12:19 implementing the embedding lookup table 00:18:35 implementing
  the hidden layer + internals of torch.Tensor: storage, views 00:29:15
  implementing the output layer 00:29:53 implementing the negative log
  likelihood loss 00:32:17 summary of the full network 00:32:49 introducing
  F.cross_entropy and why 00:37:56 implementing the training loop, overfitting
  one batch 00:41:25 training on the full dataset, minibatches 00:45:40 finding
  a good initial learning rate 00:53:20 splitting up the dataset into
  train/val/test splits and why 01:00:49 experiment: larger hidden layer
  01:05:27 visualizing the character embeddings 01:07:16 experiment: larger
  embedding size 01:11:46 summary of our final code, conclusion 01:13:24
  sampling from the model 01:14:55 google collab (new!!) notebook advertisement
canonical: >-
  https://liquidgold.cc/channel/andrejkarpathy/explore/video/building-makemore-part-2-mlp-d7ec91efb0b5
markdown: >-
  https://liquidgold.cc/channel/andrejkarpathy/video/building-makemore-part-2-mlp-d7ec91efb0b5.md
llms: null
parent: 'https://liquidgold.cc/channel/andrejkarpathy.md'
channel: andrejkarpathy
updated: '2026-10-25T03:13:13.626Z'
language: null
children: []
videoId: TCH_1BHY58I
durationSeconds: 4540
publishedAt: '2022-09-12T14:43:06.000Z'
searchChunks:
  - ordinal: 0
    text: null
    startSeconds: 0
    endSeconds: 66
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=0'
    heldReason: rights
  - ordinal: 1
    text: null
    startSeconds: 66
    endSeconds: 113
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=66'
    heldReason: rights
  - ordinal: 2
    text: null
    startSeconds: 113
    endSeconds: 158
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=113'
    heldReason: rights
  - ordinal: 3
    text: null
    startSeconds: 158
    endSeconds: 212
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=158'
    heldReason: rights
  - ordinal: 4
    text: null
    startSeconds: 212
    endSeconds: 256
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=212'
    heldReason: rights
  - ordinal: 5
    text: null
    startSeconds: 289
    endSeconds: 334
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=289'
    heldReason: rights
  - ordinal: 6
    text: null
    startSeconds: 368
    endSeconds: 422
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=368'
    heldReason: rights
  - ordinal: 7
    text: null
    startSeconds: 422
    endSeconds: 474
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=422'
    heldReason: rights
  - ordinal: 8
    text: null
    startSeconds: 474
    endSeconds: 517
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=474'
    heldReason: rights
  - ordinal: 9
    text: null
    startSeconds: 517
    endSeconds: 550
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=517'
    heldReason: rights
  - ordinal: 10
    text: null
    startSeconds: 550
    endSeconds: 606
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=550'
    heldReason: rights
  - ordinal: 11
    text: null
    startSeconds: 743
    endSeconds: 787
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=743'
    heldReason: rights
  - ordinal: 12
    text: null
    startSeconds: 906
    endSeconds: 972
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=906'
    heldReason: rights
  - ordinal: 13
    text: null
    startSeconds: 972
    endSeconds: 1027
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=972'
    heldReason: rights
  - ordinal: 14
    text: null
    startSeconds: 1117
    endSeconds: 1151
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=1117'
    heldReason: rights
  - ordinal: 15
    text: null
    startSeconds: 1280
    endSeconds: 1324
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=1280'
    heldReason: rights
  - ordinal: 16
    text: null
    startSeconds: 1357
    endSeconds: 1401
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=1357'
    heldReason: rights
  - ordinal: 17
    text: null
    startSeconds: 1456
    endSeconds: 1488
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=1456'
    heldReason: rights
  - ordinal: 18
    text: null
    startSeconds: 1565
    endSeconds: 1609
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=1565'
    heldReason: rights
  - ordinal: 19
    text: null
    startSeconds: 1710
    endSeconds: 1757
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=1710'
    heldReason: rights
  - ordinal: 20
    text: null
    startSeconds: 1757
    endSeconds: 1803
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=1757'
    heldReason: rights
  - ordinal: 21
    text: null
    startSeconds: 1970
    endSeconds: 2014
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=1970'
    heldReason: rights
  - ordinal: 22
    text: null
    startSeconds: 2201
    endSeconds: 2257
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=2201'
    heldReason: rights
  - ordinal: 23
    text: null
    startSeconds: 2334
    endSeconds: 2400
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=2334'
    heldReason: rights
  - ordinal: 24
    text: null
    startSeconds: 2635
    endSeconds: 2678
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=2635'
    heldReason: rights
  - ordinal: 25
    text: null
    startSeconds: 2938
    endSeconds: 3025
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=2938'
    heldReason: rights
  - ordinal: 26
    text: null
    startSeconds: 3118
    endSeconds: 3162
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=3118'
    heldReason: rights
  - ordinal: 27
    text: null
    startSeconds: 3274
    endSeconds: 3319
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=3274'
    heldReason: rights
  - ordinal: 28
    text: null
    startSeconds: 3619
    endSeconds: 3651
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=3619'
    heldReason: rights
  - ordinal: 29
    text: null
    startSeconds: 3958
    endSeconds: 4013
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=3958'
    heldReason: rights
  - ordinal: 30
    text: null
    startSeconds: 4211
    endSeconds: 4277
    watch: 'https://liquidgold.cc/watch/TCH_1BHY58I?t=4211'
    heldReason: rights
---


# Building makemore Part 2: MLP

We implement a multilayer perceptron (MLP) character-level language model. In this video we also introduce many basics of machine learning (e.g. model training, learning rate tuning, hyperparameters, evaluation, train/dev/test splits, under/overfitting, etc.). Links: - makemore on github: https://github.com/karpathy/makemore - jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part2_mlp.ipynb - collab notebook (new)!!!: https://colab.research.google.com/drive/1YIfmkftLrz6MPTOO9Vwqrop2Q5llHIGK?usp=sharing - Bengio et al. 2003 MLP language model paper (pdf): https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - (new) Neural Networks: Zero to Hero series Discord channel: https://discord.gg/3zy8kqD9Cp , for people who'd like to chat more and go beyond youtube comments Useful links: - PyTorch internals ref http://blog.ezyang.com/2019/05/pytorch-internals/ Exercises: - E01: Tune the hyperparameters of the training to beat my best validation loss of 2.2 - E02: I was not careful with the intialization of the network in this video. (1) What is the loss you'd get if the predicted probabilities at initialization were perfectly uniform? What loss do we achieve? (2) Can you tune the initialization to get a starting loss that is much more similar to (1)? - E03: Read the Bengio et al 2003 paper (link above), implement and try any idea from the paper. Did it work? Chapters: 00:00:00 intro 00:01:48 Bengio et al. 2003 (MLP language model) paper walkthrough 00:09:03 (re-)building our training dataset 00:12:19 implementing the embedding lookup table 00:18:35 implementing the hidden layer + internals of torch.Tensor: storage, views 00:29:15 implementing the output layer 00:29:53 implementing the negative log likelihood loss 00:32:17 summary of the full network 00:32:49 introducing F.cross_entropy and why 00:37:56 implementing the training loop, overfitting one batch 00:41:25 training on the full dataset, minibatches 00:45:40 finding a good initial learning rate 00:53:20 splitting up the dataset into train/val/test splits and why 01:00:49 experiment: larger hidden layer 01:05:27 visualizing the character embeddings 01:07:16 experiment: larger embedding size 01:11:46 summary of our final code, conclusion 01:13:24 sampling from the model 01:14:55 google collab (new!!) notebook advertisement

Canonical: https://liquidgold.cc/channel/andrejkarpathy/explore/video/building-makemore-part-2-mlp-d7ec91efb0b5 · Channel: [Andrej Karpathy](https://liquidgold.cc/channel/andrejkarpathy.md)

> Search chunk text is held until public-source provenance is verified.

## Document chunks

---

#### Chunk 0: Document excerpt (0:00 to 1:53)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 1: Document excerpt (1:53 to 3:32)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 2: Document excerpt (3:32 to 4:16)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 3: Document excerpt (4:16 to 6:08)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 4: Document excerpt (6:08 to 7:54)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 5: Document excerpt (7:54 to 8:37)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 6: Document excerpt (8:37 to 9:10)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 7: Document excerpt (9:10 to 10:06)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 8: Document excerpt (10:06 to 11:25)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 9: Document excerpt (11:25 to 12:23)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 10: Document excerpt (12:23 to 14:03)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 11: Document excerpt (14:03 to 16:12)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 12: Document excerpt (16:12 to 18:37)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 13: Document excerpt (18:37 to 20:36)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 14: Document excerpt (20:36 to 22:37)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 15: Document excerpt (22:37 to 24:16)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 16: Document excerpt (24:16 to 26:05)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 17: Document excerpt (26:05 to 27:55)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 18: Document excerpt (27:55 to 28:30)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 19: Document excerpt (28:30 to 29:17)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 20: Document excerpt (29:17 to 30:03)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 21: Document excerpt (30:03 to 31:46)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 22: Document excerpt (31:46 to 32:18)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 23: Document excerpt (32:18 to 32:50)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 24: Document excerpt (32:50 to 34:31)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 25: Document excerpt (34:31 to 36:08)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 26: Document excerpt (36:08 to 37:59)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 27: Document excerpt (37:59 to 40:00)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 28: Document excerpt (40:00 to 41:29)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 29: Document excerpt (41:29 to 42:37)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 30: Document excerpt (42:37 to 45:22)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 31: Document excerpt (45:22 to 45:46)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 32: Document excerpt (45:46 to 47:19)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 33: Document excerpt (47:19 to 48:58)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 34: Document excerpt (48:58 to 50:59)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 35: Document excerpt (50:59 to 53:14)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 36: Document excerpt (53:14 to 53:26)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 37: Document excerpt (53:26 to 55:19)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 38: Document excerpt (55:19 to 57:18)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 39: Document excerpt (57:18 to 59:24)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 40: Document excerpt (59:24 to 60:51)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 41: Document excerpt (60:51 to 63:08)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 42: Document excerpt (63:08 to 65:35)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 43: Document excerpt (65:35 to 67:16)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 44: Document excerpt (67:16 to 69:46)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 45: Document excerpt (69:46 to 71:17)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 46: Document excerpt (71:17 to 71:50)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 47: Document excerpt (71:50 to 73:25)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 48: Document excerpt (73:25 to 74:55)

> Held: transcript rights do not permit this chunk's text.

---

#### Chunk 49: Document excerpt (74:55 to 75:40)

> Held: transcript rights do not permit this chunk's text.