---
schema: 1
type: video
title: 'Building makemore Part 3: Activations & Gradients, BatchNorm'
description: >-
  We dive into some of the internals of MLPs with multiple layers and scrutinize
  the statistics of the forward pass activations, backward pass gradients, and
  some of the pitfalls when they are improperly scaled. We also look at the
  typical diagnostic tools and visualizations you'd want to use to understand
  the health of your deep network. We learn why training deep neural nets can be
  fragile and introduce the first modern innovation that made doing so much
  easier: Batch Normalization. Residual connections and the Adam optimizer
  remain notable todos for later video. Links: - makemore on github:
  https://github.com/karpathy/makemore - jupyter notebook I built in this video:
  https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part3_bn.ipynb
  - collab notebook:
  https://colab.research.google.com/drive/1H5CSy-OnisagUgDUXhHwo1ng2pjKHYSN?usp=sharing
  - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy -
  Discord channel: https://discord.gg/3zy8kqD9Cp Useful links: - "Kaiming init"
  paper: https://arxiv.org/abs/1502.01852 - BatchNorm paper:
  https://arxiv.org/abs/1502.03167 - Bengio et al. 2003 MLP language model paper
  (pdf): https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf - Good
  paper illustrating some of the problems with batchnorm in practice:
  https://arxiv.org/abs/2105.07576 Exercises: - E01: I did not get around to
  seeing what happens when you initialize all weights and biases to zero. Try
  this and train the neural net. You might think either that 1) the network
  trains just fine or 2) the network doesn't train at all, but actually it is 3)
  the network trains but only partially, and achieves a pretty bad final
  performance. Inspect the gradients and activations to figure out what is
  happening and why the network is only partially training, and what part is
  being trained exactly. - E02: BatchNorm, unlike other normalization layers
  like LayerNorm/GroupNorm etc. has the big advantage that after training, the
  batchnorm gamma/beta can be "folded into" the weights of the preceeding Linear
  layers, effectively erasing the need to forward it at test time. Set up a
  small 3-layer MLP with batchnorms, train the network, then "fold" the
  batchnorm gamma/beta into the preceeding Linear layer's W,b by creating a new
  W2, b2 and erasing the batch norm. Verify that this gives the same forward
  pass during inference. i.e. we see that the batchnorm is there just for
  stabilizing the training, and can be thrown out after training is done! pretty
  cool. Chapters: 00:00:00 intro 00:01:22 starter code 00:04:19 fixing the
  initial loss 00:12:59 fixing the saturated tanh 00:27:53 calculating the init
  scale: “Kaiming init” 00:40:40 batch normalization 01:03:07 batch
  normalization: summary 01:04:50 real example: resnet50 walkthrough 01:14:10
  summary of the lecture 01:18:35 just kidding: part2: PyTorch-ifying the code
  01:26:51 viz #1: forward pass activations statistics 01:30:54 viz #2: backward
  pass gradient statistics 01:32:07 the fully linear case of no non-linearities
  01:36:15 viz #3: parameter activation and gradient statistics 01:39:55 viz #4:
  update:data ratio over time 01:46:04 bringing back batchnorm, looking at the
  visualizations 01:51:34 summary of the lecture for real this time
canonical: 'https://liquidgold.cc/watch/P6sfmUTpUmc'
markdown: 'https://liquidgold.cc/watch/P6sfmUTpUmc.md'
llms: 'https://liquidgold.cc/watch/P6sfmUTpUmc/llms.txt'
parent: 'https://liquidgold.cc/channel/andrejkarpathy.md'
channel: andrejkarpathy
updated: '2026-10-25T03:13:13.626Z'
language: null
children: []
videoId: P6sfmUTpUmc
durationSeconds: 6958
publishedAt: '2022-10-04T16:41:03.000Z'
searchChunks:
  - ordinal: 0
    text: null
    startSeconds: 269
    endSeconds: 369
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=269'
    heldReason: rights
  - ordinal: 1
    text: null
    startSeconds: 474
    endSeconds: 585
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=474'
    heldReason: rights
  - ordinal: 2
    text: null
    startSeconds: 596
    endSeconds: 675
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=596'
    heldReason: rights
  - ordinal: 3
    text: null
    startSeconds: 686
    endSeconds: 782
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=686'
    heldReason: rights
  - ordinal: 4
    text: null
    startSeconds: 889
    endSeconds: 999
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=889'
    heldReason: rights
  - ordinal: 5
    text: null
    startSeconds: 1110
    endSeconds: 1221
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=1110'
    heldReason: rights
  - ordinal: 6
    text: null
    startSeconds: 1233
    endSeconds: 1358
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=1233'
    heldReason: rights
  - ordinal: 7
    text: null
    startSeconds: 1463
    endSeconds: 1544
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=1463'
    heldReason: rights
  - ordinal: 8
    text: null
    startSeconds: 1688
    endSeconds: 1821
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=1688'
    heldReason: rights
  - ordinal: 9
    text: null
    startSeconds: 1832
    endSeconds: 1943
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=1832'
    heldReason: rights
  - ordinal: 10
    text: null
    startSeconds: 1954
    endSeconds: 2041
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=1954'
    heldReason: rights
  - ordinal: 11
    text: null
    startSeconds: 2052
    endSeconds: 2163
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=2052'
    heldReason: rights
  - ordinal: 12
    text: null
    startSeconds: 2174
    endSeconds: 2275
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=2174'
    heldReason: rights
  - ordinal: 13
    text: null
    startSeconds: 2456
    endSeconds: 2572
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=2456'
    heldReason: rights
  - ordinal: 14
    text: null
    startSeconds: 2584
    endSeconds: 2669
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=2584'
    heldReason: rights
  - ordinal: 15
    text: null
    startSeconds: 2682
    endSeconds: 2769
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=2682'
    heldReason: rights
  - ordinal: 16
    text: null
    startSeconds: 3040
    endSeconds: 3130
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=3040'
    heldReason: rights
  - ordinal: 17
    text: null
    startSeconds: 3130
    endSeconds: 3141
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=3130'
    heldReason: rights
  - ordinal: 18
    text: null
    startSeconds: 3253
    endSeconds: 3344
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=3253'
    heldReason: rights
  - ordinal: 19
    text: null
    startSeconds: 3445
    endSeconds: 3537
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=3445'
    heldReason: rights
  - ordinal: 20
    text: null
    startSeconds: 3649
    endSeconds: 3793
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=3649'
    heldReason: rights
  - ordinal: 21
    text: null
    startSeconds: 3804
    endSeconds: 3895
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=3804'
    heldReason: rights
  - ordinal: 22
    text: null
    startSeconds: 4010
    endSeconds: 4144
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=4010'
    heldReason: rights
  - ordinal: 23
    text: null
    startSeconds: 4156
    endSeconds: 4236
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=4156'
    heldReason: rights
  - ordinal: 24
    text: null
    startSeconds: 4246
    endSeconds: 4346
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=4246'
    heldReason: rights
  - ordinal: 25
    text: null
    startSeconds: 4346
    endSeconds: 4357
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=4346'
    heldReason: rights
  - ordinal: 26
    text: null
    startSeconds: 4920
    endSeconds: 4930
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=4920'
    heldReason: rights
  - ordinal: 27
    text: null
    startSeconds: 4930
    endSeconds: 5011
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=4930'
    heldReason: rights
  - ordinal: 28
    text: null
    startSeconds: 5225
    endSeconds: 5457
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=5225'
    heldReason: rights
  - ordinal: 29
    text: null
    startSeconds: 5542
    endSeconds: 5777
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=5542'
    heldReason: rights
  - ordinal: 30
    text: null
    startSeconds: 5788
    endSeconds: 5995
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=5788'
    heldReason: rights
  - ordinal: 31
    text: null
    startSeconds: 6147
    endSeconds: 6236
    watch: 'https://liquidgold.cc/watch/P6sfmUTpUmc?t=6147'
    heldReason: rights
---


# Building makemore Part 3: Activations & Gradients, BatchNorm

We dive into some of the internals of MLPs with multiple layers and scrutinize the statistics of the forward pass activations, backward pass gradients, and some of the pitfalls when they are improperly scaled. We also look at the typical diagnostic tools and visualizations you'd want to use to understand the health of your deep network. We learn why training deep neural nets can be fragile and introduce the first modern innovation that made doing so much easier: Batch Normalization. Residual connections and the Adam optimizer remain notable todos for later video. Links: - makemore on github: https://github.com/karpathy/makemore - jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part3_bn.ipynb - collab notebook: https://colab.research.google.com/drive/1H5CSy-OnisagUgDUXhHwo1ng2pjKHYSN?usp=sharing - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - Discord channel: https://discord.gg/3zy8kqD9Cp Useful links: - "Kaiming init" paper: https://arxiv.org/abs/1502.01852 - BatchNorm paper: https://arxiv.org/abs/1502.03167 - Bengio et al. 2003 MLP language model paper (pdf): https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf - Good paper illustrating some of the problems with batchnorm in practice: https://arxiv.org/abs/2105.07576 Exercises: - E01: I did not get around to seeing what happens when you initialize all weights and biases to zero. Try this and train the neural net. You might think either that 1) the network trains just fine or 2) the network doesn't train at all, but actually it is 3) the network trains but only partially, and achieves a pretty bad final performance. Inspect the gradients and activations to figure out what is happening and why the network is only partially training, and what part is being trained exactly. - E02: BatchNorm, unlike other normalization layers like LayerNorm/GroupNorm etc. has the big advantage that after training, the batchnorm gamma/beta can be "folded into" the weights of the preceeding Linear layers, effectively erasing the need to forward it at test time. Set up a small 3-layer MLP with batchnorms, train the network, then "fold" the batchnorm gamma/beta into the preceeding Linear layer's W,b by creating a new W2, b2 and erasing the batch norm. Verify that this gives the same forward pass during inference. i.e. we see that the batchnorm is there just for stabilizing the training, and can be thrown out after training is done! pretty cool. Chapters: 00:00:00 intro 00:01:22 starter code 00:04:19 fixing the initial loss 00:12:59 fixing the saturated tanh 00:27:53 calculating the init scale: “Kaiming init” 00:40:40 batch normalization 01:03:07 batch normalization: summary 01:04:50 real example: resnet50 walkthrough 01:14:10 summary of the lecture 01:18:35 just kidding: part2: PyTorch-ifying the code 01:26:51 viz #1: forward pass activations statistics 01:30:54 viz #2: backward pass gradient statistics 01:32:07 the fully linear case of no non-linearities 01:36:15 viz #3: parameter activation and gradient statistics 01:39:55 viz #4: update:data ratio over time 01:46:04 bringing back batchnorm, looking at the visualizations 01:51:34 summary of the lecture for real this time

Canonical: https://liquidgold.cc/watch/P6sfmUTpUmc · Channel: [Andrej Karpathy](https://liquidgold.cc/channel/andrejkarpathy.md)

> Search chunk text is held until public-source provenance is verified.

## Document chunks

> Missing: no stored document chunks are available.