Neural Networks and Deep Learning

What this book is about

On the exercises and problems

Using neural nets to recognize handwritten digits

• Perceptrons
• Sigmoid neurons
• The architecture of neural networks
• A simple network to classify handwritten digits
• Learning with gradient descent
• Implementing our network to classify digits
• Toward deep learning

How the backpropagation algorithm works

• Warm up: a fast matrix-based approach to computing the output from a neural network
• The two assumptions we need about the cost function
• The Hadamard product, $s \odot t$
• The four fundamental equations behind backpropagation
• Proof of the four fundamental equations (optional)
• The backpropagation algorithm
• The code for backpropagation
• In what sense is backpropagation a fast algorithm?
• Backpropagation: the big picture

Improving the way neural networks learn

• The cross-entropy cost function
• Overfitting and regularization
• Weight initialization
• Handwriting recognition revisited: the code
• How to choose a neural network's hyper-parameters?
• Other techniques

A visual proof that neural nets can compute any function

• Two caveats
• Universality with one input and one output
• Many input variables
• Extension beyond sigmoid neurons
• Fixing up the step functions
• Conclusion

Why are deep neural networks hard to train?

• The vanishing gradient problem
• What's causing the vanishing gradient problem? Unstable gradients in deep neural nets
• Unstable gradients in more complex networks
• Other obstacles to deep learning

Deep learning

• Introducing convolutional networks
• Convolutional neural networks in practice
• The code for our convolutional networks
• Recent progress in image recognition
• Other approaches to deep neural nets
• On the future of neural networks

Appendix: Is there a simple algorithm for intelligence?

Acknowledgements

Frequently Asked Questions

---

If you benefit from the book, please make a small donation. I suggest $5, but you can choose the amount.

Alternately, you can make a donation by sending me Bitcoin, at address 1Kd6tXH5SDAmiFb49J9hknG5pqj7KStSAx

---

Sponsors

Deep Learning Workstations, Servers, and Laptops

Thanks to all the supporters who made the book possible, with especial thanks to Pavel Dudrenov. Thanks also to all the contributors to the Bugfinder Hall of Fame.

---

Resources

Michael Nielsen on Twitter

Book FAQ

Code repository

Michael Nielsen's project announcement mailing list

Deep Learning, book by Ian Goodfellow, Yoshua Bengio, and Aaron Courville

cognitivemedium.com

---

By Michael Nielsen / Dec 2019

Thankyou to the following people for their help in finding and eliminating bugs and other errors in the book:

Kartik Agaram, Martin Aulbach, Joseph Barrow, Maxim Baz, Ingo Blechschmidt, Charles Brauer, Greg Brockman, Cristi Burcă, Josue Ortega Caro, Gildas Chabot, Jen Dodd, Philipp Düren, Chris Ferrie, Tim Gowers, Timo Hannay, Vojtěch Havlíček, Nikola Henezi, Jay Huie, Michael Lennon, Hassan Masum, John McNally, pyos, Juergen Schmidhuber, Sten Sootla, Sharad Sundararajan, D. Sivakumar, Jeremy Vonderfecht, and Nico Weber.