DGY

Learning Deep Learning

Introduction

This reading list is an on-going personal challenge I’m about to start again. The challenge is to digest one paper a day in the morning (a.k.a eating a paper as breakfast). Over time I found that one can really really read a lot in an hour. And it’s a perfect way of starting a new day with some new interesting knowledge.

In general,

The old version of this post is now hosted at here. I will preserve this legacy version only for lookup reasons.

For Novice

If you have no idea about Machine Learning and Scientific Computing, I suggest you learn the following materials while you are reading Machine Learning or Deep Learning books. You don’t have to master these materials, but a basic understanding is essential. It’s hard to open a meaningful conversation if the person has no idea about matrix or single variable calculus.

Title Author or Source Tags
Introduction to Algorithms Erik Demaine and Srinivas Devadas  
Single Variable Calculus David Jerison  
Multivariable Calculus Denis Auroux  
Differential Equations Arthur Mattuck, Haynes Miller, Jeremy Orloff, John Lewis  
Linear Algebra Gilbert Strang  

Theory of Computation, Learning Theory, Neuroscience, etc

Title Author or Source Tags
Introduction to the Theory of Computation Michael Sipser  
Artificial Intelligence: A Modern Approach Stuart Russell and Peter Norvig  
Pattern Recognition and Machine Learning Christopher Bishop  
Machine Learning: A probabilistic perspective Kevin Patrick Murphy  
CS229 Machine Learning Course Materials Andrew Ng at Stanford University  
Reinforcement Learning: An Introduction Richard S. Sutton and Andrew G. Barto  
Probabilistic Graphical Models: Principles and Techniques Daphne Koller and Nir Friedman  
Convex Optimization Stephen Boyd and Lieven Vandenberghe  
An Introduction to Statistical Learning with application in R Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani  
Neuronal Dynamics: From single neurons to networks and models of cognition Wulfram Gerstner, Werner M. Kistler, Richard Naud and Liam Paninski  
Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems Peter Dayan and Laurence F. Abbott  
Michael I. Jordan Reading List of Machine Learning Hacker News  

Fundamentals of Deep Learning

Title Author or Source Tags
Deep Learning in Neural Networks: An Overview Jürgen Schmidhuber  
Deep Learning Book Yoshua Bengio, Ian Goodfellow and Aaron Courville  
Learning Deep Architectures for AI Yoshua Bengio  
Representation Learning: A Review and New Perspectives Yoshua Bengio, Aaron Courville, Pascal Vincent  
Reading lists for new MILA students MILA Lab, University of Montreal  
Tutorial on Variational Autoencoders Carl Doersch  

Tutorials, Practical Guides, and Useful Software

Title Author or Source Tags
Machine Learning Andrew Ng  
Neural Networks for Machine Learning Geoffrey Hinton  
Deep Learning Tutorial LISA Lab, University of Montreal  
Unsupervised Feature Learning and Deep Learning Tutorial AI Lab, Stanford University  
CS231n: Convolutional Neural Networks for Visual Recognition Stanford University  
CS224d: Deep Learning for Natural Language Processing Stanford University  
Theano LISA Lab, University of Montreal  
PyLearn2 LISA Lab, University of Montreal  
Caffe Berkeley Vision and Learning Center (BVLC) and community contributor Yangqing Jia  
Torch 7 active contributors  
neon Nervana  
cuDNN NVIDIA  
ConvNetJS Andrej Karpathy  
DeepLearning4j    
Chainer: Neural network framework Preferred Networks, Inc  
Blocks LISA Lab, University of Montreal  
Fuel LISA Lab, University of Montreal  
Brainstorm IDSIA, Switzerland  
Keras fchollet and active contributors  
Lasagne Lasagne  

Literature in Deep Learning and Feature Learning

Deep Learning is a fast-moving community. Therefore the line between “Recent Advances” and “Literature that matter” is kind of blurred. Here I collected articles that are either introducing fundamental algorithms, techniques or highly cited by the community.

Title Author or Source Tags
Automatic Speech Recognition - A Deep Learning Approach Dong Yu and Li Deng (Published by Springer, no Open Access)  
Backpropagation Applied to Handwritten Zip Code Recognition Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard and L. D. Jackel  
Comparison of Training Methods for Deep Neural Networks Patrick O. Glauner  
Deep Learning Yann LeCun, Yoshua Bengio, Geoffrey Hinton. (NO FREE COPY AVAILABLE)  
Distributed Representations of Words and Phrases and their Compositionality Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado and Jeffrey Dean  
Efficient Estimation of Word Representations in Vector Space Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean  
Efficient Large Scale Video Classification Balakrishnan Varadarajan, George Toderici, Sudheendra Vijayanarasimhan, Apostol Natsev  
Foundations and Trends in Signal Processing: DEEP LEARNING — Methods and Applications Li Deng and Dong Yu  
From Frequency to Meaning: Vector Space Models of Semantics Peter D. Turney and Patrick Pantel  
LSTM: A Search Space Odyssey Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, Jürgen Schmidhuber  
Supervised Sequence Labelling with Recurrent Neural Networks Alex Graves  

Recent Must-Read Advances in Deep Learning

Contacts and Suggestions

Yuhuang Hu
Email: duguyue100@gmail.com