Deep Learning

Multilayer Perceptrons

Published:

Artificial Neural Networks (ANN) are powerful classification and regression algorithms that can solve simple and complex linear and non-linear modeling problems. In this post, we demonstrate the functionality of a basic deep learning multi-layer perceptron model on PyTorch using the famous MNIST data set. In this post, we walk through the application of multilayer perceptrons.

Feature Normalization and Initialization

Published:

In this guide, we will walk through feature normalization and weight initialization schemes in PyTorch. In short, we normalize our inputs for gradient descent because large weights will dominate our updates in our attempt to find global or local minima. Separately, we use custom weight initialization schemes to improve our ability to converge during optimization or to improve our ability to use certain activation functions. In this post, we walk through the application of batch normalization and Xaviar initialization in PyTorch.

Regularization

Published:

We can think of regularization as a general means to reduce overfitting. In practice, this means that regularizing effects aim to reduce the model capacity and/or the variance of the predictions. In this post, we walk through the application of L2 regularization (manually and automatically) and dropout in PyTorch.

Optimization

Published:

Numerical optimizers provide the means to estimate our parameters by finding the values that maximize the likelihood of generating our data. This guide helps us understand how optimization algorithms find the best estimates for our coefficients in a step-by-step process. This guide uses Gradient Descent and regularization techniques in a completely manual approach to finding parameters that most likely generated our data. Another way of thinking about gradient descent is that we are inevitably asking the algorithm the following: What parameter values will push our error to zero?

Distributed Training

Published:

Spark is on open source cluster computing framework that automates the distribution of data and computations on a cluster of computers. DataBricks handles much of the architecture and cluster management for you, leveraging Jupyter style notebooks. This guide shows how to perform distributed deep learning using PyTorch on DataBricks.