Deep Neural Networks(DNNs) Overview

,

ㅤ

Acknowledgement: This course (CSCI 8980) is being offered by Prof. Ju Sun at the University of Minnesota in Fall 2020. Pictures of slides are from the course.

Why Deep Learning

What is Deep Learning (DL)?

DL is about…

  • Deep Neural Network (DNNs)
  • Data or training DNNs (e.g. images, videos, text sequences)
  • Methods for training DNNs (e.g. AdaGrad, ADAM, RMSProp, Dropout)
  • Hardware platforms for training DNNs (e.g. GPUs, TPUs, FPGAs)
  • Software platforms for training DNNs (e.g., Tensorflow, Pytorch, MXNet)
  • Applications! (e.g. vision, speech, NLP, imaging, physics, mathematics finance)

DL leads to many things …

Oxford Dictionary

Revolution: a great change in conditions, ways of working, beliefs, etc. that affects large numbers of people.

Academic breakthroughs

  • increasing successful rate in image classification (2012)
  • increasing successful rate in speech recognition (2012)
  • Go game (2017)
  • image generation

slide5

Commercial breakthroughs

  • self-driving vehicles
  • smart-home devices
  • healthcare
  • robotics

Paper are produced at an overwhelming rate.

Note: arvis (where people post preprint papers)

Why first principles?

  • Tuning and optimizing for a task require basic intuitions
  • Historial lesson: model structures in data
  • Current challenge: move toward trustworthiness
  • Future world: navigate uncertainties

Structures are crucial

slide14

Toward trustworthy AI

  • we need to know first principles in order to improve and understand
  • Trustworthiness:
    • robustness
    • fairness (what if data is primarily from certain population group)
    • explainability (what if we make human-level robot but we don’t understand how to make decision)
    • transparency

Future uncertainties

  • New types of data (e.g. 6-D tensors)
  • New hardware (e.g. better GPU memory)
  • New model pipelines (e.g. network of networks, differential programming)
  • New applications
  • New techniques replacing DL

Netruel Networks: Old and New

Start from Neurons

Model of biological neurons

slide4

Shallow to DNNs

Artificial Neurons

slide7
ReLU is the most popular activation function nowadays.

slide8
People play diverse forms of NN but only play with graph without cycles, because we cannot do auto differentiation once we have cycles.

Supervised Learning in ML

slide9

Supervised Learning in DL

slide10
Instead of finding a function from a family of functions, we try to find a group of weights (w) and offsets(b) to minimize the average loss.

Perceptron

Def: Perceptron is a single artificial neuron for binary classification.
slide12
slide13
slide14
slide16

They are all (shallow) NNs

  • Linear Regression
  • Perceptron and Logistic Regression
  • Softmax Regression
  • Multilayer Perceptron (feedforward NNs)
  • Support Vector Machine (SVM)
  • PCA (autoencoder)
  • Matrix Factorization

Some Background Knowledge

Turing Test

How we measure whether we achieve human-level AI?
slide20

Key ingredients of DL

slide26
Suprisingly, we’re still using those computational methods that were invented 25-30 years ago. Nowadays the progress of DL largely depends on improvement of hardware (GPUs), but seldom on new computational methods.

Question to myself: Are we approaching the third AI winter? Should we focus more on foundamental research instead of picking new applications?

To be continued...