Advance Deep Learning & Artificial Intelligence Program

1. Mathematical Foundations of Data Science:

A. Linear Algebra:

1. Vectors, Matrices
2. Tensors
3. Matrix Operations
4. Projections
5. Eigenvalue decomposition of a matrix
6. LU Decomposition
7. QR Decomposition/Factorization
8. Symmetric Matrices
9. Orthogonalization & Orthonormalization
10. Real and Complex Analysis (Sets and Sequences, Topology, Metric Spaces, Single-Valued and Continuous Functions, Limits, Cauchy Kernel, Fourier Transforms)
11. Information Theory (Entropy, Information Gain)
12. Function Spaces and Manifolds
13. Relational Algebra and SQL

B. Multivariate Calculus

1. Differential and Integral Calculus
2. Partial Derivatives
3. Vector-Values Functions
4. Directional Gradient
5. Hessian
6. Jacobian
7. Laplacian and Lagragian Distribution

2: Probability for Data Scientists

1. Probability Theory and Statistics
2. Combinatorics
3. Random Variables
4. Probability Rules & Axioms
5. Bayes' Theorem
6. Variance and Expectation
7. Conditional and Joint Distributions
8. Standard Distributions (Bernoulli, Binomial, Multinomial, Uniform and Gaussian)
9. Moment Generating Functions
10. Maximum Likelihood Estimation (MLE)
11. Prior and Posterior
12. Maximum a Posteriori Estimation (MAP) and Sampling Methods
13. Descriptive Statistics
14. Hypothesis Testing
15. Goodness of Fit
16. Analysis of Variance
17. Correlation
18. Chi2 test
19. Design of Experiments

3: Algorithms and Data Structures:

A. Graph Theory: Basic Concepts and Algorithms
B. Algorithmic Complexity

1. Algorithm Analysis
2. Greedy Algorithms
3. Divide and Conquer and Dynamic Programming

C. Data Structures

1. Array, List, Hashing, Binary Trees, Hashing, Heap, Stack etc
2. Dynamic Programming
3. Randomized & Sublinear Algorithm
4. Graphs

4: R and Python

A. R Programming Language

1. Vectors
2. Matrices
3. Lists
4. Data frame
5. Basic Syntax
6. Basic Statistics
7. Data Manipulation (dplyr)
8. Visualization (ggplot2)
9. Connecting to databases (RJDBC)

Python Programming Language

1. Python language fundamentals
2. Data Structures
3. Beautiful Soup
4. Regular Expressions
5. JSON
6. Restful Web Services (Flask)
7. NumPy
8. Plots in matplotlib, seaborn
9. Pandas

In this course, DankoNikolic brain scientists and AI inventor explains the fundamental of intelligence needed for everyone interested in creating ambitious AI solutions. What you do when things get tough? In the course, you will learn the differences between machine intelligence and human intelligence. You will understand why AI fails and when. AI does not have a narrowly limited working memory (a.k.a., short-term memory) but we humans do. How does our working memory make us more intelligent than machines? Why do we understand the world and machines don't? You will also learn fundamental theorems for machine learning and see how they apply to machine intelligence and human intelligence. After having learned that, you will be able to judge whether an ML project is too ambitious or is likely to succeed. You will be able to identify fundamental problems that plagued some of the ambitious AI projects in the past. You will understand why it is nearly impossible for machines to reach human levels of intelligence. Also, you will learn why some of the tricks in machine learning sometimes work and other times not. You will understand why it is so difficult to build self-driving cars.

The course offers fundamentals that you cannot find in any other course or a book. These fundamentals will be invaluable for your future work on ML and AI.

1. This course explores the nature of intelligence, ranging from machines to the biological brain. Information provided in the course is useful when undergoing ambitious projects in machine learning and AI. It will help you avoid pitfalls in those projects.
2. What are the differences between the real brain and machine intelligence and how can you use this knowledge to prevent failures in your work? What are the limits of today's AI technology? How to assess early in your AI project whether it has chances of success?
3. What are the most fundamental mathematical theorems in machine learning and how they are relevant for your everyday work?

UNIT 1:

1. Introduction to deep learning
2. Course logistics
3. History and cognitive basis of neural computation
4. The perceptron / multi-layer perceptron

UNIT 2:

1. The neural net as a universal approximator

UNIT 3A:

1. Training a neural network
2. Perceptron learning rule
3. Empirical Risk Minimization
4. Optimization by gradient descent

UNIT 3B:

1. Single hidden layered networks and universal approximation
2. Motivation for more than 1 hidden layer
3. Feature engineering versus co-learning features and estimation

UNIT 4:

1. Back propagation
2. Calculus of back propagation

UNIT 5:

1. Convergence in neural networks
2. Rates of convergence
3. Loss surfaces, Learning rates, and optimization methods
4. RMSProp, Adagrad, Momentum

UNIT 6:

1. Stochastic gradient descent
2. Acceleration
3. Overfitting and regularization
4. Tricks of the trade:
5. Choosing a divergence (loss) function
6. Batch normalization
7. Dropout

UNIT 7:

1. Recap of Training Q&A session for students

UNIT 8:

1. Optimization continued

UNIT 9A:

1. Models of vision
2. Neocognitron
3. Mathematical details of CNNs, Alexnet, Inception, VGG

UNIT 9 B:

1. Architecture
2. Convolution, Pooling, Normalization
3. Training strategies
4. Visualizing and understanding convolutional networks
5. Some MORE well-known convolution networks (LeNet/ZFNet/GoogleNet)

UNIT 10:

1. Recurrent Neural Networks (RNNs)
2. Modeling series
3. Back propogation through time
4. Bidirectional RNNs

UNIT 11:

1. Stability
2. Exploding/vanishing gradients
3. Long Short-Term Memory Units (LSTMs) and variants
4. Resnets

UNIT 12:

1. Loss functions for recurrent networks
2. Sequence Prediction

UNIT 13:

1. Sequence To Sequence Methods
2. Connectionist Temporal Classification (CTC)

UNIT 14:

1. Sequence-to-sequence models
2. Attention models, examples from speech and language

UNIT 15:

1. What to networks represent
2. Autoencoders and dimensionality reduction
3. Learning representations

UNIT 16:

1. Variational Autoencoders (VAEs)

UNIT 17:

1. Generative Adversarial Networks (GANs) Part 1

UNIT 18:

1. Generative Adversarial Networks (GANs) Part 2

UNIT 19:

1. Regularization

UNIT 20:

1. Transfer Learning

UNIT 21:

1. Hopfield Networks
2. Boltzmann Machines

UNIT 22:

1. Training Hopfield Networks
2. Stochastic Hopfield Networks

UNIT 23:

1. Restricted Boltzmann Machines
2. Deep Boltzmann Machines

UNIT 24:

1. Reinforcement Learning 1

UNIT 25:

1. Reinforcement Learning 2

UNIT 26:

1. Reinforcement Learning 3

UNIT 27:

1. Reinforcement Learning 4

UNIT 28:

1. Q Learning Deep Q Learning

1. Case Study:Computer Vision (auto-colorization of b/w images; attention)
2. Case Study:Natural Language Processing (caption generation, Word2Vec)

1. Image Processing:
   • Projects: Medical image analysis -- pathology, tracking suspicious movement for security.
2. Natural Language Processing:
   • Projects: Spam detector, Q&A/Conversational systems.
3. Speech:
   • Project: Conversational systems.
4. Deep Learning in Cyber Security

1. Signal Processing

• Fourier transform, Short-time Fourier Transform
• Filtering

2. Speech Processing

• Mel-frequency cepstral coefficients
• Perceptual linear prediction
• Probabilistic linear discriminant analysis

3. Image processing

• Edge detecting and linking
• Texture
• Morphological features
• Scale-invariant feature transform
• Histogram of Gaussians f. Color spaces

1. Statistical Language Modeling
2. Computational Linguistics
3. Statistical Decision Making and the Source-Channel Paradigm
4. Sparseness; Smoothing
5. Measuring Success: Information Theory, Entropy and Perplexity Maximum Entropy Models, Whole-Sentence Models, Semantic Modeling
6. EM for sound separation
7. Probabilistic Context Free Grammars (PCFG), the Inside-Outside Algorithm
8. Syntactic Language Models
9. Decision Tree Language Models

1. Introduction

• Introduction to Neural Networks
• Example Tasks and Their Difficulties
• What Neural Nets Can Do To Help

2. Predicting the Next Word in a Sentence

• Computational Graphs
• Feed-forward Neural Network Language Models
• Measuring Model Performance: Likelihood and Perplexity

3. Distributional Semantics and Word Vectors

• Describing a word by the company that it keeps
• Counting and predicting
• Skip-grams and CBOW
• Evaluating/Visualizing Word Vectors
• Advanced Methods for Word Vectors

4. Why is word2vec So Fast?: Speed Tricks for Neural Nets

• Softmax Approximations: Negative Sampling, Hierarchical Softmax
• Parallel Training
• Tips for Training on GPUs

5. Convolutional Networks for Text

• Bag of Words, Bag of n-grams, and Convolution
• Applications of Convolution: Context Windows and Sentence Modeling
• Stacked and Dilated Convolutions
• Structured Convolution
• Convolutional Models of Sentence Pairs
• Visualization for CNNs

6. Recurrent Networks for Sentence or Language Modeling

• Recurrent Networks
• Vanishing Gradient and LSTMs
• Strengths and Weaknesses of Recurrence in Sentence Modeling
• Pre-training for RNNs

7. Using/Evaluating Sentence Representations

• Sentence Similarity
• Textual Entailment
• Paraphrase Identification
• Retrieval

8. Conditioned Generation

• Encoder-Decoder Models
• Conditional Generation and Search
• Ensembling
• Evaluation
• Types of Data to Condition On

9. Attention

• Attention
• What do We Attend To?
• Improvements to Attention
• Specialized Attention Varieties

1. Case Study: "Attention is All You Need"

1. Foundations of Natural language Processing

• Word embedding
• Named entity recognition
• Parts-of-Speech tagging
• Language modeling
• Segmentation
• Paraphrasing
• Machine translation
• Information Extraction
• Text Summarization
• Conditional Random Fields
• Dimensionality Reduction: Matrix Factorization, Topic Models

2. Text Classification

• Tokenization
• Lemmatization
• Vectorization
• Bag of Words representation
• Language Models
• Tfidf
• Singular Value Decomposition
• Topic Models
• Discourse Modelling
• Coreference Resolution
• Question Answering Systems
• Visualizing complex and high dimensional data
• Sentiment Analysis

1. Case Study: (Spam detector, Q&A, Conversational systems)

1. Acoustic modeling
2. Highway deep neural network for low resource acoustic models
3. Temporal classification
4. Concentrating information in time
5. Speech recognition
6. Speaker invariant recognition
7. Robustness to noise
8. Speech synthesis

1. Case Study: Conversational Systems

• Object detection
  • Selective search
  • R-CNN
  • Fast R-CNN
  • YOLO
• Video analysis and object tracking
• Face Recognition
• Emotional Recognition

1. Case Study: Medical image analysis -- Pathology
2. Case Study: Tracking Suspicious movement for security