Short history of Neural Network approaches to sequence processing

• 2001: Neural Language Models
• 2013: Word Embeddings
• 2014: Sequence-to-Sequence models
• 2015: Attention
• 2016: Neural Machine Translation boom
• 2017: Transformers
• 2018: Pretrained Contextualized Word Embeddings (ELMo)
• 2019+: Massive Transformer Models (BERT, GPT-2, ...)
• 2020: Current Frontiers

Short history of Neural Network approaches to sequence processing

• 2001: Neural Language Models
• 2013: Word Embeddings *
• 2014: Sequence-to-Sequence models
• 2015: Attention *
• 2016: Neural Machine Translation boom
• 2017: Transformers *
• 2018: Pretrained Contextualized Word Embeddings (ELMo)
• 2019+: Massive Transformer Models (BERT, GPT-2, ...) *
• 2020: Current Frontiers

* Our today's (loose) agenda

Word Embeddings

"How do we convert free text into a format understandable by neural networks?"

One-hot encoding.

Word Embeddings

"How do we convert free text into a format understandable by neural networks?"

One-hot encoding.

"That results in extra large vectors. Can we do better?"

Well, there is this thing called dense representations.

Word Embeddings

"How do we convert free text into a format understandable by neural networks?"

One-hot encoding.

"That results in extra large vectors. Can we do better?"

Well, there is this thing called dense representations.

"Cool! While we are at it, can we make the format semantically meaningful?"

We don't even know how to train them quickly, but we can try...

Word Embeddings: A real-world example

A GloVe word embedding for king (trained on Wikipedia):

[ 
  0.50451  , 0.68607  , -0.59517 , -0.022801, 0.60046  , -0.13498 , -0.08813 ,
  0.47377  , -0.61798 , -0.31012 , -0.076666, 1.493    , -0.034189, -0.98173 ,
  0.68229  , 0.81722  , -0.51874 , -0.31503 , -0.55809 , 0.66421  , 0.1961l  ,
  -0.13495 , -0.11476 , -0.30344 , 0.41177  , -2.223   , -1.0756  , -1.0783  ,
  -0.34354 , 0.33505  , 1.9927   , -0.04234 , -0.64319 , 0.71125  , 0.49159  , 
  0.16754  , 0.34344  , -0.25663 , -0.8523  , 0.1661   , 0.40102  , 1.1685   , 
  -1.0137  , -0.21585 , -0.15155 , 0.78321  , -0.91241 , -1.6106  , -0.64426 , 
  -0.51042
]

Example and subsequent images taken from the great Illustrated Word2Vec

Word Embeddings: Visualized

By converting the numbers to colored bars we get a totally different picture:

Word Embeddings: Visualized

By converting the numbers to colored bars we get a totally different picture:

Especially in context:

Word Embeddings: Visualized II

Word Embeddings: Visualized III

It seems that some semantic meaning does indeed get encoded!*

Word Embeddings: Visualized III

It seems that some semantic meaning does indeed get encoded!*

* Note that king-man+woman does not exactly equal queen -- it's just that queen was the closest word in the space to the result of king-man+woman

Word Embeddings: Training

So how would one come up with these?

Word Embeddings: Training

So how would one come up with these?

Let a neural net learn the proper values on its own!

Word Embeddings: Training

So how would one come up with these?

Let a neural net learn the proper values on its own!

But how?

Word Embeddings: Training

So how would one come up with these?

Let a neural net learn the proper values on its own!

But how?

Give it some context and let it predict the center word.

Word Embeddings: Training

So how would one come up with these?

Let a neural net learn the proper values on its own!

But how?

Give it some context and let it predict the center word.

Image from https://amitness.com/2020/06/fasttext-embeddings/

Word Embeddings: CBOW Training

Image from https://lilianweng.github.io/lil-log/2017/10/15/learning-word-embedding.html

Word Embeddings: Limitations

Out Of Vocabulary Words

Morphology

Zipf's Law: the fundamental limitation

The general home-take point seems to be:

You never see most things.

Image from https://shadycharacters.co.uk/2015/10/zipfs-law/

Word Embeddings: FastText

The idea is relatively simple: split words into their own character n-grams

Word Embeddings: FastText

The idea is relatively simple: split words into their own character n-grams

Sum the n-grams together to create the embedding.

Word Embeddings: FastText Classification

This concept can be very easily extended to classification, which you can try out in Google Colab:

Text Classification with FastText in Google Colab

Word Embeddings: Recap

• Alternative to one-hot encoding

Word Embeddings: Recap

• Alternative to one-hot encoding

• Created as a byproduct of training a neural network

Word Embeddings: Recap

• Alternative to one-hot encoding

• Created as a byproduct of training a neural network

• Seems to encode some semantic meaning and allows for quick comparison

Word Embeddings: Recap

• Alternative to one-hot encoding

• Created as a byproduct of training a neural network

• Seems to encode some semantic meaning and allows for quick comparison

• Suffers from the implications of the Zipf's law

Word Embeddings: Recap

• Alternative to one-hot encoding

• Created as a byproduct of training a neural network

• Seems to encode some semantic meaning and allows for quick comparison

• Suffers from the implications of the Zipf's law

• FastText tries to alleviate this, can be directly used for classification as well.

LSTM: Conveyor belt

LSTM: Conveyor belt

• The cell state is sort of a "conveyor belt"

LSTM: Conveyor belt

• The cell state is sort of a "conveyor belt"

• Allows information to stay unchanged or get slightly updated

All the following nice images are from http://colah.github.io/posts/2015-08-Understanding-LSTMs/ which I highly recommend

LSTM: Conveyor belt

Step 1: Decide what to forget

LSTM: Conveyor belt

Step 2: Decide

• which values to update (it)
• what should the new values be (ˆCt)

LSTM: Conveyor belt

Step 2.5: perform forgetting and update

LSTM: Conveyor belt

Step 3: produce output (ht)

LSTM: Conveyor belt

A coveyor belt that can pick

• what to remember
• what to forget
• what to output

GRU: Simplified conveyor belt

GRU: Simplified conveyor belt

• Forget and input combined into a single "update gate" (zt)

GRU: Simplified conveyor belt

• Forget and input combined into a single "update gate" (zt)

• Cell state (Ct in LSTM) merged with the hidden state (ht)

GRU vs LSTM

• GRU is smaller and hence requires less compute
• But it turns out it cannot count (especially longer sequences)

GRU vs LSTM

• GRU is smaller and hence requires less compute
• But it turns out it cannot count (especially longer sequences)

On the Practical Computational Power of Finite Precision RNNs for Language Recognition (2018)

Application: Handwriting from Text

Input: "He dismissed the idea"

Application: Handwriting from Text

Input: "He dismissed the idea"

Output:

Application: Handwriting from Text

Input: "He dismissed the idea"

Output:

Generating Sequences With Recurrent Neural Networks, Alex Graves, 2013

Demo at https://www.cs.toronto.edu/~graves/handwriting.html

Application: Character-Level Text Generation

"The Unreasonable Effectiveness of Recurrent Neural Networks", Andrej Karpathy, 2015

Application: Image Question Answering

Application: Image Question Answering

Application: Image Caption Generation

Application: Video Caption Generation

Application: Video Caption Generation

Application: Adding Audion to Silent Film

Application: Adding Audion to Silent Film

Application: Medical Diagnosis

Application: End-to-End Driving *

Application: End-to-End Driving *

Input: features extracted from CNN Output: predicted steering angle

* On relatively straight roads

Application: Sentiment Analysis

Try it yourself at https://demo.allennlp.org/sentiment-analysis/

Application: Named Entity Recognition (NER)

Try it yourself at https://demo.allennlp.org/named-entity-recognition/

Application: Trump2Cash

Application: Trump2Cash

• A combination of Sentiment Analysis and Named Entity Recognition

Application: Trump2Cash

• A combination of Sentiment Analysis and Named Entity Recognition

How it works:

1. Monitor tweets of Donald Trump
2. Use NER to see if some of them mention a publicly traded company
3. Apply sentiment analysis and use its result to decide whether to buy or sell
4. Profit?

Application: Trump2Cash

See predictions live at https://twitter.com/Trump2Cash

Recurrent Neural Networks

• A "neural" way of handling sequences

Recurrent Neural Networks

• A "neural" way of handling sequences

• Their training usually happens by "unfolding" the network in time (Back Propagation Through Time -- BPTT)

Recurrent Neural Networks

• A "neural" way of handling sequences

• Their training usually happens by "unfolding" the network in time (Back Propagation Through Time -- BPTT)

• In theory they can handle sequences of any length. In practice it is difficult due to exploding and vanishing gradient.

Recurrent Neural Networks

• A "neural" way of handling sequences

• Their training usually happens by "unfolding" the network in time (Back Propagation Through Time -- BPTT)

• In theory they can handle sequences of any length. In practice it is difficult due to exploding and vanishing gradient.

• One solution is to use a different the network architecture, such as LSTM or GRU

Recurrent Neural Networks

• A "neural" way of handling sequences

• Their training usually happens by "unfolding" the network in time (Back Propagation Through Time -- BPTT)

• In theory they can handle sequences of any length. In practice it is difficult due to exploding and vanishing gradient.

• One solution is to use a different the network architecture, such as LSTM or GRU

• Both of these are used with great success in many practical applications, especially in the sequence-to-sequence setting

History of Deep Learning Milestones

From Deep Learning State of the Art (2020) by Lex Fridman at MIT

Attention visualized

See nice demo at https://distill.pub/2016/augmented-rnns/

What's Self Attention?

The animal didn't cross the street because it was too tired.

What does "it" refer to?

What's Self Attention?

The animal didn't cross the street because it was too tired.

What does "it" refer to?

Self Attention mechanics

The full Transformer seq2seq process

Big Transformers Wins: GPT-2

Try it yourself at https://transformer.huggingface.co/doc/gpt2-large

Big Transformers Wins: BERT

Big Transformers Wins: BERT

BERT Applications: any classification task

• Thanks to contextualized word vectors that BERT provides, the performance on many tasks has increased significantly

BERT Applications: any classification task

• Thanks to contextualized word vectors that BERT provides, the performance on many tasks has increased significantly

• (This is what it means to come from word vectors to transformers and back)

BERT Applications: any classification task

• Thanks to contextualized word vectors that BERT provides, the performance on many tasks has increased significantly

• (This is what it means to come from word vectors to transformers and back)

• For a real-life example of what it means to work with it, I recommend the PyTorch Sentiment Analysis tutorial

BERT Applications: ViLBERT

• A single model that can perform various vision and language tasks

vilbert.cloudcv.org

BERT Applications: Better Ctrl+F

• Repurposing question answering model for asking questions about the webpage the user is currently on.
• Check it out at github.com/model-zoo/shift-ctrl-f

Big Transformer Wins: Huggingface transformers

• A very nicely done library that allows anyone with some Python knowledge to play with pretrained state-of-the-art models (more in the docs).

Big Transformer Wins: Huggingface transformers

• A very nicely done library that allows anyone with some Python knowledge to play with pretrained state-of-the-art models (more in the docs).

• A small example: English to Slovak translator in about 10 lines of Python code: *

from transformers import MarianTokenizer, MarianMTModel
src = 'en'  # source language
trg = 'sk'  # target language
sample_text = "When will this presentation end ?"
mname = f'Helsinki-NLP/opus-mt-{src}-{trg}'
model = MarianMTModel.from_pretrained(mname)
tok = MarianTokenizer.from_pretrained(mname)
batch = tok.prepare_seq2seq_batch(src_texts=[sample_text]) 
gen = model.generate(**batch)
words = tok.batch_decode(gen, skip_special_tokens=True)
print(words)

Works with many other languages as well -- the full list is here

Attention and Transformers: Recap

Attention and Transformers: Recap

• Attention was a fix for sequence models that did not really work too wel

Attention and Transformers: Recap

• Attention was a fix for sequence models that did not really work too wel

• It turned out it was all that was needed for (bounded) sequence processing

Attention and Transformers: Recap

• Attention was a fix for sequence models that did not really work too wel

• It turned out it was all that was needed for (bounded) sequence processing

• Transformer is an encoder-decoder architecture that is "all the rage" now

Attention and Transformers: Recap

• Attention was a fix for sequence models that did not really work too wel

• It turned out it was all that was needed for (bounded) sequence processing

• Transformer is an encoder-decoder architecture that is "all the rage" now

• It has no time-depencency due to self-attention and is therefore easy to paralelize

Attention and Transformers: Recap

• Attention was a fix for sequence models that did not really work too wel

• It turned out it was all that was needed for (bounded) sequence processing

• Transformer is an encoder-decoder architecture that is "all the rage" now

• It has no time-depencency due to self-attention and is therefore easy to paralelize

• Well known models like BERT and GPT-* took the world of NLP by storm

Attention and Transformers: Recap

• Attention was a fix for sequence models that did not really work too wel

• It turned out it was all that was needed for (bounded) sequence processing

• Transformer is an encoder-decoder architecture that is "all the rage" now

• It has no time-depencency due to self-attention and is therefore easy to paralelize

• Well known models like BERT and GPT-* took the world of NLP by storm

• Very helpful in many tasks, easy to play with thanks to the Huggingface transformers library

Our path today

• From text to word vectors

Our path today

• From text to word vectors

• From word vectors to sequence processing

Our path today

• From text to word vectors

• From word vectors to sequence processing

• From sequence processing to attention

Our path today

• From text to word vectors

• From word vectors to sequence processing

• From sequence processing to attention

• From attention to transformers

Our path today

• From text to word vectors

• From word vectors to sequence processing

• From sequence processing to attention

• From attention to transformers

• Via transformers back to (better, contextualized) word vectors

Smaller models

• BERT (Large) is about 430MB when serialized on disk. Even for Google it's at best impractical to put it to production
• The trend is to make models smaller while keeping their performance roughly the same
  • DistilBERT
  • ALBERT
  • TinyBERT
  • MobileBERT
• Various avenues of research:
  • Knowledge Distillation
  • Quantization
  • ...

Non-English NLP

• Most of the cutting-edge research in NLP happens in English

Non-English NLP

• Most of the cutting-edge research in NLP happens in English

• Most of the world does not speak English

Non-English NLP

• Most of the cutting-edge research in NLP happens in English

• Most of the world does not speak English

• Many problems "solved" in English are still open in other (smaller) languages like Slovak or Ukrainian

Doing more with less data

• Data is almost always the bottleneck for NLP projects

Doing more with less data

• Data is almost always the bottleneck for NLP projects

• Being able to do data augmentation can push a project from "no-go" to "doable"

Doing more with less data

• Data is almost always the bottleneck for NLP projects

• Being able to do data augmentation can push a project from "no-go" to "doable"

• Open are a of research, some examples include backtranslation or MixUp for text