name: inverse layout: true class: center, middle, inverse --- # Sequence Processing Recurrent Neural Networks and more .footnote[Marek Šuppa<br>ESS 2020, Bratislava] --- layout: false class: center # Types of Neural Networks  .footnote[.font-small[Image from https://karpathy.github.io/2015/05/21/rnn-effectiveness/]] --- layout: false class: center # Unfolded RNN  --- layout: false class: center # Unfolded RNN  --- layout: false class: center # Training unfolded RNN  This concept is called Back Propagation Through Time (**BPTT**) --- layout: false class: center # Training unfolded RNN  This concept is called Back Propagation Through Time (**BPTT**) --- layout: false class: center # Training unfolded RNN  Note how various parts of the unfolded RNN impact $h_2$ --- layout: false class: center # Problems with long-term dependencies  --- layout: false class: center ## LSTM: what to forget and what to remember  --- layout: false ## LSTM: Conveyor belt  -- - The cell state is sort of a "conveyor belt" -- - Allows information to stay unchanged or get slightly updated .footnote[.font-small[All the following nice images are from http://colah.github.io/posts/2015-08-Understanding-LSTMs/ which I highly recommend]] --- layout: false calls: center, middle ## LSTM: Conveyor belt  **Step 1**: Decide what to forget --- layout: false calls: center, middle ## LSTM: Conveyor belt  **Step 2**: Decide - which values to update ($i_t$) - what should the new values be ($\hat{C}_t$) --- layout: false calls: center, middle ## LSTM: Conveyor belt  **Step 2.5**: perform forgetting and update --- layout: false calls: center, middle ## LSTM: Conveyor belt  **Step 3**: produce output ($h_t$) --- layout: false calls: center, middle ## LSTM: Conveyor belt  A coveyor belt that can pick - what to remember - what to forget - what to output --- layout: false calls: center, middle ## GRU: Simplified conveyor belt  -- - Forget and input combined into a single "update gate" ($z_t$) -- - Cell state ($C_t$ in LSTM) merged with the hidden state ($h_t$) --- layout: false ## GRU vs LSTM - GRU is smaller and hence requires less compute - But it turns out it cannot count (especially longer sequences) -- .center[] [On the Practical Computational Power of Finite Precision RNNs for Language Recognition (2018)](https://arxiv.org/abs/1805.04908) --- class: middle ## Application: Machine Translation .center[] --- ## Application: Handwriting from Text **Input:** "He dismissed the idea" -- **Output:** .center[] -- [Generating Sequences With Recurrent Neural Networks, Alex Graves, 2013](https://arxiv.org/abs/1308.0850) Demo at https://www.cs.toronto.edu/~graves/handwriting.html --- ## Application: Character-Level Text Generation .center[] .footnote[.font-small[["The Unreasonable Effectiveness of Recurrent Neural Networks"](https://karpathy.github.io/2015/05/21/rnn-effectiveness/), Andrej Karpathy, 2015]] --- ## Application: Image Question Answering .center[] -- .left-eq-column[ .center[] .font-small[Exploring Models and Data for Image Question Answering, 2015] ] .right-eq-column[ Live Demo at https://vqa.cloudcv.org/ ] --- ## Application: Image Caption Generation .center[] --- ## Application: Video Caption Generation .center[] -- .left-eq-column[ .center[] .font-small[Sequence to Sequence - Video to Text, Venugopalan et al., 2015] ] .right-eq-column[ More at https://vsubhashini.github.io/s2vt.html ] --- ## Application: Adding Audion to Silent Film .center[] -- .left-eq-column[ .center[] .font-small[Visually Indicated Sounds, Owens et al., 2015] ] .right-eq-column[ More at http://andrewowens.com/vis/ ] --- ## Application: Medical Diagnosis .center[] --- ## Application: End-to-End Driving .red[*] <br><br> .left-eq-column[] .right-eq-column[] -- <div style="float: clear"></div> **Input**: features extracted from CNN **Output**: predicted steering angle .footnoe[.red[*] On relatively straight roads] --- ## Application: Stock Market Prediction .center[] --- ## Application: Sentiment Analysis .center[] Try it yourself at https://demo.allennlp.org/sentiment-analysis/ --- ## Application: Named Entity Recognition (NER) .center[] Try it yourself at https://demo.allennlp.org/named-entity-recognition/ --- ## Application: Trump2Cash .center[] --- ## Application: Trump2Cash - A combination of Sentiment Analysis and Named Entity Recognition -- How it works: 1. Monitor tweets of Donald Trump 2. Use NER to see if some of them mention a publicly traded company 3. Apply sentiment analysis 4. Profit? --- ## Application: Trump2Cash .center[] See predictions live at https://twitter.com/Trump2Cash --- class: middle, inverse # Attention and Transformers --- ## History of Deep Learning Milestones  .footnote[ From [Deep Learning State of the Art (2020)](https://www.youtube.com/watch?v=0VH1Lim8gL8) by Lex Fridman at MIT] --- class: middle ## The perils of seq2seq modeling <video width="100%" height="auto" loop="" autoplay="" controls=""> <source src="/images/seq2seq_6.mp4" type="video/mp4" /> Your browser does not support the video tag. </video> -- Aren't we throwing out a bit too much? .footnote[.font-small[Videos from https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/]] --- class: middle ## The fix Let's use the full encoder output! <video width="100%" height="auto" loop="" autoplay="" controls=""> <source src="/images/seq2seq_7.mp4" type="video/mp4" /> Your browser does not support the video tag. </video> -- But how do we combine all the hidden states together? --- class: middle ## The mechanics of Attention <video width="100%" height="auto" loop="" autoplay="" controls=""> <source src="/images/seq2seq_8.mp4" type="video/mp4" /> Your browser does not support the video tag. </video> --- class: middle ## Getting alignment with attention <video width="100%" height="auto" loop="" autoplay="" controls=""> <source src="/images/seq2seq_9.mp4" type="video/mp4" /> Your browser does not support the video tag. </video> --- ## Attention visualized .center[] See nice demo at https://distill.pub/2016/augmented-rnns/ --- ## Attention also helps explainability of stock prediction .center[] .footnote[.font-small[ [News-Driven Stock Prediction With Attention-Based Noisy Recurrent State Transition, 2020](https://arxiv.org/abs/2004.01878) ]] --- class: middle # What if we only used attention? --- class: middle ## The Transformer architecture .center[] .footnote[.font-small[Images from https://jalammar.github.io/illustrated-transformer/]] --- class: middle ## The Transformer's Encoder .center[] --- ## What's Self Attention? .center[ *The animal didn't cross the street because it was too tired.* ] What does "it" refer to? -- .center[] --- ## Self Attention mechanics .center[] --- ## The full Transformer seq2seq process .center[] --- ## Transformer recap - Encoder-decoder architecture - No time-depencency due to self-attention - Easy to paralelize - Very helpful in many tasks --- ## Big Transformers Wins: GPT-2 .center[] Try it yourself at https://transformer.huggingface.co/doc/gpt2-large --- ## Big Transformers Wins: BERT .center[] --- ## BERT: for Forex Movement Prediction .center[] --- ## BERT: for Forex Movement Prediction .center[] [Group, Extract and Aggregate: Summarizing a Large Amount of Finance News for Forex Movement Prediction](https://www.aclweb.org/anthology/D19-5106.pdf)