rnn on Tom Roth

rnn on Tom Rothhttps://tomroth.dev/tags/rnn/Recent content in rnn on Tom RothHugo -- gohugo.ioenFri, 07 Oct 2022 00:00:00 +0000How does weight-tying work in a RNN?https://tomroth.dev/weight_tying/Fri, 07 Oct 2022 00:00:00 +0000https://tomroth.dev/weight_tying/Weight-tying is where you have a language model and use the same weight matrix for the input-to-embedding layer (the input embedding) and the hidden-to-softmax layer (the output embedding). The idea is that these two matrices contain essentially the same information, each having a row per word in the vocabulary. The concept seems to be first mentioned by Press and Wolf in 2016. Below we illustrate this with a RNN. This post assumes that you are somewhat familiar already with how a RNN works.