https://tomroth.dev/tags/algorithm/Recent content in algorithm on Tom RothHugo -- gohugo.ioenSun, 01 Dec 2019 00:00:00 +0000https://tomroth.dev/backprop-toposort/Sun, 01 Dec 2019 00:00:00 +0000https://tomroth.dev/backprop-toposort/Neural networks need to update their weights with respect to a loss function. Rather than compute these weight updates manually, today we usually use automatic differentiation packages and tools. We call these autodiff systems. To do their job, autodiff systems construct computational graphs of the function they’re trying to find the gradient of. These computational graphs are key to backpropagation. I won’t go into how these graphs are constructed here.https://tomroth.dev/dqn-nnet/Thu, 01 Nov 2018 00:00:00 +0000https://tomroth.dev/dqn-nnet/After building the basic DQN, I started trying some different neural network architectures. I found vastly different results between them. Spoiler: more complex doesn’t equal better. Almost the opposite. Here are some results for the rewards obtained in each episode. I trained these models on cartpole-v1. Each was trained for 1000 episodes. Disclaimer: I did only one run with each architectures. I kept other hyperparameters the same as the previous post, but there are likely some interaction effects unaccounted for.https://tomroth.dev/dqn-simple/Wed, 31 Oct 2018 00:00:00 +0000https://tomroth.dev/dqn-simple/Click the image above to see the source code. Last time, we established some theory for our DQN. Now it’s time to implement it. We won’t build a full DQN from the DeepMind paper, not yet. Instead we’ll build a simplified version. I call it the basic DQN. The basic DQN is the same as the full DQN, but missing a target network and reward clipping. We’ll get to that in the next post.https://tomroth.dev/dqn-basics/Tue, 23 Oct 2018 00:00:00 +0000https://tomroth.dev/dqn-basics/In February 2015, Google Brain published a paper combining deep neural networks and reinforcement learning for the first time - called DQN (deep q-network). It was a landmark moment. DQN was the first algorithm that could successfully play a wide range of Atari games. Other algorithms at the time could perform well at a single game but couldn’t generalise across games. Impressively, DQN was able to perform above human level at a range of Atari games using only the screen pixels as input.https://tomroth.dev/sarsa-qlearning/Mon, 08 Oct 2018 00:00:00 +0000https://tomroth.dev/sarsa-qlearning/In this post, we’ll see how three commonly-used reinforcement algorithms - sarsa, expected sarsa and q-learning - stack up on the OpenAI Gym Taxi (v2) environment. Note: this post assumes that the reader is familiar with basic RL concepts. A good resource for learning these is the textbook by Sutton and Barto (2018), which is freely available online. Sarsa, Expected Sarsa and Q-Learning Link to heading So, what are these algorithms?https://tomroth.dev/nearest-neighbour/Sat, 26 Aug 2017 00:00:00 +0000https://tomroth.dev/nearest-neighbour/The nearest neighbour classifier is a very simple algorithm for image classification. While not used much in practice, it is simple to implement and it helps to gain a deeper understanding of the problems in image classification. Just like other classifiers, if we give the nearest neighbour classifier an image, it’ll try and find its closest match. We “train” the classifier by giving it a large collection of images that it allowed to search through to find matches.https://tomroth.dev/bandit/Wed, 24 May 2017 00:00:00 +0000https://tomroth.dev/bandit/Machine learning can be split up into many subfields. One of these subfields growing in relevance is reinforcement learning. One of the core problems in reinforcement learning is the multi-armed bandit problem. This problem has been well studied and is commonly used to explore the tradeoff between exploration and exploitation integral to reinforcement learning. To illustrate this tradeoff and to visualise different ways of solving the multi-armed bandit problem, I created a simulation using the JavaScript library d3.https://tomroth.dev/kadane/Tue, 09 May 2017 00:00:00 +0000https://tomroth.dev/kadane/There is something called the maximum subarray problem, and it goes something like this. You are given an array of numbers: something like 3,5,-2,-1,5,3,1,-4,-6,5. Your job is to come up with the subarray with the highest sum. For this example it would be the subarray [3,5,-2,-1,5,3,1], which has a sum of 14. No other subarray gives a higher sum. Let’s look at a few more: The sequence -3,-3,-2,-4,-2,-5,-4 has the highest sum being -2, given by the subarray [-2].