https://tomroth.dev/tags/reinforcement-learning/Recent content in reinforcement learning on Tom RothHugo -- gohugo.ioenTue, 07 Jan 2020 00:00:00 +0000https://tomroth.dev/intdim/Tue, 07 Jan 2020 00:00:00 +0000https://tomroth.dev/intdim/The paper in one sentence. The Pareto principle for neural networks: what’s the least number of parameters needed for to achieve most of the results? Structure of this article. I give an introduction to objective landscapes and talk about how to define solutions. I use these concepts to describe the subspace training method. Finally, I summarise the key results of the paper. Objective landscapes Link to heading For every combination of neural network architecture and dataset, the shape of the objective/loss function is fixed.https://tomroth.dev/dvrl/Thu, 17 Oct 2019 00:00:00 +0000https://tomroth.dev/dvrl/Data Valuation using Reinforcement Learning Link to heading Paper link: https://arxiv.org/pdf/1909.11671.pdf At a glance Link to heading Machine learning models use data for training. But not all data points are equally valuable. A data point can be low-quality for a variety of reasons, such as incorrect labelling, noisy input, being too common, or it’s not from the same distribution as the test set. Whatever the case, removing bad datapoints tends to increase model performance.https://tomroth.dev/seq2sql/Thu, 20 Jun 2019 00:00:00 +0000https://tomroth.dev/seq2sql/Seq2SQL: Generating structured queries from Natural Language using Reinforcement Learning (2017) Link to heading Paper link: https://arxiv.org/pdf/1709.00103.pdf Authors: Victor Zhong, Caiming Xiong, Richard Socher Here’s my notes from this paper. Seq2SQL uses the Seq2Seq model as a starting point The output of the seq2seq model is unnecessarily wide for SQL queries. In other words, the paper is only interested in a small subset of the seq2seq output space. As a result, the output of Seq2SQL is limited to the union of words in the question, SQL commands and table headers The paper created its own datasets.https://tomroth.dev/dqn-nnet/Thu, 01 Nov 2018 00:00:00 +0000https://tomroth.dev/dqn-nnet/After building the basic DQN, I started trying some different neural network architectures. I found vastly different results between them. Spoiler: more complex doesn’t equal better. Almost the opposite. Here are some results for the rewards obtained in each episode. I trained these models on cartpole-v1. Each was trained for 1000 episodes. Disclaimer: I did only one run with each architectures. I kept other hyperparameters the same as the previous post, but there are likely some interaction effects unaccounted for.https://tomroth.dev/dqn-simple/Wed, 31 Oct 2018 00:00:00 +0000https://tomroth.dev/dqn-simple/Click the image above to see the source code. Last time, we established some theory for our DQN. Now it’s time to implement it. We won’t build a full DQN from the DeepMind paper, not yet. Instead we’ll build a simplified version. I call it the basic DQN. The basic DQN is the same as the full DQN, but missing a target network and reward clipping. We’ll get to that in the next post.https://tomroth.dev/dqn-basics/Tue, 23 Oct 2018 00:00:00 +0000https://tomroth.dev/dqn-basics/In February 2015, Google Brain published a paper combining deep neural networks and reinforcement learning for the first time - called DQN (deep q-network). It was a landmark moment. DQN was the first algorithm that could successfully play a wide range of Atari games. Other algorithms at the time could perform well at a single game but couldn’t generalise across games. Impressively, DQN was able to perform above human level at a range of Atari games using only the screen pixels as input.https://tomroth.dev/gym-play/Sat, 13 Oct 2018 00:00:00 +0000https://tomroth.dev/gym-play/Ever wanted to play the games that your reinforcement agents are learning? OpenAI have a keyboard agent to let you try out the gym environments yourself. Here’s how to set it up. First, if you haven’t already, install the gym package with pip install gym, or by following the instructions here. Then: Go to this link. Click “raw”. Copy the text and save it to a file called keyboard_agent.py Open up a shell prompt and navigate to where you saved the file.https://tomroth.dev/policy-eval/Sun, 27 May 2018 00:00:00 +0000https://tomroth.dev/policy-eval/Today, we explore policy evaluation. We’ll explain it by example. Given a policy, policy evaluation aims to find the value of each state $ v_\pi (s) $. That’s what we’ll be doing with the equiprobable random policy. The scenario Link to heading Taken from the textbook “An Introduction to Reinforcement Learning” (Sutton and Barto), Gridworld is the following scenario. There is a grid with sixteen squares. The top left and the bottom right square are terminal squares.https://tomroth.dev/bandit/Wed, 24 May 2017 00:00:00 +0000https://tomroth.dev/bandit/Machine learning can be split up into many subfields. One of these subfields growing in relevance is reinforcement learning. One of the core problems in reinforcement learning is the multi-armed bandit problem. This problem has been well studied and is commonly used to explore the tradeoff between exploration and exploitation integral to reinforcement learning. To illustrate this tradeoff and to visualise different ways of solving the multi-armed bandit problem, I created a simulation using the JavaScript library d3.