Jekyll2022-04-29T20:26:20+00:00http://localhost:4000/feed.xmlTushar JainGrad Student <br> New York UniversityTushar Jaintushar@nyu.eduhttp://tushar.usCuriosity-driven Exploration2018-10-22T00:00:00+00:002018-10-22T00:00:00+00:00http://localhost:4000/ml%20blog/curiosity<h1 id="curiosity-driven-exploration-by-self-supervised-prediction">Curiosity-driven Exploration by Self-supervised Prediction</h1> <h2 id="tldr">TL;DR</h2> <p>Curiosity = the error in agent’s ability to predict the consequence of its own actions in a feature space learned by a self-supervised inverse dynamics model.</p> <h2 id="introduction">Introduction</h2> <p>Objective of Reinforcement Learning is to find an agent parameterised by $$\theta$$ that maximise expected the sum of rewards over the policy trajectory.</p> <p>Mathematically, it can be shown as:</p> $\theta^* = arg \max_{\theta} E_{\tau \sim p_{\theta}(\tau)} [\sum_t r_t]$ <p>The reinforcement learning framework is usually represented as Markov Decision Process which is tuple of $$\{S,A,T,r\}$$, where: <br /></p> <p>S: State Space A: Action Space T: Transition Operator r: rewards</p> <p>And this can pictured as:</p> <figure class=""> <img src="/assets/images/curiosity/MDP.png" alt="RL as MDP" /> <figcaption>Reinforcement Learning as Markov Decision Process (MDP). </figcaption> </figure> <p>So, clearly the only learning signal we are providing to the agent is through the rewards.</p> <h3 id="problem-with-rewards">Problem with Rewards</h3> <ul> <li>Extrinsic rewards are extremely sparse, or absent completely in more realistic scenarios</li> <li>Extrinsic Rewards are usually hand-made</li> <li>Reward Shaping in itself is a very tedious &amp; complex task</li> <li>Only learning signal for RL agents is from reward signals</li> </ul> <h2 id="a-solution-add-more-rewards">A Solution: Add more rewards?</h2> <h3 id="intrinsic-motiviation">Intrinsic Motiviation</h3> <p>We want to make the reward structure more dense so that it’s easier for agents to learn. So, we want to create some extra rewards using all the information availbale to us.</p> <figure class=""> <img src="/assets/images/curiosity/Motivation.png" alt="Motivation" /> <figcaption>From all available information get Motivation ( i.e. intrinsic reward) </figcaption> </figure> <p>Using all the current information we have, we want to motivate the agent to expore more, i.e. generate rewards for unexplored states. Mathematically,</p> $r_t = r_t^i + r_t^e \\ \theta^* = arg \max_{\theta} E_{\tau \sim p_{\theta}(\tau)} [\sum_t r_t]$ <p>There are 2 broad classes of techniques to achieve this:</p> <ol> <li>encourage the agent to explore “novel” states (Bellemare et al., 2016; Lopes et al., 2012; Poupart et al., 2006) or,</li> <li>encourage the agent to perform actions that reduce the error/uncertainty in the agent’s ability to predict the consequence of its own actions (i.e. its knowledge about the environment)</li> </ol> <p>Curiosity falls in the category of “generate an intrinsic reward signal based on how hard it is for the agent to predict the consequences of its own actions” i.e. 2nd Category.</p> <h2 id="prediction-error-as-curiosity-reward">Prediction Error as Curiosity Reward</h2> <h3 id="simple-solution">Simple Solution</h3> $\hat{s}_{t+1} = f(s_t, a_t) \\ r^i_t = \eta \lVert \hat{s}_{t+1} - s_{t+1} \rVert_k$ <p>However:</p> <ul> <li>It’s very hard to predict in raw sensory state like pixels</li> <li>Also, it’s unclear if predicting the pixels is the right objective</li> </ul> <p>The underlying problem is that the agent is <strong>unaware that some parts of the state space simply cannot be modeled</strong> and thus the agent can fall into an artificial curiosity trap and stall its exploration. This is one of the pitfalls of count based exploration, as they try to model a feature space from states directly.</p> <p>This can better understood with an example such as when the agent tries to the movement of leaves due to a breeze. This is super hard to learn and is also irrelvant for the agent as it does not affect the agent and nor is it controllable by the agent.</p> <figure class=""> <img src="https://media.giphy.com/media/O9GevoMfFkl6E/giphy.gif" alt="Leaves" /> <figcaption>Trying to predict the movement of leaves is hard, even so in pixel space </figcaption> </figure> <p>Then what to do? Let’s look at the factors contributing to observation space:</p> <ol> <li>things that can be controlled by the agent,</li> <li>things that the agent cannot control but that can affect the agent (e.g. a vehicle driven by another agent), and</li> <li>things out of the agent’s control and not affecting the agent (e.g. moving leaves)</li> </ol> <p>A good feature space for curiosity should model (1) and (2) and be unaffected by (3)</p> <p><strong>Key Insight</strong>: we only predict those changes in the environment that could possibly be due to the actions of our agent or affect the agent, and ignore the rest.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>* This helps avoid pitfalls of previous work. * Transform input to feature space learnt by self-supervision * Since the neural network is only required to predict the action, it has no incentive to represent within its feature embedding space the factors of variation in the environment that don't affect the agent. * And then use this feature space to learn a forward model </code></pre></div></div> <h2 id="self-supervised-prediction-for-exploration">Self-supervised prediction for exploration</h2> <h3 id="inverse-dynamics">Inverse Dynamics</h3> <p>This has 2 Sub-modules:</p> <ul> <li>State Encoder: $$s_t \to \phi (s_t)$$</li> <li>Action Predictor: $$\phi (s_{t+1}), \phi (s_t) \to \hat{a_t}$$</li> </ul> <p>This is equivalient to learning function $$g$$ (Inverse Dynamics Model)</p> $\hat{a}_t = g(s_t, s_{t+1}; \theta_I) \\ \min_{\theta_I} L_I(\hat{a_t}, a_t)$ <h3 id="forward-model">Forward Model</h3> $\hat{\phi}(s_{t+1}) = f(\phi(s_t), a_t; \theta_{F}) \\ L_F (\phi(s_{t+1}), \hat{\phi}(s_{t+1})) = \frac{1}{2} \lVert \phi(s_{t+1}) - \hat{\phi}(s_{t+1}) \rVert^2_2 \\ r_t^i = \frac{\eta}{2} \lVert \phi(s_{t+1}) - \hat{\phi}(s_{t+1}) \rVert^2_2$ <ul> <li>Jointly Optimise $$I$$ and $$F$$</li> <li>As there is no incentive for this feature space to encode any environmental features that are not influenced by the agent’s actions, our agent will receive no rewards for reaching environmental states that are inherently unpredictable and its exploration strategy will be robust to the presence of distractor objects, changes in illumination, or other nuisance sources of variation in the environment.</li> </ul> <h2 id="roles-of-curiosity">Roles of Curiosity</h2> <ol> <li>Solve Tasks with Sparse Rewards</li> <li>Explore its env in quest for knowledge, improves as agents gain mroe knwoledge</li> <li>Learn skills for future scenarios</li> </ol> <h2 id="experiments">Experiments</h2> <ul> <li>Diff. levels for train and test</li> <li>Trained using visual Inputs</li> <li>Current State Stacked with previous 3 states</li> <li>Action repeat: 4x VizDoom 6x SuperMario Bros <ul> <li>No Action repeat during inference</li> </ul> </li> </ul> <h3 id="environments">Environments</h3> <ol> <li> <p>VizDoom</p> </li> <li> <p>SuperMario Bros (Paquette 2016)</p> <ul> <li>Trained only on curiosity reward</li> </ul> </li> </ol> <p><strong>Main Contribution</strong>: designing an intrinsic reward signal based on prediction error of the agent’s knowledge about its environment that scales to high-dimensional continuous state spaces like images, bypasses the hard problem of predicting pixels and is unaffected by the unpredictable aspects of the environment that do not affect the agent.</p>Tushar Jaintushar@nyu.eduhttp://tushar.usUsing error in Self-supervised Prediction to explore the world.