Navigating complex decision spaces

Problems and paradigms in sequential choice

Bibliographic Data

ID7983216
AuthorsMatthew M Walsh (0000-0002-1457-3016, Wright-Patterson Air Force Base), John R Anderson (0000-0002-8377-548X, Carnegie Mellon University)
Year2014
Volume140
Issue2
Pages466-486
Publication date2014-01-01
Open AccessNo
TypeARTICLE
VenuePsychological Bulletin (JOURNAL)
Journal identifiersISSN: 0033-2909 • E-ISSN: 1939-1455
PublisherAmerican Psychological Association (APA) (PUBLISHER)
DOI10.1037/a0033455
PMID23834192
PMCIDPMC4309984
OpenAlexW1998992789
LanguageEN
Citations received1
References cited28

To behave adaptively, we must learn from the consequences of our actions. Doing so is difficult when the consequences of an action follow a delay. This introduces the problem of temporal credit assignment. When feedback follows a sequence of decisions, how should the individual assign credit to the intermediate actions that comprise the sequence? Research in reinforcement learning provides 2 general solutions to this problem: model-free reinforcement learning and model-based reinforcement learning. In this review, we examine connections between stimulus-response and cognitive learning theories, habitual and goal-directed control, and model-free and model-based reinforcement learning. We then consider a range of problems related to temporal credit assignment. These include second-order conditioning and secondary reinforcers, latent learning and detour behavior, partially observable Markov decision processes, actions with distributed outcomes, and hierarchical learning. We ask whether humans and animals, when faced with these problems, behave in a manner consistent with reinforcement learning techniques. Throughout, we seek to identify neural substrates of model-free and model-based reinforcement learning. The former class of techniques is understood in terms of the neurotransmitter dopamine and its effects in the basal ganglia. The latter is understood in terms of a distributed network of regions including the prefrontal cortex, medial temporal lobes, cerebellum, and basal ganglia. Not only do reinforcement learning techniques have a natural interpretation in terms of human and animal behavior but they also provide a useful framework for understanding neural reward valuation and action selection

Action selection · Cognition · Cognitive psychology · Machine learning · Markov decision process · Markov process · Perception · Reinforcement · Reinforcement learning · Sequence learning · Artificial Intelligence · Attention Deficit Hyperactivity Disorder · Computer Science · Neural and Behavioral Psychology Studies · Neuroscience · Neurotransmitter Receptor Influence on Behavior · Psychology · Social Psychology

Tools

Open DOISci-HubOpen Access