Hrvoje Stojić
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Hrvoje Stojić has a total of 51 co-authors. Here are the top co-authors by number of shared publications:
- Maarten Speekenbrink
Experimental Psychology, University College London
2 shared publications Pantelis P. Analytis
Danish Institute of Advanced Studies
2 shared publications- Alejandro de la Vega
The University of Texas at Austin
1 shared publication - Alexandros Gelastopoulos
Department of Social and Behavioral Sciences, Toulouse School of Economics, University of Toulouse Capitole, Toulouse 31080, France
1 shared publication - Ashraf Galal Eid
Finance and Economics Department, College of Business and Economics, Qatar University, Doha, Qatar
1 shared publication - Christopher Johnson Markiewicz
Department of Psychology, Stanford University, Stanford, CA, USA
1 shared publication
Latest Publications
- Open Access
- 1 December 2023
- Published by Institute of Mathematical Statistics in The Annals of Applied Probability
- Vol. 33 (6A) , 4366-4394
Ranking-based rich-get-richer processes
We study a discrete-time Markov process Xn∈Rd for which the distribution of the future increments depends only on the relative ranking of its components (descending order by value). We endow the process with a rich-get-richer assumption and show that, together with a finite second moments assumption, it is enough to guarantee almost sure convergence of Xn/n. We characterize the possible limits if one is free to choose the initial state and we give a condition under which the initial state is irrelevant. Finally, we show how our framework can account for ranking-based Pólya urns and can be used to study ranking algorithms for web interfaces.
- 1 December 2023
- Published by Institute of Mathematical Statistics in The Annals of Applied Probability
- Vol. 33 (6A) , 4366-4394
Open AccessRanking-based rich-get-richer processes
We study a discrete-time Markov process Xn∈Rd for which the distribution of the future increments depends only on the relative ranking of its components (descending order by value). We endow the process with a rich-get-richer assumption and show that, together with a finite second moments assumption, it is enough to guarantee almost sure convergence of Xn/n. We characterize the possible limits if one is free to choose the initial state and we give a condition under which the initial state is irrelevant. Finally, we show how our framework can account for ranking-based Pólya urns and can be used to study ranking algorithms for web interfaces.
- Open Access
- 1 December 2021
- Published by Elsevier in Neural Networks
- Vol. 144, 229-246
An empirical evaluation of active inference in multi-armed bandits
- 1 December 2021
- Published by Elsevier in Neural Networks
- Vol. 144, 229-246
Open AccessAn empirical evaluation of active inference in multi-armed bandits
- Open Access
Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow
In the past decade, model-free reinforcement learning (RL) has provided solutions to challenging domains such as robotics. Model-based RL shows the prospect of being more sample-efficient than model-free methods in terms of agent-environment interactions, because the model enables to extrapolate to unseen situations. In the more recent past, model-based methods have shown superior results compared to model-free methods in some challenging domains with non-linear state transitions. At the same time, it has become apparent that RL is not market-ready yet and that many real-world applications are going to require model-based approaches, because model-free methods are too sample-inefficient and show poor performance in early stages of training. The latter is particularly important in industry, e.g. in production systems that directly impact a company's revenue. This demonstrates the necessity for a toolbox to push the boundaries for model-based RL. While there is a plethora of toolboxes for model-free RL, model-based RL has received little attention in terms of toolbox development. Bellman aims to fill this gap and introduces the first thoroughly designed and tested model-based RL toolbox using state-of-the-art software engineering practices. Our modular approach enables to combine a wide range of environment models with generic model-based agent classes that recover state-of-the-art algorithms. We also provide an experiment harness to compare both model-free and model-based agents in a systematic fashion w.r.t. user-defined evaluation metrics (e.g. cumulative reward). This paves the way for new research directions, e.g. investigating uncertainty-aware environment models that are not necessarily neural-network-based, or developing algorithms to solve industrially-motivated benchmarks that share characteristics with real-world problems.
Open AccessBellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow
In the past decade, model-free reinforcement learning (RL) has provided solutions to challenging domains such as robotics. Model-based RL shows the prospect of being more sample-efficient than model-free methods in terms of agent-environment interactions, because the model enables to extrapolate to unseen situations. In the more recent past, model-based methods have shown superior results compared to model-free methods in some challenging domains with non-linear state transitions. At the same time, it has become apparent that RL is not market-ready yet and that many real-world applications are going to require model-based approaches, because model-free methods are too sample-inefficient and show poor performance in early stages of training. The latter is particularly important in industry, e.g. in production systems that directly impact a company's revenue. This demonstrates the necessity for a toolbox to push the boundaries for model-based RL. While there is a plethora of toolboxes for model-free RL, model-based RL has received little attention in terms of toolbox development. Bellman aims to fill this gap and introduces the first thoroughly designed and tested model-based RL toolbox using state-of-the-art software engineering practices. Our modular approach enables to combine a wide range of environment models with generic model-based agent classes that recover state-of-the-art algorithms. We also provide an experiment harness to compare both model-free and model-based agents in a systematic fashion w.r.t. user-defined evaluation metrics (e.g. cumulative reward). This paves the way for new research directions, e.g. investigating uncertainty-aware environment models that are not necessarily neural-network-based, or developing algorithms to solve industrially-motivated benchmarks that share characteristics with real-world problems.
- 1 October 2020
- Published by American Psychological Association (APA) in Journal of Experimental Psychology: Learning, Memory, and Cognition
- Vol. 46 (10) , 1836-1856
Linear integration and lexicographic models of choice: A cue weight learning perspective.
Choosing between options characterized by multiple cues can be a daunting task. People may integrate all information at hand or just use lexicographic strategies that ignore most of it. Notably, integrative strategies require knowing exact cue weights, whereas lexicographic heuristics can operate by merely knowing the importance order of cues. Here we study how using integrative or lexicographic strategies interacts with learning about cues. In our choice-learning-estimation paradigm people first make choices, learning about cues from the experienced qualities of chosen options, and then estimate qualities of new options. We developed delta-elimination (DE), a new lexicographic strategy that generalizes previous heuristics to any type of environment, and compared it to the integrative weighted-additive (WADD) strategy. Our results show that participants learned cue weights, regardless of whether the DE strategy or the WADD strategy described their choices the best. Still, there was an interaction between the adopted strategy and the cue weight learning process: the DE users learned cue weights slower than the WADD users. This work advances the study of lexicographic choice strategies, both empirically and theoretically, and deepens our understanding of strategy selection, in particular the interaction between the strategy used and learning the structure of the environment.
- 1 October 2020
- Published by American Psychological Association (APA) in Journal of Experimental Psychology: Learning, Memory, and Cognition
- Vol. 46 (10) , 1836-1856
Linear integration and lexicographic models of choice: A cue weight learning perspective.
Choosing between options characterized by multiple cues can be a daunting task. People may integrate all information at hand or just use lexicographic strategies that ignore most of it. Notably, integrative strategies require knowing exact cue weights, whereas lexicographic heuristics can operate by merely knowing the importance order of cues. Here we study how using integrative or lexicographic strategies interacts with learning about cues. In our choice-learning-estimation paradigm people first make choices, learning about cues from the experienced qualities of chosen options, and then estimate qualities of new options. We developed delta-elimination (DE), a new lexicographic strategy that generalizes previous heuristics to any type of environment, and compared it to the integrative weighted-additive (WADD) strategy. Our results show that participants learned cue weights, regardless of whether the DE strategy or the WADD strategy described their choices the best. Still, there was an interaction between the adopted strategy and the cue weight learning process: the DE users learned cue weights slower than the WADD users. This work advances the study of lexicographic choice strategies, both empirically and theoretically, and deepens our understanding of strategy selection, in particular the interaction between the strategy used and learning the structure of the environment.
- 1 October 2020
- Published by American Psychological Association (APA) in Journal of Experimental Psychology: General
- Vol. 149 (10) , 1878-1907
It’s new, but is it good? How generalization and uncertainty guide the exploration of novel options.
How do people decide whether to try out novel options as opposed to tried-and-tested ones? We argue that they infer a novel option's reward from contextual information learned from functional relations and take uncertainty into account when making a decision. We propose a Bayesian optimization model to describe their learning and decision making. This model relies on similarity-based learning of functional relationships between features and rewards, and a choice rule that balances exploration and exploitation by combining predicted rewards and the uncertainty of these predictions. Our model makes 2 main predictions. First, decision makers who learn functional relationships will generalize based on the learned reward function, choosing novel options only if their predicted reward is high. Second, they will take uncertainty about the function into account, and prefer novel options that can reduce this uncertainty. We test these predictions in 3 preregistered experiments in which we examine participants' preferences for novel options using a feature-based multiarmed bandit task in which rewards are a noisy function of observable features. Our results reveal strong evidence for functional exploration and moderate evidence for uncertainty-guided exploration. However, whether or not participants chose a novel option also depended on their attention, as well as reflecting on the value of the options. These results advance our understanding of people's reactions in the face of novelty. (PsycInfo Database Record (c) 2020 APA, all rights reserved).
- 1 October 2020
- Published by American Psychological Association (APA) in Journal of Experimental Psychology: General
- Vol. 149 (10) , 1878-1907
It’s new, but is it good? How generalization and uncertainty guide the exploration of novel options.
How do people decide whether to try out novel options as opposed to tried-and-tested ones? We argue that they infer a novel option's reward from contextual information learned from functional relations and take uncertainty into account when making a decision. We propose a Bayesian optimization model to describe their learning and decision making. This model relies on similarity-based learning of functional relationships between features and rewards, and a choice rule that balances exploration and exploitation by combining predicted rewards and the uncertainty of these predictions. Our model makes 2 main predictions. First, decision makers who learn functional relationships will generalize based on the learned reward function, choosing novel options only if their predicted reward is high. Second, they will take uncertainty about the function into account, and prefer novel options that can reduce this uncertainty. We test these predictions in 3 preregistered experiments in which we examine participants' preferences for novel options using a feature-based multiarmed bandit task in which rewards are a noisy function of observable features. Our results reveal strong evidence for functional exploration and moderate evidence for uncertainty-guided exploration. However, whether or not participants chose a novel option also depended on their attention, as well as reflecting on the value of the options. These results advance our understanding of people's reactions in the face of novelty. (PsycInfo Database Record (c) 2020 APA, all rights reserved).
- Open Access
- 24 January 2020
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 117 (6) , 3291-3300
Uncertainty in learning, choice, and visual fixation
Uncertainty plays a critical role in reinforcement learning and decision making. However, exactly how it influences behavior remains unclear. Multiarmed-bandit tasks offer an ideal test bed, since computational tools such as approximate Kalman filters can closely characterize the interplay between trial-by-trial values, uncertainty, learning, and choice. To gain additional insight into learning and choice processes, we obtained data from subjects’ overt allocation of gaze. The estimated value and estimation uncertainty of options influenced what subjects looked at before choosing; these same quantities also influenced choice, as additionally did fixation itself. A momentary measure of uncertainty in the form of absolute prediction errors determined how long participants looked at the obtained outcomes. These findings affirm the importance of uncertainty in multiple facets of behavior and help delineate its effects on decision making.
- 24 January 2020
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 117 (6) , 3291-3300
Open AccessUncertainty in learning, choice, and visual fixation
Uncertainty plays a critical role in reinforcement learning and decision making. However, exactly how it influences behavior remains unclear. Multiarmed-bandit tasks offer an ideal test bed, since computational tools such as approximate Kalman filters can closely characterize the interplay between trial-by-trial values, uncertainty, learning, and choice. To gain additional insight into learning and choice processes, we obtained data from subjects’ overt allocation of gaze. The estimated value and estimation uncertainty of options influenced what subjects looked at before choosing; these same quantities also influenced choice, as additionally did fixation itself. A momentary measure of uncertainty in the form of absolute prediction errors determined how long participants looked at the obtained outcomes. These findings affirm the importance of uncertainty in multiple facets of behavior and help delineate its effects on decision making.
- Open Access
- 1 October 2019
- Published by Springer Nature in Journal of Risk and Uncertainty
- Vol. 59 (2) , 171-184
Gender effects for loss aversion: Yes, no, maybe?
Gender effects in risk taking have attracted much attention by economists, and remain debated. Loss aversion—the stylized finding that a given loss carries substantially greater weight than a monetarily equivalent gain—is a fundamental driver of risk aversion. We deploy four definitions of loss aversion commonly used in the literature to investigate gender effects. Even though the definitions only differ in subtle ways, we find women to be more loss averse than men according to one definition, while another definition results in no gender differences, and the remaining two definitions point to women beinglessloss averse than men. Conceptually, these contradictory effects can be organized by systematic measurement error resulting from model mis-specifications relative to the true underlying decision process.
- 1 October 2019
- Published by Springer Nature in Journal of Risk and Uncertainty
- Vol. 59 (2) , 171-184
Open AccessGender effects for loss aversion: Yes, no, maybe?
Gender effects in risk taking have attracted much attention by economists, and remain debated. Loss aversion—the stylized finding that a given loss carries substantially greater weight than a monetarily equivalent gain—is a fundamental driver of risk aversion. We deploy four definitions of loss aversion commonly used in the literature to investigate gender effects. Even though the definitions only differ in subtle ways, we find women to be more loss averse than men according to one definition, while another definition results in no gender differences, and the remaining two definitions point to women beinglessloss averse than men. Conceptually, these contradictory effects can be organized by systematic measurement error resulting from model mis-specifications relative to the true underlying decision process.
- Open Access
Analysis of task-based functional MRI data preprocessed with fMRIPrep
Functional magnetic resonance imaging (fMRI) is a standard tool to investigate the neural correlates of cognition. fMRI noninvasively measures brain activity, allowing identification of patterns evoked by tasks performed during scanning. Despite the long history of this technique, the idiosyncrasies of each dataset have led to the use of ad-hoc preprocessing protocols customized for nearly every different study. This approach is time-consuming, error-prone, and unsuitable for combining datasets from many sources. Here we showcase fMRIPrep ( http://fmriprep.org ), a robust tool to prepare human fMRI data for statistical analysis. This software instrument addresses the reproducibility concerns of the established protocols for fMRI preprocessing. By leveraging the Brain Imaging Data Structure (BIDS) to standardize both the input datasets —MRI data as stored by the scanner— and the outputs —data ready for modeling and analysis—, fMRIPrep is capable of preprocessing a diversity of datasets without manual intervention. In support of the growing popularity of fMRIPrep , this protocol describes how to integrate the tool in a task-based fMRI investigation workflow.
Open AccessAnalysis of task-based functional MRI data preprocessed with fMRIPrep
Functional magnetic resonance imaging (fMRI) is a standard tool to investigate the neural correlates of cognition. fMRI noninvasively measures brain activity, allowing identification of patterns evoked by tasks performed during scanning. Despite the long history of this technique, the idiosyncrasies of each dataset have led to the use of ad-hoc preprocessing protocols customized for nearly every different study. This approach is time-consuming, error-prone, and unsuitable for combining datasets from many sources. Here we showcase fMRIPrep ( http://fmriprep.org ), a robust tool to prepare human fMRI data for statistical analysis. This software instrument addresses the reproducibility concerns of the established protocols for fMRI preprocessing. By leveraging the Brain Imaging Data Structure (BIDS) to standardize both the input datasets —MRI data as stored by the scanner— and the outputs —data ready for modeling and analysis—, fMRIPrep is capable of preprocessing a diversity of datasets without manual intervention. In support of the growing popularity of fMRIPrep , this protocol describes how to integrate the tool in a task-based fMRI investigation workflow.