YJYanwei Jia

Yanwei Jia

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Yanwei Jia has a total of 19 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • A General Framework for Importance Sampling with Markov Random Walks

    by Cheng-Der Fuh, Yanwei Jia, Steven Kou

    Although stochastic models driven by latent Markov processes are widely used, the classical importance sampling methods based on the exponential tilting for these models suffer from difficulties in computing associated eigenvalues and eigenfunctions and the plausibility of the indirect asymptotic large deviation regime for the variance of the estimator. We propose an importance sampling framework that twists the observable and latent processes separately using a link function that directly minimizes the estimator’s variance. An optimal link function is chosen within the locally asymptotically normal family. We show the logarithmic efficiency of the estimator. As applications, we estimate an overflow probability under a pandemic model and the CoVaR, a measurement of the co-dependent financial systemic risk; both are beyond the scope of traditional importance sampling methods due to nonlinear features.

  • Open Access

    When Does Reinforcement Learning Improve upon Estimate-Then-Plug-In for Portfolio Selection?

    by Min Dai, Yanwei Jia, Zhichao Lu

    We examine the relationship between reinforcement learning and estimate-then-plug-in methods for dynamic portfolio selection. We show that, given finite, discretely observed stock prices, a plug-in method that estimates parameters and computes the optimal strategy under a geometric Brownian motion specification is equivalent to a particular offline q-learning algorithm trained on the same dataset, provided that the true data-generating process has continuous paths-even when the specification is otherwise misspecified. This equivalence, however, breaks down in the presence of jumps. Simulations and empirical tests show that reinforcement learning outperforms plug-in methods under jumps, model misspecification, or market nonstationarity, possibly because reinforcement learning learns decision rules directly rather than an unnecessary full model.

  • Accuracy of Discretely Sampled Stochastic Policies in Continuous-Time Reinforcement Learning

    by Yanwei Jia, Du Ouyang, Yufei Zhang

    Stochastic policies (also known as relaxed controls) are widely used in continuous-time reinforcement learning algorithms. However, executing a stochastic policy and evaluating its performance in a continuous-time environment remain open challenges. This work introduces and rigorously analyzes a policy execution framework that samples actions from a stochastic policy at discrete time points and implements them as piecewise constant controls. We prove that as the sampling mesh size tends to zero, the controlled state process converges weakly to the dynamics with coefficients aggregated according to the stochastic policy. We explicitly quantify the convergence rate based on the regularity of the coefficients and establish an optimal first-order convergence rate for sufficiently regular coefficients. Additionally, we prove a [Formula: see text]-order weak convergence rate that holds uniformly over the sampling noise with high probability and establish a [Formula: see text]-order pathwise convergence for each realization of the system noise in the absence of volatility control. Building on these results, we analyze the bias and variance of various policy evaluation and policy gradient estimators based on discrete-time observations. Our results provide theoretical justification for the exploratory stochastic control framework in [H. Wang, T. Zariphopoulou, and X. Y. Zhou, J. Mach. Learn. Res., 21 (2020), pp. 1–34].


  • Open Access

    Continuous-Time Risk-Sensitive Reinforcement Learning via Quadratic Variation Penalty

    by Yanwei Jia

    This paper studies continuous-time risk-sensitive reinforcement learning (RL) under the entropy-regularized, exploratory diffusion process formulation with the exponential-form objective. The risk-sensitive objective arises either as the agent’s risk attitude or as a distributionally robust approach against the model uncertainty. Owing to the martingale perspective in Jia and Zhou (J Mach Learn Res 24(161):1–61, 2023), the risk-sensitive RL problem is shown to be equivalent to ensuring the martingale property of a process involving both the value function and the q-function, augmented by an additional penalty term: the quadratic variation of the value process, capturing the variability of the value-to-go along the trajectory. This characterization allows for the straightforward adaptation of existing RL algorithms developed for non-risk-sensitive scenarios to incorporate risk sensitivity by adding the realized variance of the value process. Additionally, I highlight that the conventional policy gradient representation is inadequate for risk-sensitive problems due to the nonlinear nature of quadratic variation; however, q-learning offers a solution and extends to infinite horizon settings. Finally, I prove the convergence of the proposed algorithm for Merton’s investment problem and quantify the impact of temperature parameter on the behavior of the learning procedure. I also conduct simulation experiments to demonstrate how risk-sensitive RL improves the finite-sample performance in the linear-quadratic control problem.


  • Open Access

    Merton's Problem with Recursive Perturbed Utility

    by Min Dai, Yuchao Dong, Yanwei Jia, Xunyu Zhou

    The classical Merton investment problem predicts deterministic, state-dependent portfolio rules; however, laboratory and field evidence suggests that individuals often prefer randomized decisions leading to stochastic and noisy choices. Fudenberg et al. (2015) develop the additive perturbed utility theory to explain the preference for randomization in the static setting, which, however, becomes ill-posed or intractable in the dynamic setting. We introduce the recursive perturbed utility (RPU), a special stochastic differential utility that incorporates an entropy-based preference for randomization into a recursive aggregator. RPU endogenizes the intertemporal trade-off between utilities from randomization and bequest via a discounting term dependent on past accumulated randomization, thereby avoiding excessive randomization and yielding a well-posed problem. In a general Markovian incomplete market with CRRA preferences, we prove that the RPU-optimal portfolio policy (in terms of the risk exposure ratio) is Gaussian and can be expressed in closed form, independent of wealth. Its variance is inversely proportional to risk aversion and stock volatility, while its mean is based on the solution to a partial differential equation. Moreover, the mean is the sum of a myopic term and an intertemporal hedging term (against market incompleteness) that intertwines with policy randomization. Finally, we carry out an asymptotic expansion in terms of the perturbed utility weight to show that the optimal mean policy deviates from the classical Merton policy at first order, while the associated relative wealth loss is of a higher order, quantifying the financial cost of the preference for randomization.


  • Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems

    by Yilie Huang, Yanwei Jia, Xun Yu Zhou

    We study reinforcement learning (RL) for a class of continuous-time linear-quadratic (LQ) control problems for diffusions, where states are scalar-valued and running control rewards are absent but volatilities of the state processes depend on both state and control variables. We apply a model-free approach that relies neither on knowledge of model parameters nor on their estimations, and devise an RL algorithm to learn the optimal policy parameter directly. Our main contributions include the introduction of an exploration schedule and a regret analysis of the proposed algorithm. We provide the convergence rate of the policy parameter to the optimal one and prove that the algorithm achieves a regret bound of [Formula: see text] up to a logarithmic factor, where [Formula: see text] is the number of learning episodes. We conduct a simulation study to validate the theoretical results and demonstrate the effectiveness and reliability of the proposed algorithm. We also perform numerical comparisons between our method and those of the recent model-based stochastic LQ RL studies adapted to the state- and control-dependent volatility setting, demonstrating a better performance of the former in terms of regret bounds.


  • Open Access

    Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty

    by Yanwei Jia

    This paper studies continuous-time risk-sensitive reinforcement learning (RL) under the entropy-regularized, exploratory diffusion process formulation with the exponential-form objective. The risk-sensitive objective arises either as the agent's risk attitude or as a distributionally robust approach against the model uncertainty. Owing to the martingale perspective in Jia and Zhou (2023), the risk-sensitive RL problem is shown to be equivalent to ensuring the martingale property of a process involving both the value function and the q-function, augmented by an additional penalty term: the quadratic variation of the value process, capturing the variability of the value-to-go along the trajectory. This characterization allows for the straightforward adaptation of existing RL algorithms developed for non-risk-sensitive scenarios to incorporate risk sensitivity by adding the realized variance of the value process. Additionally, I highlight that the conventional policy gradient representation is inadequate for risk-sensitive problems due to the nonlinear nature of quadratic variation; however, q-learning offers a solution and extends to infinite horizon settings. Finally, I prove the convergence of the proposed algorithm for Merton's investment problem and quantify the impact of temperature parameter on the behavior of the learning procedure. I also conduct simulation experiments to demonstrate how risk-sensitive RL improves the finite-sample performance in the linear-quadratic control problem.

  • Open Access

    A General Framework for Importance Sampling with Markov Random Walks

    by Cheng-Der Fuh, Yanwei Jia, Steven Kou

    Although stochastic models driven by latent Markov processes are widely used, the classical importance sampling methods based on the exponential tilting for these models suffer from difficulties in computing associated eigenvalues and eigenfunctions and the plausibility of the indirect asymptotic large deviation regime for the variance of the estimator. We propose an importance sampling framework that twists the observable and latent processes separately using a link function that directly minimizes the estimator's variance. An optimal link function is chosen within the locally asymptotically normal family. We show the logarithmic efficiency of the estimator. As applications, we estimate an overflow probability under a pandemic model and the CoVaR, a measurement of the co-dependent financial systemic risk; both are beyond the scope of traditional importance sampling methods due to nonlinear features.


  • Top Cited Papers

    Learning equilibrium mean‐variance strategy

    by Min Dai, Yuchao Dong, Yanwei Jia

    We study a dynamic mean‐variance portfolio optimization problem under the reinforcement learning framework, where an entropy regularizer is introduced to induce exploration. Due to the time–inconsistency involved in a mean‐variance criterion, we aim to learn an equilibrium policy. Under an incomplete market setting, we obtain a semi‐analytical, exploratory, equilibrium mean‐variance policy that turns out to follow a Gaussian distribution. We then focus on a Gaussian mean return model and propose a reinforcement learning algorithm to find the equilibrium policy. Thanks to a thoroughly designed policy iteration procedure in our algorithm, we prove the convergence of our algorithm under mild conditions, despite that dynamic programming principle and the usual policy improvement theorem failing to hold for an equilibrium policy. Numerical experiments are given to demonstrate our algorithm. The design and implementation of our reinforcement learning algorithm apply to a general market setup.


  • Herding in Probabilistic Forecasts

    by Yanwei Jia, Jussi Keppo, Ville Satopää

    Decision makers often ask experts to forecast a future state. Experts, however, can be biased. In the economics and psychology literature, one extensively studied behavioral bias is called herding. Under strong levels of herding, disclosure of public information may lower forecasting accuracy. This result, however, is derived only for point forecasts. In this paper, we consider experts’ probabilistic forecasts under herding, find a closed-form expression for the first two moments of a unique equilibrium forecast, and show that the experts report too similar locations and inflate the variance of their forecasts because of herding. Furthermore, we show that the negative externality of public information no longer holds. In addition to reacting to new information as expected, probabilistic forecasts contain more information about the experts’ full beliefs and interpersonal structure. This facilitates model estimation. To this end, we consider a one-shot setting with one forecast per expert and show that our model is identifiable up to an infinite number of solutions based on point forecasts but up to two solutions based on probabilistic forecasts. We then provide a Bayesian estimation procedure for these two solutions and apply it to economic forecasting data collected by the European Central Bank and the Federal Reserve Bank of Philadelphia. We find that, on average, the experts invest around 19% of their efforts into making similar forecasts. The level of herding shows an increasing trend from 1999 to 2007 but drops sharply during the financial crisis of 2007–2009 and then rises again until 2019.This paper was accepted by Yan Chen, behavioral economics and decision analysis.Supplemental Material: The electronic companion and data are available at https://doi.org/10.1287/mnsc.2022.4487 .