RKRahul V. Kulkarni

Rahul V. Kulkarni

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Rahul V. Kulkarni has a total of 42 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Bootstrapped Reward Shaping

    by Jacob Adamczyk, Volodymyr Makarenko, Stas Tiomkin, Rahul V. Kulkarni

    In reinforcement learning, especially in sparse-reward domains, many environment steps are required to observe reward information. In order to increase the frequency of such observations, "potential-based reward shaping" (PBRS) has been proposed as a method of providing a more dense reward signal while leaving the optimal policy invariant. However, the required potential function must be carefully designed with task-dependent knowledge to not deter training performance. In this work, we propose a bootstrapped method of reward shaping, termed BS-RS, in which the agent's current estimate of the state-value function acts as the potential function for PBRS. We provide convergence proofs for the tabular setting, give insights into training dynamics for deep RL, and show that the proposed method improves training speed in the Atari suite.


  • Open Access

    EVAL: EigenVector-based Average-reward Learning

    by Jacob Adamczyk, Volodymyr Makarenko, Stas Tiomkin, Rahul V. Kulkarni

    In reinforcement learning, two objective functions have been developed extensively in the literature: discounted and averaged rewards. The generalization to an entropy-regularized setting has led to improved robustness and exploration for both of these objectives. Recently, the entropy-regularized average-reward problem was addressed using tools from large deviation theory in the tabular setting. This method has the advantage of linearity, providing access to both the optimal policy and average reward-rate through properties of a single matrix. In this paper, we extend that framework to more general settings by developing approaches based on function approximation by neural networks. This formulation reveals new theoretical insights into the relationship between different objectives used in RL. Additionally, we combine our algorithm with a posterior policy iteration scheme, showing how our approach can also solve the average-reward RL problem without entropy-regularization. Using classic control benchmarks, we experimentally find that our method compares favorably with other algorithms in terms of stability and rate of convergence.

  • Open Access

    Entropy regularized reinforcement learning using large deviation theory

    by Argenis Arriojas, Jacob Adamczyk, Stas Tiomkin, Rahul V. Kulkarni

    Reinforcement learning (RL) is an important field of research in machine learning that is increasingly being applied to complex optimization problems in physics. In parallel, concepts from physics have contributed to important advances in RL with developments such as entropy-regularized RL. While these developments have led to advances in both fields, obtaining analytical solutions for optimization in entropy-regularized RL is currently an open problem. In this paper, we establish a mapping between entropy-regularized RL and research in nonequilibrium statistical mechanics focusing on Markovian processes conditioned on rare events. In the long-time limit, we apply approaches from large deviation theory to derive exact analytical results for the optimal policy and optimal dynamics in Markov decision process (MDP) models of reinforcement learning. The results obtained lead to an analytical and computational framework for entropy-regularized RL which is validated by simulations. The mapping established in this work connects current research in reinforcement learning and nonequilibrium statistical mechanics, thereby opening avenues for the application of analytical and computational approaches from one field to cutting-edge problems in the other.


  • Open Access

    Entropy Regularized Reinforcement Learning Using Large Deviation Theory

    by Argenis Arriojas, Jacob Adamczyk, Stas Tiomkin, Rahul V. Kulkarni

    Reinforcement learning (RL) is an important field of research in machine learning that is increasingly being applied to complex optimization problems in physics. In parallel, concepts from physics have contributed to important advances in RL with developments such as entropy-regularized RL. While these developments have led to advances in both fields, obtaining analytical solutions for optimization in entropy-regularized RL is currently an open problem. In this paper, we establish a mapping between entropy-regularized RL and research in non-equilibrium statistical mechanics focusing on Markovian processes conditioned on rare events. In the long-time limit, we apply approaches from large deviation theory to derive exact analytical results for the optimal policy and optimal dynamics in Markov Decision Process (MDP) models of reinforcement learning. The results obtained lead to a novel analytical and computational framework for entropy-regularized RL which is validated by simulations. The mapping established in this work connects current research in reinforcement learning and non-equilibrium statistical mechanics, thereby opening new avenues for the application of analytical and computational approaches from one field to cutting-edge problems in the other.

  • Open Access

    Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning

    by Jacob Adamczyk, Argenis Arriojas, Stas Tiomkin, Rahul V. Kulkarni

    In reinforcement learning (RL), the ability to utilize prior knowledge from previously solved tasks can allow agents to quickly solve new problems. In some cases, these new problems may be approximately solved by composing the solutions of previously solved primitive tasks (task composition). Otherwise, prior knowledge can be used to adjust the reward function for a new problem, in a way that leaves the optimal policy unchanged but enables quicker learning (reward shaping). In this work, we develop a general framework for reward shaping and task composition in entropy-regularized RL. To do so, we derive an exact relation connecting the optimal soft value functions for two entropy-regularized RL problems with different reward functions and dynamics. We show how the derived relation leads to a general result for reward shaping in entropy-regularized RL. We then generalize this approach to derive an exact relation connecting optimal value functions for the composition of multiple tasks in entropy-regularized RL. We validate these theoretical contributions with experiments showing that reward shaping and task composition lead to faster learning in various settings.

  • Open Access

    Stochastic modeling of phenotypic switching and chemoresistance in cancer cell populations

    by Niraj Kumar, Gwendolyn M. Cramer, Seyed Alireza Zamani Dahaj, Bala Sundaram, Jonathan P. Celli, Rahul V. Kulkarni

    Phenotypic heterogeneity in cancer cells is widely observed and is often linked to drug resistance. In several cases, such heterogeneity in drug sensitivity of tumors is driven by stochastic and reversible acquisition of a drug tolerant phenotype by individual cells even in an isogenic population. Accumulating evidence further suggests that cell-fate transitions such as the epithelial to mesenchymal transition (EMT) are associated with drug resistance. In this study, we analyze stochastic models of phenotypic switching to provide a framework for analyzing cell-fate transitions such as EMT as a source of phenotypic variability in drug sensitivity. Motivated by our cell-culture based experimental observations connecting phenotypic switching in EMT and drug resistance, we analyze a coarse-grained model of phenotypic switching between two states in the presence of cytotoxic stress from chemotherapy. We derive analytical results for time-dependent probability distributions that provide insights into the rates of phenotypic switching and characterize initial phenotypic heterogeneity of cancer cells. The results obtained can also shed light on fundamental questions relating to adaptation and selection scenarios in tumor response to cytotoxic therapy.


  • Bioinformatic Approach for Prediction of CsrA/RsmA-Regulating Small RNAs in Bacteria

    by Carl T. Fakhry, Kourosh Zarringhalam, Rahul V. Kulkarni

    CsrA/RsmA is a RNA-binding protein that functions as a global regulator controlling important processes such as virulence, secondary metabolism, motility, and biofilm formation in diverse bacterial species. The activity of CsrA/RsmA is regulated by small RNAs that contain multiple binding sites for the protein. The expression of these noncoding RNAs effectively sequesters the protein and reduces free cellular levels of CsrA/RsmA. While multiple bacterial small RNAs that bind to and regulate CsrA/RsmA levels have been discovered, it is anticipated that there are several such small RNAs that remain undiscovered. To assist in the discovery of these small RNAs, we have developed a bioinformatics approach that combines sequence- and structure-based features to predict small RNA regulators of CsrA/RsmA. This approach analyzes structural motifs in the ensemble of low energy secondary structures of known small RNA regulators of CsrA/RsmA and trains a binary classifier on these features. The proposed machine learning approach leads to several testable predictions for small RNA regulators of CsrA/RsmA, thereby complementing and accelerating experimental efforts aimed at discovery of noncoding RNAs in the CsrA/RsmA pathway.


  • Open Access

    Prediction of bacterial small RNAs in the RsmA (CsrA) and ToxT pathways: a machine learning approach

    by Carl Tony Fakhry, Prajna Kulkarni, Ping Chen, Rahul Kulkarni, Kourosh Zarringhalam

    Small RNAs (sRNAs) constitute an important class of post-transcriptional regulators that control critical cellular processes in bacteria. Recent research using high-throughput transcriptomic approaches has led to a dramatic increase in the discovery of bacterial sRNAs. However, it is generally believed that the currently identified sRNAs constitute a limited subset of the bacterial sRNA repertoire. In several cases, sRNAs belonging to a specific class are already known and the challenge is to identify additional sRNAs belonging to the same class. In such cases, machine-learning approaches can be used to predict novel sRNAs in a given class. In this work, we develop novel bioinformatics approaches that integrate sequence and structure-based features to train machine-learning models for the discovery of bacterial sRNAs. We show that features derived from recurrent structural motifs in the ensemble of low energy secondary structures can distinguish the RNA classes with high accuracy. We apply this approach to predict new members in two broad classes of bacterial small RNAs: 1) sRNAs that bind to the RNA-binding protein RsmA/CsrA in diverse bacterial species and 2) sRNAs regulated by the master regulator of virulence, ToxT, in Vibrio cholerae. The involvement of sRNAs in bacterial adaptation to changing environments is an increasingly recurring theme in current research in microbiology. It is likely that future research, combining experimental and computational approaches, will discover many more examples of sRNAs as components of critical regulatory pathways in bacteria. We have developed a novel approach for prediction of small RNA regulators in important bacterial pathways. This approach can be applied to specific classes of sRNAs for which several members have been identified and the challenge is to identify additional sRNAs.


  • Open Access

    Stochastic gene expression conditioned on large deviations

    by Jordan M Horowitz, Rahul V Kulkarni

    The intrinsic stochasticity of gene expression can give rise to large fluctuations and rare events that drive phenotypic variation in a population of genetically identical cells. Characterizing the fluctuations that give rise to such rare events motivates the analysis of large deviations in stochastic models of gene expression. Recent developments in non-equilibrium statistical mechanics have led to a framework for analyzing Markovian processes conditioned on rare events and for representing such processes by conditioning-free driven Markovian processes. We use this framework, in combination with approaches based on queueing theory, to analyze a general class of stochastic models of gene expression. Modeling gene expression as a Batch Markovian Arrival Process (BMAP), we derive exact analytical results quantifying large deviations of time-integrated random variables such as promoter activity fluctuations. We find that the conditioning-free driven process can also be represented by a BMAP that has the same form as the original process, but with renormalized parameters. The results obtained can be used to quantify the likelihood of large deviations, to characterize system fluctuations conditional on rare events and to identify combinations of model parameters that can give rise to dynamical phase transitions in system dynamics.


  • Open Access

    Frequency modulation of stochastic gene expression bursts by strongly interacting small RNAs

    by Niraj Kumar, Tao Jia, Kourosh Zarringhalam, Rahul V. Kulkarni

    The sporadic nature of gene expression at the single-cell level—long periods of inactivity punctuated by bursts of mRNA or protein production—plays a critical role in diverse cellular processes. To elucidate the cellular role of bursting in gene expression, synthetic biology approaches have been used to design simple genetic circuits with bursty mRNA or protein production. Understanding how such genetic circuits can be designed with the ability to control burst-related parameters requires the development of quantitative stochastic models of gene expression. In this work, we analyze stochastic models for the regulation of gene expression bursts by strongly interacting small RNAs. For the parameter range considered, results based on mean-field approaches are significantly inaccurate and alternative analytical approaches are needed. Using simplifying approximations, we obtain analytical results for the corresponding steady-state distributions that are in agreement with results from stochastic simulations. These results indicate that regulation by small RNAs, in the strong interaction limit, can be used to effectively modulate the frequency of bursting. We explore the consequences of such regulation for simple genetic circuits involving feedback effects and switching between promoter states.