Neeti Pokharna
Research Subjects
Neeti Pokharna has a total of 11 co-authors. Here are the top co-authors by number of shared publications:
- Aleksei Ustimenko
ShareChat, United Kingdom
3 shared publications - Olivier Jeunen
Independent Researcher, Antwerp, Belgium
3 shared publications - Christopher Green
ShareChat, USA
1 shared publication - Danish Kalim
Sharechat, Bengaluru, India
1 shared publication - Himanshu Doi
ShareChat, India
1 shared publication - Hitesh Sagtani
Sharechat, Bengaluru, India
1 shared publication
Latest Publications
- Open Access
- 19 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 4832-4836
Variance Reduction for Heavy-Tailed Monetization Metrics in Ranking Experiments via Post-Stratification
- 19 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 4832-4836
Open AccessVariance Reduction for Heavy-Tailed Monetization Metrics in Ranking Experiments via Post-Stratification
- Open Access
- 8 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 816-818
Powerful A/B-Testing Metrics and Where to Find Them
Online controlled experiments, colloquially known as A/B-tests, are the bread and butter of real-world recommender system evaluation. Typically, end-users are randomly assigned some system variant, and a plethora of metrics are then tracked, collected, and aggregated throughout the experiment. A North Star metric (e.g. long-term growth or revenue) is used to assess which system variant should be deemed superior. As a result, most collected metrics are supporting in nature, and serve to either (i) provide an understanding of how the experiment impacts user experience, or (ii) allow for confident decision-making when the North Star metric moves insignificantly (i.e. a false negative or type-II error). The latter is not straightforward: suppose a treatment variant leads to fewer but longer sessions, with more views but fewer engagements; should this be considered a positive or negative outcome? The question then becomes: how do we assess a supporting metric’s utility when it comes to decision-making using A/B-testing? Online platforms typically run dozens of experiments at any given time. This provides a wealth of information about interventions and treatment effects that can be used to evaluate metrics’ utility for online evaluation. We propose to collect this information and leverage it to quantify type-I, type-II, and type-III errors for the metrics of interest, alongside a distribution of measurements of their statistical power (e.g. z-scores and p-values). We present results and insights from building this pipeline at scale for two large-scale short-video platforms: ShareChat and Moj; leveraging hundreds of past experiments to find online metrics with high statistical power.
- 8 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 816-818
Open AccessPowerful A/B-Testing Metrics and Where to Find Them
Online controlled experiments, colloquially known as A/B-tests, are the bread and butter of real-world recommender system evaluation. Typically, end-users are randomly assigned some system variant, and a plethora of metrics are then tracked, collected, and aggregated throughout the experiment. A North Star metric (e.g. long-term growth or revenue) is used to assess which system variant should be deemed superior. As a result, most collected metrics are supporting in nature, and serve to either (i) provide an understanding of how the experiment impacts user experience, or (ii) allow for confident decision-making when the North Star metric moves insignificantly (i.e. a false negative or type-II error). The latter is not straightforward: suppose a treatment variant leads to fewer but longer sessions, with more views but fewer engagements; should this be considered a positive or negative outcome? The question then becomes: how do we assess a supporting metric’s utility when it comes to decision-making using A/B-testing? Online platforms typically run dozens of experiments at any given time. This provides a wealth of information about interventions and treatment effects that can be used to evaluate metrics’ utility for online evaluation. We propose to collect this information and leverage it to quantify type-I, type-II, and type-III errors for the metrics of interest, alongside a distribution of measurements of their statistical power (e.g. z-scores and p-values). We present results and insights from building this pipeline at scale for two large-scale short-video platforms: ShareChat and Moj; leveraging hundreds of past experiments to find online metrics with high statistical power.
- Open Access
- 15 December 2023
- Published by Association for Computing Machinery (ACM)
- p. 136-141
On Gradient Boosted Decision Trees and Neural Rankers: A Case-Study on Short-Video Recommendations at ShareChat
- 15 December 2023
- Published by Association for Computing Machinery (ACM)
- p. 136-141
Open AccessOn Gradient Boosted Decision Trees and Neural Rankers: A Case-Study on Short-Video Recommendations at ShareChat