LMLidiya Murakhovs'Ka

Lidiya Murakhovs'Ka

*This is an algorithmically generated author record based on publicly available information.

Lidiya Murakhovs'Ka has a total of 10 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment

    by Philippe Laban, Lidiya Murakhovs'Ka, Caiming Xiong, Chien-Sheng Wu

    The interactive nature of Large Language Models (LLMs) theoretically allows models to refine and improve their answers, yet systematic analysis of the multi-turn behavior of LLMs remains limited. In this paper, we propose the FlipFlop experiment: in the first round of the conversation, an LLM completes a classification task. In a second round, the LLM is challenged with a follow-up phrase like "Are you sure?", offering an opportunity for the model to reflect on its initial answer, and decide whether to confirm or flip its answer. A systematic study of ten LLMs on seven classification tasks reveals that models flip their answers on average 46% of the time and that all models see a deterioration of accuracy between their first and final prediction, with an average drop of 17% (the FlipFlop effect). We conduct finetuning experiments on an open-source LLM and find that finetuning on synthetically created data can mitigate - reducing performance deterioration by 60% - but not resolve sycophantic behavior entirely. The FlipFlop experiment illustrates the universality of sycophantic behavior in LLMs and provides a robust framework to analyze model behavior and evaluate future models.


  • Open Access

    Salespeople vs SalesBot: Exploring the Role of Educational Value in Conversational Recommender Systems

    by Lidiya Murakhovs'Ka, Philippe Laban, Tian Xie, Caiming Xiong, Chien-Sheng Wu

    Making big purchases requires consumers to research or consult a salesperson to gain domain expertise. However, existing conversational recommender systems (CRS) often overlook users' lack of background knowledge, focusing solely on gathering preferences. In this work, we define a new problem space for conversational agents that aim to provide both product recommendations and educational value through mixed-type mixed-initiative dialog. We introduce SalesOps, a framework that facilitates the simulation and evaluation of such systems by leveraging recent advancements in large language models (LLMs). We build SalesBot and ShopperBot, a pair of LLM-powered agents that can simulate either side of the framework. A comprehensive human study compares SalesBot against professional salespeople, revealing that although SalesBot approaches professional performance in terms of fluency and informativeness, it lags behind in recommendation quality. We emphasize the distinct limitations both face in providing truthful information, highlighting the challenges of ensuring faithfulness in the CRS context. We release our code and make all data available.

  • Open Access

    Marvista: Exploring the Design of a Human-AI Collaborative News Reading Tool

    by Xiang 'Anthony' Chen, Chien-Sheng Wu, Lidiya Murakhovs'Ka, Philippe Laban, Tong Niu, Wenhao Liu, Caiming Xiong

    We explore the design of Marvista -- a human-AI collaborative tool that employs a suite of natural language processing models to provide end-to-end support for reading online news articles. Before reading an article, Marvista helps a user plan what to read by filtering text based on how much time one can spend and what questions one is interested to find out from the article. During reading, Marvista helps the user reflect on their understanding of each paragraph with AI-generated questions. After reading, Marvista generates an explainable human-AI summary that combines both AI's processing of the text, the user's reading behavior, and user-generated data in the reading process. In contrast to prior work that offered (content-independent) interaction techniques or devices for reading, Marvista takes a human-AI collaborative approach that contributes text-specific guidance (content-aware) to support the entire reading process.


  • Open Access

    Salespeople vs SalesBot: Exploring the Role of Educational Value in Conversational Recommender Systems

    by Lidiya Murakhovs’ka, Philippe Laban, Tian Xie, Caiming Xiong, Chien-Sheng Wu

    Lidiya Murakhovs’ka, Philippe Laban, Tian Xie, Caiming Xiong, Chien-Sheng Wu. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.


  • Open Access

    Discord Questions: A Computational Approach To Diversity Analysis in News Coverage

    by Philippe Laban, Chien-Sheng Wu, Lidiya Murakhovs'Ka, Xiang 'Anthony' Chen, Caiming Xiong

    There are many potential benefits to news readers accessing diverse sources. Modern news aggregators do the hard work of organizing the news, offering readers a plethora of source options, but choosing which source to read remains challenging. We propose a new framework to assist readers in identifying source differences and gaining an understanding of news coverage diversity. The framework is based on the generation of Discord Questions: questions with a diverse answer pool, explicitly illustrating source differences. To assemble a prototype of the framework, we focus on two components: (1) discord question generation, the task of generating questions answered differently by sources, for which we propose an automatic scoring method, and create a model that improves performance from current question generation (QG) methods by 5%, (2) answer consolidation, the task of grouping answers to a question that are semantically similar, for which we collect data and repurpose a method that achieves 81% balanced accuracy on our realistic test set. We illustrate the framework's feasibility through a prototype interface. Even though model performance at discord QG still lags human performance by more than 15%, generated questions are judged to be more interesting than factoid questions and can reveal differences in the level of detail, sentiment, and reasoning of sources in news coverage.

  • Open Access

    MixQG: Neural Question Generation with Mixed Answer Types

    by Lidiya Murakhovs'Ka, Chien-Sheng Wu, Philippe Laban, Tong Niu, Wenhao Liu, Caiming Xiong

    Asking good questions is an essential ability for both human and machine intelligence. However, existing neural question generation approaches mainly focus on the short factoid type of answers. In this paper, we propose a neural question generator, MixQG, to bridge this gap. We combine 9 question answering datasets with diverse answer types, including yes/no, multiple-choice, extractive, and abstractive answers, to train a single generative model. We show with empirical results that our model outperforms existing work in both seen and unseen domains and can generate questions with different cognitive levels when conditioned on different answer types. Our code is released and well-integrated with the Huggingface library to facilitate various downstream applications.