Masaru Shirasuna

Computational capacity and knowledge of humans are more limited than those of large language models (LLMs). However, in buzzer quizzes, human trivia experts can often identify the correct answer even from insufficient information such as only a few words. Investigating how they can make fast and accurate judgments is expected to highlight new characteristics of human intelligence, but little is known about that. In this exploratory and case-based analysis, we predicted that trivia experts and LLMs would differ in which words/phrases in a question are important for identifying the answer, and compared experts’ performance with LLMs’ performance in Japanese trivia questions through an LLM-as-a-judge approach: We regarded LLMs as evaluators and then used their outputs as comparative tools for experts’ evaluations. First, we constructed a quiz question processing system that tokenized question texts based on morphological analysis and then numerically evaluated the importance of each token using GPT-4o/GPT-4.1. Then, we conducted a behavioral experiment wherein actual trivia experts were asked to numerically evaluate the importance of each token, just as LLMs had performed. As a result, trivia experts treated a variety of words/phrases as important, even if each word/phrase did not appear to be strongly associated with the answer. Their evaluation scores tended to accumulate faster than those of the LLMs. This may indicate that trivia experts can make inductive inferences faster (e.g., finding a common concept even from few items). More advanced question-answering systems may be designed by applying trivia experts’ cognitive processes to LLMs’ information processing, and our findings may provide a scaffolding toward such goals.

Many workers today engage in straightforward judgment tasks, increasing the need for interventions to improve accuracy. We propose a resource-rational and psychohygienic intervention, “wait short time”, which introduces a brief pause before displaying alternatives. This pause works as a harmonious triad: it clears the mind of prior judgment bias, restores present attention, and prepares the mind for future judgments; and all this without additional instructions. Based on a resource rationality framework, cognitive costs (e.g., irritation, cognitive conflict) are associated with prolonged thinking because of humans limited cognitive resources. Therefore, there should be an appropriately short thinking time to achieve higher accuracy with minimal workload. We investigated the effectiveness of the proposed intervention both theoretically and empirically. The computer simulations demonstrated that, under assumptions of limited cognitive resources, there was an optimal time at the early stages for maximizing total benefits. The results of behavioral experiment was consistent with the theoretical findings: Providing a waiting time (1 s or 2.5 s) improved judgment accuracy, but cognitive conflicts increased over time and an unnecessarily long time (2.5 s) induced more subjective irritation. Consequently, an appropriate time (1 s) could enhance judgment accuracy with less workload. We discuss the implications and limitations of the proposed intervention.

When people have to solve many tasks, they can aggregate diverse individuals’ judgments using the majority rule, which often improves the accuracy of judgments (wisdom of crowds). When aggregating judgments, individuals’ subjective confidence is a useful cue for deciding which judgments to accept. However, can confidence in one task set predict performance not only in the same task set, but also in another? We examined this issue through computer simulations using behavioral data obtained from binary-choice experimental tasks. In our simulations, we developed a “training-test” approach: We split the questions used in the behavioral experiments into “training questions” (as questions to identify individuals’ confidence levels) and “test questions” (as questions to be solved), similar to the cross-validation method in machine learning. We found that (i) through analyses of behavioral data, confidence in a certain question could predict accuracy in the same question, but not always well in another question. (ii) Through a computer simulation for the accordance of two individuals’ judgments, individuals with high confidence in one training question tended to make less diverse judgments in other test questions. (iii) Through a computer simulation of group judgments, the groups constructed from individuals with high confidence in the training question(s) generally performed well; however, their performance sometimes largely decreased in the test questions especially when only one training question was available. These results suggest that when situations are highly uncertain, an effective strategy is to aggregate various individuals regardless of confidence levels in the training questions to avoid decreasing the group accuracy in test questions. We believe that our simulations, which follow a “training-test” approach, provide practical implications in terms of retaining groups’ ability to solve many tasks.

In group judgments in a binary choice task, the judgments of individuals with low confidence (i.e., they feel that the judgment was not correct) may be regarded as unreliable. Previous studies have shown that aggregating individuals’ diverse judgments can lead to high accuracy in group judgments, a phenomenon known as the wisdom of crowds. Therefore, if low-confidence individuals make diverse judgments between individuals and the mean of accuracy of their judgments is above the chance level (.50), it is likely that they will not always decrease the accuracy of group judgments. To investigate this issue, the present study conducted behavioral experiments using binary choice inferential tasks, and computer simulations of group judgments by manipulating group sizes and individuals’ confidence levels. Results revealed that (I) judgment patterns were highly similar between individuals regardless of their confidence levels; (II) the low-confidence group could make judgments as accurate as the high-confidence group, as the group size increased; and (III) even if there were low-confidence individuals in a group, they generally did not inhibit group judgment accuracy. The results suggest the usefulness of low-confidence individuals’ judgments in a group and provide practical implications for real-world group judgments.

Previous studies have shown that people often use heuristics in making inferences and that subjective memory experiences, such as recognition or familiarity of objects, can be valid cues for inferences. So far, many researchers have used the binary choice task in which two objects are presented as alternatives (e.g., “Which city has the larger population, city A or city B?”). However, objects can be presented not only as alternatives but also in a question (e.g., “Which country is city X in, country A or country B?”). In such a situation, people can make inferences based on the relationship between the object in the question and each object given as an alternative. In the present study, we call this type of task a “relationships‐comparison task.” We modeled the three inference strategies that people could apply to solve it (familiarity‐matching [FM; a new heuristic we propose in this study], familiarity heuristic [FH], and knowledge‐based inference [KI]) to examine people's inference processes. Through Studies 1, 2, and 3, we found that (a) people tended to rely on heuristics, and that FM (inferences based on similarity in familiarity between objects) well explained participants' inference patterns; (b) FM could work as an ecologically rational strategy for the relationships–comparison task since it could effectively reflect environmental structures, and that the use of FM could be highly replicable and robust; and (c) people could sometimes use a decision strategy like FM, even in their daily lives (consumer behaviors). The nature of the relationships–comparison task and human cognitive processes is discussed.

Previous studies have shown that when choosing one of two logically equivalent frames (e.g., “half full” or “half empty”), people tend to choose based on a reference point. For example, when the amount of water in a glass with 500 ml capacity was originally 0 ml (or 500 ml), and then increased (or decreased) to 250 ml, people tend to express the amount of water in the glass as “half full” (or “half empty”). In the present study, we examined whether participants explicitly made a frame choice based on the reference point. We conducted four behavioral experiments relating to frame choice tasks. Specifically, participants were presented with a story-based or prime-based reference point and then made a frame choice. Furthermore, participants provided their reasons for the choice. Our findings on frame choices and their reasons can be summarized as follows. First, when participants were presented with a story-based reference point, some of them reported that they made frame choices based on the reference point. Second, when a reference point was presented as a prime, participants’ frame choices were affected by this reference point. However, almost no participants reported that they made frame choices based on the reference point. These results indicate that the effect of reference points on frame choices is robust and that people do not always explicitly make frame choices based on the reference point.