Patrick Charles Kyllonen

Social and Emotional Learning (SEL) helps students to develop healthy identities, regulate emotions, build relationships, and make responsible decisions. Effective SEL instruction often relies on group-based activities; however, traditional classrooms without technological support make it difficult to capture the fine-grained interactions among students. As a result, teachers struggle to provide timely feedback and interventions. This paper reports on an ongoing project that leverages a computer-supported collaborative learning (CSCL) environment to implement 36 SEL lessons with fifth and sixth-grade students over the course of a semester. We describe the design and implementation of the curriculum and present preliminary findings from a pilot study. Early results indicate positive feedback from both teachers and students, along with a negative correlation between students’ off-topic chat behavior, characterized by nonsensical or “gibberish” messages, and their learning performance.

Theoretically, the generalized graded unfolding model (GGUM) is more flexible than the generalized partial credit model (GPCM), a dominance model. For item responses generated by the GPCM, the GGUM estimations can generate overlapping item response curves with those from the GPCM over a range of latent trait scores covering almost all of the population. The discrimination and category threshold estimates from the two models are approximately equal. It is necessary to use an informative prior around an extreme location (e.g., 4 for a positive GPCM item) or fix the extreme locations in the GGUM estimation of GPCM items to achieve the desired estimation. The simulation study and the applications on two real datasets support the theoretical claims. Various practical implications are discussed, and suggestions for future research are provided.

A process is proposed to create the one-dimensional expected item characteristic curve (ICC) and test characteristic curve (TCC) for each trait in multidimensional forced-choice questionnaires based on the Rank-2PL (two-parameter logistic) item response theory models for forced-choice items with two or three statements. Some examples of ICC and TCC plots from a real pair form and triplet form are provided, and they appear to help identify misfit trait scores at the item and test levels. TCC plots with negative statements changed to positive ones are also proposed.

The rank two-parameter logistic (Rank-2PL) item response theory models refer to a set of models applying the 2PL model in a sequential ranking process that occurs in forced-choice questionnaires. The multi-unidimensional pairwise preference with 2PL model (MUPP-2PL) is a Rank-2PL model for items with two statements. Focusing on items with three statements, we develop a maximum marginal likelihood estimation with an expectation-maximization algorithm to estimate item parameters and their standard errors. A simulation study is conducted to check parameter recovery, and then the model is applied to a real dataset. Finally, the findings are summarized and discussed, and future research is suggested.

Collaborative problem solving (CPS) is widely recognized as a critical 21st century skill. Efficiently coding communication data is an outstanding challenge in scaling up research on assessing CPS. This paper reports the findings on using ChatGPT to directly code CPS chat data by benchmarking performance across multiple datasets and coding frameworks. We found that ChatGPT-based coding outperformed human coding in tasks where the discussions were characterized by colloquial languages but fell short in tasks where the discussions dealt with specialized scientific terminology and contexts. The findings offer practical guidelines for researchers to develop strategies for efficient and scalable analysis of communication data from CPS tasks.

Users of forced-choice questionnaires (FCQs) to measure personality commonly assume statement parameter invariance across contexts—between Likert and forced-choice (FC) items and between different FC items that share a common statement. In this paper, an empirical study was designed to check these two assumptions for an FCQ assessment measuring interpersonal and intrapersonal skills. We compared parameters of common statements between two Likert forms and two FCQ forms with a block size of two (statement pairs) and among five FCQ pairs forms. In three of the five FCQ forms, statements were paired only with a statement they had appeared with in a triplet block. In the other two FCQ forms, statements were paired with statements they had not been paired with in a triplet block. This design allowed us to evaluate statement parameter changes due to changes in context. The results did not support the statement parameter invariance assumption between Likert and FC items or the assumption between FC items when recombining statements formed new items. However, the assumption between FC items was generally held for pairs formed by dropping a statement from a triplet item. There were some suggestions for sources of context effects, but the analyses were not definitive. Implications of the findings for test practice are discussed.

This paper presents the item and test information functions of the Rank two‐parameter logistic models (Rank‐2PLM) for items with two (pair) and three (triplet) statements in forced‐choice questionnaires. The Rank‐2PLM model for pairs is the MUPP‐2PLM (Multi‐Unidimensional Pairwise Preference) and, for triplets, is the Triplet‐2PLM. Fisher's information and directional information are described, and the test information for Maximum Likelihood (ML), Maximum A Posterior (MAP), and Expected A Posterior (EAP) trait score estimates is distinguished. Expected item/test information indexes at various levels are proposed and plotted to provide diagnostic information on items and tests. The expected test information indexes for EAP scores may be difficult to compute due to a typical test's vast number of item response patterns. The relationships of item/test information with discrimination parameters of statements, standard error, and reliability estimates of trait score estimates are discussed and demonstrated using real data. Practical suggestions for checking the various expected item/test information indexes and plots are provided.

The assessment of creativity presents major challenges. The many competing and complementary ideas on measuring creativity have resulted in a wide diversity of measures, making it difficult for potential users to decide on their appropriateness. Prior research has proposed creativity assessment taxonomies, but we argue that these have shortcomings because they often were not designed to (a) assess the essential assessment features and (b) are insufficiently specified for reliably categorizing extant measures. Based on prior categorization approaches, we propose a new framework for categorizing creativity measures including the following attributes: (a) measurement approach (self-report, other-report, ability tests), (b) construct (e.g., creative interests and attitudes, creative achievements, divergent thinking), (c) data type generated (e.g., questionnaire data vs. accomplishments counts), (d) prototypical scoring method (e.g., consensual assessment technique; CAT), and (e) psychometric problems. We identified 228 creativity measures appearing in the literature since 1900 and classified each measure according to their task attributes by two independent raters (rater agreement Cohen’s kappa .83 to 1.00 for construct). We provide a summary of convergent validity evidence and psychometric shortcomings. We conclude with recommendations for using the taxonomy and some psychometric desiderata for future research.

We evaluate the measurement invariance of a 48-item instrument designed to measure general social and emotional skills of youth in low resource environments. We refer to the skills measured as positive self-concept, negative self-concept, higher order thinking skills, and social and communication skills. These skills are often associated with economic development and can be used to evaluate programs designed to enhance economic development. Our evaluation is based on a sample of 1,794 in and out-of-school youth from Uganda and Guatemala’s Western Highlands. We conduct the analyses using a multiple group confirmatory factor analysis approach, breaking the sample by country, gender, and socio-economic status (high vs. low). Overall, our analysis points to strong invariance for all four measures across the different groups being compared. These findings contribute to the validity of the instrument as a tool for better understanding youth in diverse, developing economies.