Zhimin Zhao

A Software Bill of Materials (SBOM) provides transparency by documenting software component metadata and dependencies. However, SBOM adoption depends on tool ecosystems. With two dominant formats: SPDX and CycloneDX - the ecosystems vary significantly in maturity, tool support, and community engagement. We conduct a quantitative comparison of use cases for 170 publicly advertised SBOM tools, identifying enhancement areas for each format. We compare health metrics of both ecosystems (171 CycloneDX versus 470 SPDX tools) to evaluate robustness and maturity. We quantitatively compare 36,990 issue reports from open-source tools to identify challenges and development opportunities. Finally, we investigate the top 250 open-source projects using each tool ecosystem and compare their health metrics. Our findings reveal distinct characteristics: projects using CycloneDX tools demonstrate higher developer engagement and certain health indicators, while SPDX tools benefit from a more mature ecosystem with broader tool availability and established industry adoption. This research provides insights for developers, contributors, and practitioners regarding complementary strengths of these ecosystems and identifies opportunities for mutual enhancement.

Foundation Models (FMs), such as OpenAI's GPT, are fundamentally transforming the practice of software engineering by enabling the development of \emph{FMware} -- applications and infrastructures built around these models. FMware systems now support tasks such as code generation, natural-language interaction, knowledge integration, and multi-modal content creation, underscoring their disruptive impact on current software engineering workflows. However, the design, implementation, and evolution of FMware present significant new challenges, particularly across cloud-based and on-premise platforms where goals, processes, and tools often diverge from those of traditional software development. To our knowledge, this is the first large-scale analysis of FMware development across both cloud-based platforms and open-source repositories. We empirically investigate the FMware ecosystem through three focus areas: (1) the most common application domains of FMware, (2) the key challenges developers encounter, and (3) the types of issues that demand the greatest effort to resolve. Our analysis draws on data from GitHub repositories and from leading FMware platforms, including HuggingFace, GPTStore, Ora, and Poe. Our findings reveal a strong focus on education, content creation, and business strategy, alongside persistent technical challenges in memory management, dependency handling, and tokenizer configuration. On GitHub, bug reports and core functionality issues are the most frequently reported problems, while code review, similarity search, and prompt template design are the most time-consuming to resolve. By uncovering developer practices and pain points, this study points to opportunities to improve FMware tools, workflows, and community support, and provides actionable insights to help guide the future of FMware development.

Foundation models (FM), such as large language models (LLMs), which are large-scale machine learning (ML) models, have demonstrated remarkable adaptability in various downstream software engineering (SE) tasks, such as code completion, code understanding, and software development. As a result, FM leaderboards have become essential tools for SE teams to compare and select the best third-party FMs for their specific products and purposes. However, the lack of standardized guidelines for FM evaluation and comparison threatens the transparency of FM leaderboards and limits stakeholders’ ability to perform effective FM selection. As a first step towards addressing this challenge, our research focuses on understanding how these FM leaderboards operate in real-world scenarios (“leaderboard operations”) and identifying potential pitfalls and areas for improvement (“leaderboard smells”). In this regard, we collect up to 1, 045 FM leaderboards from five different sources: GitHub, Hugging Face Spaces, Papers With Code, spreadsheet and independent platform, to examine their documentation and engage in direct communication with leaderboard operators to understand their workflows. Through card sorting and negotiated agreement, we identify five distinct workflow patterns and develop a domain model that captures the key components and their interactions within these workflows. We then identify eight unique types of leaderboard smells in LBOps. By mitigating these smells, SE teams can improve transparency, accountability, and collaboration in current LBOps practices, fostering a more robust and responsible ecosystem for FM comparison and selection.

In machine learning (ML), efficient asset management, including ML models, datasets, algorithms, and tools, is vital for resource optimization, consistent performance, and a streamlined development lifecycle. This enables quicker iterations, adaptability, reduced development-to-deployment time, and reliable outputs. Despite existing research, a significant knowledge gap remains in operational challenges like model versioning, data traceability, and collaboration, which are crucial for the success of ML projects. Our study aims to address this gap by analyzing 15,065 posts from developer forums and platforms, employing a mixed-method approach to classify inquiries, extract challenges using BERTopic, and identify solutions through open card sorting and BERTopic clustering. We uncover 133 topics related to asset management challenges, grouped into 16 macro-topics, with software dependency, model deployment, and model training being the most discussed. We also find 79 solution topics, categorized under 18 macro-topics, highlighting software dependency, feature development, and file management as key solutions. This research underscores the need for further exploration of identified pain points and the importance of collaborative efforts across academia, industry, and the research community.