Yuan Tian

Most video compression methods focus on human visual perception, neglecting semantic preservation. This leads to severe semantic loss during the compression, hampering downstream video analysis tasks. In this paper, we propose a Masked Video Modeling (MVM)-powered compression framework that particularly preserves video semantics, by jointly mining and compressing the semantics in a self-supervised manner. While MVM is proficient at learning generalizable semantics through the masked patch prediction task, it may also encode non-semantic information like trivial textural details, wasting bitcost and bringing semantic noises. To suppress this, we explicitly regularize the non-semantic entropy of the compressed video in the MVM token space. The proposed framework is instantiated as a simple Semantic-Mining-then-Compression (SMC) model. Furthermore, we extend SMC as an advanced SMC++ model from several aspects. First, we equip it with a masked motion prediction objective, leading to better temporal semantic learning ability. Second, we introduce a Transformer-based compression module, to improve the semantic compression efficacy. Considering that directly mining the complex redundancy among heterogeneous features in different coding stages is non-trivial, we introduce a compact blueprint semantic representation to align these features into a similar form, fully unleashing the power of the Transformer-based compression module. Extensive results demonstrate the proposed SMC and SMC++ models show remarkable superiority over previous traditional, learnable, and perceptual quality-oriented video codecs, on three video analysis tasks and seven datasets.

Dysregulation of the bone marrow (BM) niche in multiple myeloma (MM) alters the composition and state of resident immune cells, potentially impeding antitumor immunity. One common mechanism of immune inhibition in solid tumors is the induction of exhaustion in tumor-specific T cells. However, the extent of T-cell exhaustion is not well characterized in MM. As the specific mechanisms of immune evasion are critical for devising effective therapeutic strategies, we deeply profiled the CD8+ T-cell compartment of patients with newly diagnosed MM (NDMM) for evidence of T-cell activation and exhaustion. We applied single-cell multiomic sequencing and mass cytometry to longitudinal BM and peripheral blood (PB) samples taken from time points spanning from diagnosis to induction therapy, autologous stem cell transplant, and maintenance therapy. We identified an exhausted-like population that lacked several canonical exhaustion markers, was not significantly enriched in patients with NDMM, and consisted of small, nonpersistent clonotypes. We also observed an activated population with increased frequency in the PB of patients with NDMM exhibiting phenotypic and clonal features consistent with homeostatic, cytokine-driven activation. As an orthogonal measurement of T-cell exhaustion, we performed intracellular cytokine staining and found that patients with NDMM lacked functionally exhausted T cells. In summary, there was no evidence of “tumor-experienced” T cells displaying hallmarks of terminal exhaustion and/or antigen-driven activation/expansion in patients with NDMM at any time point.

The outstanding performance of Large Multimodal Models (LMMs) has made them widely applied in vision-related tasks. However, various corruptions in the real world mean that images will not be as ideal as in simulations, presenting significant challenges for the practical application of LMMs. To address this issue, we introduce R-Bench, a benchmark focused on the Real-world Robustness of LMMs. Specifically, we: (a) model the complete link from user capture to LMMs reception, comprising 33 corruption dimensions, including 7 steps according to the corruption sequence, and 7 groups based on low-level attributes; (b) collect reference/distorted image dataset before/after corruption, including 2,970 question-answer pairs with human labeling; (c) propose comprehensive evaluation for absolute/relative robustness and benchmark 20 mainstream LMMs. Results show that while LMMs can correctly handle the original reference images, their performance is not stable when faced with distorted images, and there is a significant gap in robustness compared to the human visual system. We hope that R-Bench will inspire improving the robustness of LMMs, extending them from experimental simulations to the real-world application.

Large Multimodal Models (LMMs) have demonstrated impressive capabilities across a wide range of multimodal tasks, achieving ever-increasing performance on various evaluation benchmarks. However, existing benchmarks are typically static and often overlap with pre-training datasets, leading to fixed complexity constraints and substantial data contamination issues. Meanwhile, manually annotated datasets are labor-intensive, time-consuming, and subject to human bias and inconsistency, leading to reliability and reproducibility issues. To address these problems, we propose a fully dynamic multimodal evaluation framework, named Open-ended Visual Puzzle Generation (OVPG), which aims to generate fresh, diverse, and verifiable evaluation data automatically in puzzle-solving tasks. Specifically, the OVPG pipeline consists of a raw material sampling module, a visual content generation module, and a puzzle rule design module, which ensures that each evaluation instance is primitive, highly randomized, and uniquely solvable, enabling continual adaptation to the evolving capabilities of LMMs. Built upon OVPG, we construct PuzzleBench, a dynamic and scalable benchmark comprising 11,840 VQA samples. It features six carefully designed puzzle tasks targeting three core LMM competencies, visual recognition, logical reasoning, and context understanding. PuzzleBench differs from static benchmarks that quickly become outdated. It enables ongoing dataset refreshing through OVPG and a rich set of open-ended puzzle designs, allowing seamless adaptation to the evolving capabilities of LMMs.

Video compression is indispensable to most video analysis systems. Despite saving the transportation bandwidth, it also deteriorates downstream video understanding tasks, especially at low-bitrate settings. To systematically investigate this problem, we first thoroughly review the previous methods, revealing that three principles, i.e. , task-decoupled, label-free, and data-emerged semantic prior, are critical to a machine-friendly coding framework but are not fully satisfied so far. In this paper, we propose a traditional-neural mixed coding framework that simultaneously fulfills all these principles, by taking advantage of both traditional codecs and neural networks (NNs). On one hand, the traditional codecs can efficiently encode the pixel signal of videos but may distort the semantic information. On the other hand, highly non-linear NNs are proficient in condensing video semantics into a compact representation. The framework is optimized by ensuring that a transportation-efficient semantic representation of the video is preserved w.r.t. the coding procedure, which is spontaneously learned from unlabeled data in a self-supervised manner. The videos collaboratively decoded from two streams (codec and NN) are of rich semantics, as well as visually photo-realistic, empirically boosting several mainstream downstream video analysis task performances without any post-adaptation procedure. Furthermore, by introducing the attention mechanism and adaptive modeling scheme, the video semantic modeling ability of our approach is further enhanced. Fianlly, we build a low-bitrate video understanding benchmark with three downstream tasks on eight datasets, demonstrating the notable superiority of our approach. All codes, data, and models will be open-sourced for facilitating future research.

With the widespread adoption of digital signature technology, when a signer needs to perform numerous signature activities but has limited computing power, the efficiency of the general signature scheme can be severely restricted. To address this challenge, the linearly homomorphic signature scheme with a designated combiner (LHSDC) allows the signer to transfer the primary computing task to a cloud server, thereby reducing the computational burden on the signer. The signature algorithm of LHSDC comprises two parts: the homomorphic compression function and the binding function. However, due to the high computational requirements of these two parts, existing schemes still suffer from low efficiency for signers. In this paper, an efficient linearly homomorphic signature scheme with a designated combiner (ELHSDC) is presented, which is tailored for secure proxy signature scenarios. Firstly, we enhance the homomorphic compression function in the signature algorithm by reducing the number of exponential operations required from n to 1. Secondly, we simplify the binding function of the signature algorithm by enabling the signer to directly generate a secret parameter that is shared with the designated combiner, thus eliminating the need for public and private keys for the designated combiner. As a result of the above improvements, our scheme significantly reduces the signer’s amount of calculation during a complete signature process by reducing (n-1) exponential operations, 1 pairing operation, and 2 hash operations, at the expense of only 1 multiplication operation. Finally, the scheme is proved secure against existential forgery on adaptively chosen message attacks under the random oracle model.

Video rescaling has recently drawn extensive attention for its practical applications such as video compression. Compared to video super-resolution, which focuses on upscaling bicubic-downscaled videos, video rescaling methods jointly optimize a downscaler and a upscaler. However, the inevitable loss of information during downscaling makes the upscaling procedure still ill-posed. Furthermore, the network architecture of previous methods mostly relies on convolution to aggregate information within local regions, which cannot effectively capture the relationship between distant locations. To address the above two issues, we propose a unified video rescaling framework by introducing the following designs. First, we propose to regularize the information of the downscaled videos via a contrastive learning framework, where, particularly, hard negative samples for learning are synthesized online. With this auxiliary contrastive learning objective, the downscaler tends to retain more information that benefits the upscaler. Second, we present a selective global aggregation module (SGAM) to efficiently capture long-range redundancy in high-resolution videos, where only a few representative locations are adaptively selected to participate in the computationally-heavy self-attention (SA) operations. SGAM enjoys the efficiency of the sparse modeling scheme while preserving the global modeling capability of SA. We refer to the proposed framework as Contrastive Learning framework with Selective Aggregation (CLSA) for video rescaling. Comprehensive experimental results show that CLSA outperforms video rescaling and rescaling-based video compression methods on five datasets, achieving state-of-the-art performance.