Jun Yu

Adversarial training (AT) has been shown to effectively enhance a network's resilience against adversarial attack. However, conventional AT, which relies on a fixed pre-specified perturbation budget, suffers from several limitations when training robust models. First, enforcing the same perturbation budget across networks with different capacities leads to varying levels of robustness disparity between natural and robust accuracies, which deviates from the desired outcome of a robust network. Second, because the perturbation budget is fixed throughout training, the attack strength fails to scale adaptively with the evolving robustness of the model. This mismatch often results in robust overfitting and further degradation of adversarial robustness. To address these limitations, we propose a novel technique called Strength-Adaptive Adversarial Training (SAAT). In SAAT, the adversary incorporates an adversarial-loss constraint to guide the generation of adversarial training data. This constraint allows the perturbation budget to adapt dynamically based on the current training state, which effectively mitigates robust overfitting. Moreover, by explicitly regulating the attack strength through the adversarial loss, SAAT enables precise control over the robustness disparity between natural accuracy and adversarial robustness. Extensive experiments demonstrate that SAAT substantially improves adversarial robustness over standard AT.

In the era of booming large models, the significance of data scale in deep learning has been well-acknowledged. Nevertheless, obtaining large-scale datasets remains a challenging task. Large pre-trained diffusion models, boasting remarkable generative capabilities, offer a promising solution for dataset generation. This study zeroes in on the labor-intensive semantic segmentation task, where image annotation is a major bottleneck. We introduce a novel data generation pipeline leveraging multimodal conditional control. This pipeline not only enhances the universality and practicality of our method but also ensures high-precision alignment between generated images and corresponding masks. Specifically, we feed mask, Canny edge, and depth information into ControlNet and employ captions generated by the BLIP model as prompts to precisely control image generation. Notably, we pioneer the exploration of zero-shot generation in this context. This approach enables direct image generation without the need for fine-tuning or alignment with segmentation protocols. Experimental results demonstrate its effectiveness: in the CityScapes dataset, zero-shot generation leads to a 0.6% increase in the mean Intersection over Union (mIOU). Even without zero-shot generation, our method achieves significant mIOU improvements of up to 2.53% on the ADE20K dataset and 1.93% on the COCO-Stuff dataset. These findings highlight the potential of our proposed method in advancing semantic segmentation tasks.