AngleRoCL: Angle-Robust Concept Learning for Physically View-Invariant T2I Adversarial Patches
Abstract
Cutting-edge works have demonstrated that text-to-image (T2I) diffusion models can generate adversarial patches that mislead state-of-the-art object detectors in the physical world, revealing detectors' vulnerabilities and risks. However, these methods neglect the T2I patches' attack effectiveness when observed from different views in the physical world (i.e., angle robustness of the T2I adversarial patches). In this paper, we study the angle robustness of T2I adversarial patches comprehensively, revealing their angle-robust issues, demonstrating that texts affect the angle robustness of generated patches significantly, and task-specific linguistic instructions fail to enhance the angle robustness. Motivated by the studies, we introduce Angle-Robust Concept Learning (AngleRoCL), a simple and flexible approach that learns a generalizable concept (i.e., text embeddings in implementation) representing the capability of generating angle-robust patches. The learned concept can be incorporated into textual prompts and guides T2I models to generate patches with their attack effectiveness inherently resistant to viewpoint variations. Through extensive simulation and physical-world experiments on five SOTA detectors across multiple views, we demonstrate that AngleRoCL significantly enhances the angle robustness of T2I adversarial patches compared to baseline methods. Our patches maintain high attack success rates even under challenging viewing conditions, with over 50% average relative improvement in attack effectiveness across multiple angles. This research advances the understanding of physically angle-robust patches and provides insights into the relationship between textual concepts and physical properties in T2I-generated contents. We released our code at https://github.com/tsingqguo/anglerocl.
Empirical Studies on Angle Robustness
Average detection confidence at different viewing angles across image sets generated by given attack strategy. The y-axis labels indicate average detection confidence and the x-axis labels indicate different viewing angles. The rightmost panel demonstrates removed features: Color (C), Pattern (P), Shape (S), and Text (T). Patch generated by (a) NDDA prompt and (b) Task-specific instruction prompt.
Methodology: Angle-Robust Concept Learning (AngleRoCL)
(a) Pipeline of our angle-robust concept learning (AngleRoCL). The latent code represents the learned concept <angle-robust>. (b) Shows the inference results when we use the learned concept.
Experimental Results
Digital Environment Results: Angle-Aware Attack Success Rate (AASR) across five detectors in six environments, measured from -90° to 90° with 1° intervals. AngleRoCL significantly enhances angle robustness with 51.4% improvement for NDDA and 23.8% for MAGIC.
Physical Environment Results: AASR across five detectors, measured from -70° to 70° with 10° intervals. AngleRoCL achieves 82.4% improvement for NDDA and 189.9% for MAGIC in real-world scenarios.
Ablation Study and Discussion
(a) Direct optimization shows severe overfitting to training angles (orange) and trained detector (YOLOv5). Our AngleRoCL maintains consistent performance across all conditions. (b) Cosine similarity between learned <angle-robust> embedding and top-correlated tokens, revealing meaningful associations with robustness-related features.
Video Presentation
BibTeX
@inproceedings{Ji2024AngleRoCL,
title={AngleRoCL: Angle-Robust Concept Learning for Physically View-Invariant T2I Adversarial Patches},
author={Ji, Wenjun and Fu, Yuxiang and Ying, Luyang and Fan, Deng-Ping and Wang, Yuyi and Cheng, Ming-Ming and Tsang, Ivor and Guo, Qing},
booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
year={2025}
}