Newly Blog

a) data selection, coreset selection, dataset pruning: select a subset of training data

• survey on coreset selection: https://arxiv.org/pdf/2505.17799

• some papers of dataset pruning for generative model:

    * Li, Yize, et al. "Pruning then reweighting: Towards data-efficient training of diffusion models." ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025.
  
    * Moser, Brian B., Federico Raue, and Andreas Dengel. "A study in dataset pruning for image super-resolution." International Conference on Artificial Neural Networks. Cham: Springer Nature Switzerland, 2024.
  

• dataset quantization: divide the training set into different bins and select representative samples in each bin. It could be used as a data selection strategy.

• data attribution: survey on data attribution: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5451054. It could be used as a measurement for data selection. Some papers of data attribution for generative model:

    * Georgiev, Kristian, et al. "The journey, not the destination: How data guides diffusion models." arXiv preprint arXiv:2312.06205 (2023).
    * Zheng, Xiaosen, et al. "Intriguing properties of data attribution on diffusion models." arXiv preprint arXiv:2311.00500 (2023).
    * Lin, Jinxu, et al. "Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Models." arXiv preprint arXiv:2410.18639 (2024).
  

b) dataset distillation: optimize the training set, the optimized training images are not realistic images. There is no work using distilled images to train generative model.

a) clustering: select representative samples and remove outliers. clustering based on loss, gradient, etc.

b) data contribution: measure the contribution of each sample

• The performance difference using or without using this sample

c) learn the weights of training samples: train with weighted loss and test on the validation test

Global color mapping:

• 3D LUT: [1], [2] non-uniform LUT
• curve function: [1]
• linear transformation: [1]

Local color mapping:

• 3D LUT: [1]
• curve function: [1], DCE[2]
• linear transformation: HDRNet[1]

• image classication: CLIP [1], learnable prompt [2]

• video classification: ActionCLIP [3]

• object detection: ViLD [4], ZSD-YOLO [6]

• segmentation: [8] [9]

• visual grounding: CPT [5]

• image translation: StyleClip [7]

Reference

[1] Radford, Alec, et al. “Learning transferable visual models from natural language supervision.” arXiv preprint arXiv:2103.00020 (2021).

[2] Zhou, Kaiyang, et al. “Learning to Prompt for Vision-Language Models.” arXiv preprint arXiv:2109.01134 (2021).

[3] Wang, Mengmeng, Jiazheng Xing, and Yong Liu. “ActionCLIP: A New Paradigm for Video Action Recognition.” arXiv preprint arXiv:2109.08472 (2021).

[4] Gu, Xiuye, et al. “Zero-Shot Detection via Vision and Language Knowledge Distillation.” arXiv preprint arXiv:2104.13921 (2021).

[5] Yao, Yuan, et al. “CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models.” arXiv preprint arXiv:2109.11797 (2021).

[6] Xie, Johnathan, and Shuai Zheng. “ZSD-YOLO: Zero-Shot YOLO Detection using Vision-Language KnowledgeDistillation.” arXiv preprint arXiv:2109.12066 (2021).

[7] Patashnik, Or, et al. “Styleclip: Text-driven manipulation of stylegan imagery.” ICCV, 2021.

[8] Xu, Mengde, et al. “A simple baseline for zero-shot semantic segmentation with pre-trained vision-language model.” arXiv preprint arXiv:2112.14757 (2021).

[9] Lüddecke, Timo, and Alexander Ecker. “Image Segmentation Using Text and Image Prompts.” CVPR, 2022.

• Typical works: CapsNet [1], CapProNet [2] [code]

• The robustness of Capsule network: [3]

Reference

[1] Sabour S, Frosst N, Hinton G E. Dynamic routing between capsules[C]//Advances in Neural Information Processing Systems. 2017: 3856-3866.

[2] Zhang L, Edraki M, Qi G J. CapProNet: Deep feature learning via orthogonal projections onto capsule subspaces[J]. arXiv preprint arXiv:1805.07621, 2018.

[3] Jindong Gu, Volker Tresp, Han Hu, “Capsule Network is Not More Robust than Convolutional Network”, CVPR 2021.

1. Use the first few layers to simulate neuron activation in human brain [1]

2. Use the attention learnt by network to mimick human attention [2]

Reference

[1] Dapello, Joel, et al. “Simulating a primary visual cortex at the front of CNNs improves robustness to image perturbations.” BioRxiv (2020).

[2] Linsley, Drew, et al. “Learning what and where to attend.” arXiv preprint arXiv:1805.08819 (2018).

Attention in CNN:

According to [4], attention can be categorized into bottom-up attention (visual saliency, unsupervised) and top-down attention (task-driven, supervised).

According to [5], attention can be categorized into forward attention, post-hoc attention, and query-based attention.

• forward attention: spatial attention [16], channel attention [10] [17] [18], full attention [11], Deformable CNN v1 [8] v2 [9],

• post-hoc attention: CAM [6], GradCAM [7], scoreCAM [14], trainable CAM [20][21]

• query-based attention: [5]

• erase attention: [19] [12] [13]

• high-order attention [15]

Attention in RNN:

survey paper: survey on the attention based RNN model and its applications in computer vision [1]

• soft/hard attention: binary weight or soft weight

• item-wise/location-wise attention: location-wise attention is to convert an image to a sequence of local regions, which is essentially item-wise.

Earliest papers [2] [3] are basically the same except design specs of RNN unit.

Reference

[1] Wang, Feng, and David MJ Tax. “Survey on the attention based RNN model and its applications in computer vision.” arXiv preprint arXiv:1601.06823 (2016).

[2] Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. “Neural machine translation by jointly learning to align and translate.” arXiv preprint arXiv:1409.0473 (2014).

[3] Vinyals, Oriol, et al. “Grammar as a foreign language.” NIPS, 2015.

[4] Drew Linsley, Dan Shiebler, Sven Eberhardt, Thomas Serre: Learning what and where to attend. ICLR, 2019.

[5] Saumya Jetley, Nicholas A. Lord, Namhoon Lee, Philip H. S. Torr: Learn to Pay Attention. ICLR, 2018.

[6] Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, Antonio Torralba: Learning Deep Features for Discriminative Localization. CVPR 2016.

[7] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra:
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. ICCV 2017.

[8] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei:
Deformable Convolutional Networks. ICCV 2017.

[9] Xizhou Zhu, Han Hu, Stephen Lin, Jifeng Dai: Deformable ConvNets v2: More Deformable, Better Results. CoRR abs/1811.11168 (2018)

[10] Wei Li, Xiatian Zhu, Shaogang Gong: Harmonious Attention Network for Person Re-Identification. CVPR 2018.

[11] Cheng Wang, Qian Zhang, Chang Huang, Wenyu Liu, Xinggang Wang: Mancs: A Multi-task Attentional Network with Curriculum Sampling for Person Re-Identification. ECCV (4) 2018.

[12] Zintgraf, Luisa M., et al. “Visualizing deep neural network decisions: Prediction difference analysis.” arXiv preprint arXiv:1702.04595 (2017).

[13] Fong, Ruth C., and Andrea Vedaldi. “Interpretable explanations of black boxes by meaningful perturbation.” ICCV, 2017.

[14] Wang, Haofan, et al. “Score-CAM: Improved Visual Explanations Via Score-Weighted Class Activation Mapping.” arXiv preprint arXiv:1910.01279 (2019).

[15] Chen, Binghui, Weihong Deng, and Jiani Hu. “Mixed high-order attention network for person re-identification.” Proceedings of the IEEE International Conference on Computer Vision. 2019.

[16] Zhu, Xizhou, et al. “An empirical study of spatial attention mechanisms in deep networks.” ICCV, 2019.

[17] Wang, Qilong, et al. “ECA-net: Efficient channel attention for deep convolutional neural networks.” CVPR, 2020.

[18] Qin, Zequn, et al. “FcaNet: Frequency Channel Attention Networks.” arXiv preprint arXiv:2012.11879 (2020).

[19] Zhang, Xiaolin, et al. “Adversarial complementary learning for weakly supervised object localization.” CVPR, 2018.

[20] Jo, Sanhyun, and In-Jae Yu. “Puzzle-CAM: Improved localization via matching partial and full features.” arXiv preprint arXiv:2101.11253 (2021).

[21] Araslanov, Nikita, and Stefan Roth. “Single-stage semantic segmentation from image labels.” CVPR, 2020.

Transfer strategy

• Feature transfer: AdaIN [1], WCT [2], SANet [3].

• Color transfer: Learn color transformation (explicit function or implicit function (e.g., look-up table) conditioned on color values, location, semantic information, or other guidance.

Compare Different Backbones

[6] [7]

Losses:

• paired supervision: L2 loss
• unpaired supervision: adversarial loss
• smooth loss: variation loss, Poisson loss
• content loss: perception loss
• style loss: Gram loss, AdaIn loss

Multi-scale stylization

• parallel: [4]

• sequential: [5]

Reference

[1] Huang, Xun, and Serge Belongie. “Arbitrary style transfer in real-time with adaptive instance normalization.” ICCV, 2017.

[2] Li, Yijun, et al. “Universal style transfer via feature transforms.” NeurIPS, 2017.

[3] Park, Dae Young, and Kwang Hee Lee. “Arbitrary style transfer with style-attentional networks.” CVPR, 2019.

[4] Liu, Songhua, et al. “Adaattn: Revisit attention mechanism in arbitrary neural style transfer.” ICCV, 2021.

[5] Xia, Xide, et al. “Joint bilateral learning for real-time universal photorealistic style transfer.” ECCV, 2020.

[6] Wang, Pei, Yijun Li, and Nuno Vasconcelos. “Rethinking and improving the robustness of image style transfer.” CVPR, 2021.

[7] Wei, Hua-Peng, et al. “A Comparative Study of CNN-and Transformer-Based Visual Style Transfer.” Journal of Computer Science and Technology 37.3 (2022): 601-614.

Resources

• Awesome-Aesthetic-Evaluation-and-Cropping

Focus on certain aspects:

• image composition: CADB
• color: ICAA17K

A comprehensive survey can be found here.

Terminology:

• black-box/white-box attack: the adversarial example is generated with or without knowing the prior knowledge of the target model.

• targeted/non-targeted attack: whether predicting a specific label for the adversarial example.

• universal perturbation: fool a given model on any image with high probability.

Attack

1. Backward Update

   • add imperceptible distortion and increase the classification loss

   • universal adversarial perturbation: learn a residual perturbation that works on most clean images

2. Forward Update

   • one-pixel attack: use differential evolution algorithm

   • Adversarial Transformation Networks: learn a network to translate clean image to adversarial example.

Defense

1. Use modified training samples during training or modified test samples during testing

2. Modify network: model parameters regularization, add a layer/module

3. Adversarial example detector: classify an example as adversarial or clean based on certain statistics

New perspective

Adversarial examples are not bugs, they are features.