Newly Blog

• Typical works: CapsNet [1], CapProNet [2] [code]

• The robustness of Capsule network: [3]

Reference

[1] Sabour S, Frosst N, Hinton G E. Dynamic routing between capsules[C]//Advances in Neural Information Processing Systems. 2017: 3856-3866.

[2] Zhang L, Edraki M, Qi G J. CapProNet: Deep feature learning via orthogonal projections onto capsule subspaces[J]. arXiv preprint arXiv:1805.07621, 2018.

[3] Jindong Gu, Volker Tresp, Han Hu, “Capsule Network is Not More Robust than Convolutional Network”, CVPR 2021.

1. Use the first few layers to simulate neuron activation in human brain [1]

2. Use the attention learnt by network to mimick human attention [2]

Reference

[1] Dapello, Joel, et al. “Simulating a primary visual cortex at the front of CNNs improves robustness to image perturbations.” BioRxiv (2020).

[2] Linsley, Drew, et al. “Learning what and where to attend.” arXiv preprint arXiv:1805.08819 (2018).

Attention in CNN:

According to [4], attention can be categorized into bottom-up attention (visual saliency, unsupervised) and top-down attention (task-driven, supervised).

According to [5], attention can be categorized into forward attention, post-hoc attention, and query-based attention.

• forward attention: spatial attention [16], channel attention [10] [17] [18], full attention [11], Deformable CNN v1 [8] v2 [9],

• post-hoc attention: CAM [6], GradCAM [7], scoreCAM [14], trainable CAM [20][21]

• query-based attention: [5]

• erase attention: [19] [12] [13]

• high-order attention [15]

Attention in RNN:

survey paper: survey on the attention based RNN model and its applications in computer vision [1]

• soft/hard attention: binary weight or soft weight

• item-wise/location-wise attention: location-wise attention is to convert an image to a sequence of local regions, which is essentially item-wise.

Earliest papers [2] [3] are basically the same except design specs of RNN unit.

Reference

[1] Wang, Feng, and David MJ Tax. “Survey on the attention based RNN model and its applications in computer vision.” arXiv preprint arXiv:1601.06823 (2016).

[2] Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. “Neural machine translation by jointly learning to align and translate.” arXiv preprint arXiv:1409.0473 (2014).

[3] Vinyals, Oriol, et al. “Grammar as a foreign language.” NIPS, 2015.

[4] Drew Linsley, Dan Shiebler, Sven Eberhardt, Thomas Serre: Learning what and where to attend. ICLR, 2019.

[5] Saumya Jetley, Nicholas A. Lord, Namhoon Lee, Philip H. S. Torr: Learn to Pay Attention. ICLR, 2018.

[6] Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, Antonio Torralba: Learning Deep Features for Discriminative Localization. CVPR 2016.

[7] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra:
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. ICCV 2017.

[8] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei:
Deformable Convolutional Networks. ICCV 2017.

[9] Xizhou Zhu, Han Hu, Stephen Lin, Jifeng Dai: Deformable ConvNets v2: More Deformable, Better Results. CoRR abs/1811.11168 (2018)

[10] Wei Li, Xiatian Zhu, Shaogang Gong: Harmonious Attention Network for Person Re-Identification. CVPR 2018.

[11] Cheng Wang, Qian Zhang, Chang Huang, Wenyu Liu, Xinggang Wang: Mancs: A Multi-task Attentional Network with Curriculum Sampling for Person Re-Identification. ECCV (4) 2018.

[12] Zintgraf, Luisa M., et al. “Visualizing deep neural network decisions: Prediction difference analysis.” arXiv preprint arXiv:1702.04595 (2017).

[13] Fong, Ruth C., and Andrea Vedaldi. “Interpretable explanations of black boxes by meaningful perturbation.” ICCV, 2017.

[14] Wang, Haofan, et al. “Score-CAM: Improved Visual Explanations Via Score-Weighted Class Activation Mapping.” arXiv preprint arXiv:1910.01279 (2019).

[15] Chen, Binghui, Weihong Deng, and Jiani Hu. “Mixed high-order attention network for person re-identification.” Proceedings of the IEEE International Conference on Computer Vision. 2019.

[16] Zhu, Xizhou, et al. “An empirical study of spatial attention mechanisms in deep networks.” ICCV, 2019.

[17] Wang, Qilong, et al. “ECA-net: Efficient channel attention for deep convolutional neural networks.” CVPR, 2020.

[18] Qin, Zequn, et al. “FcaNet: Frequency Channel Attention Networks.” arXiv preprint arXiv:2012.11879 (2020).

[19] Zhang, Xiaolin, et al. “Adversarial complementary learning for weakly supervised object localization.” CVPR, 2018.

[20] Jo, Sanhyun, and In-Jae Yu. “Puzzle-CAM: Improved localization via matching partial and full features.” arXiv preprint arXiv:2101.11253 (2021).

[21] Araslanov, Nikita, and Stefan Roth. “Single-stage semantic segmentation from image labels.” CVPR, 2020.

Transfer strategy

• Feature transfer: AdaIN [1], WCT [2], SANet [3].

• Color transfer: Learn color transformation (explicit function or implicit function (e.g., look-up table) conditioned on color values, location, semantic information, or other guidance.

Compare Different Backbones

[6] [7]

Losses:

• paired supervision: L2 loss
• unpaired supervision: adversarial loss
• smooth loss: variation loss, Poisson loss
• content loss: perception loss
• style loss: Gram loss, AdaIn loss

Multi-scale stylization

• parallel: [4]

• sequential: [5]

Reference

[1] Huang, Xun, and Serge Belongie. “Arbitrary style transfer in real-time with adaptive instance normalization.” ICCV, 2017.

[2] Li, Yijun, et al. “Universal style transfer via feature transforms.” NeurIPS, 2017.

[3] Park, Dae Young, and Kwang Hee Lee. “Arbitrary style transfer with style-attentional networks.” CVPR, 2019.

[4] Liu, Songhua, et al. “Adaattn: Revisit attention mechanism in arbitrary neural style transfer.” ICCV, 2021.

[5] Xia, Xide, et al. “Joint bilateral learning for real-time universal photorealistic style transfer.” ECCV, 2020.

[6] Wang, Pei, Yijun Li, and Nuno Vasconcelos. “Rethinking and improving the robustness of image style transfer.” CVPR, 2021.

[7] Wei, Hua-Peng, et al. “A Comparative Study of CNN-and Transformer-Based Visual Style Transfer.” Journal of Computer Science and Technology 37.3 (2022): 601-614.

Resources

• Awesome-Aesthetic-Evaluation-and-Cropping

Focus on certain aspects:

• image composition: CADB
• color: ICAA17K

A comprehensive survey can be found here.

Terminology:

• black-box/white-box attack: the adversarial example is generated with or without knowing the prior knowledge of the target model.

• targeted/non-targeted attack: whether predicting a specific label for the adversarial example.

• universal perturbation: fool a given model on any image with high probability.

Attack

1. Backward Update

   • add imperceptible distortion and increase the classification loss

   • universal adversarial perturbation: learn a residual perturbation that works on most clean images

2. Forward Update

   • one-pixel attack: use differential evolution algorithm

   • Adversarial Transformation Networks: learn a network to translate clean image to adversarial example.

Defense

1. Use modified training samples during training or modified test samples during testing

2. Modify network: model parameters regularization, add a layer/module

3. Adversarial example detector: classify an example as adversarial or clean based on certain statistics

New perspective

Adversarial examples are not bugs, they are features.

1. Layered Depth Image (LDI): [1]

2. MultiPlane Image (MPI): [2] [3]

3. Point cloud: [4]

Reference

[1] Shih, Meng-Li, et al. “3d photography using context-aware layered depth inpainting.” CVPR, 2020.

[2] Tucker, Richard, and Noah Snavely. “Single-view view synthesis with multiplane images.” CVPR, 2020.

[3] Li, Jiaxin, et al. “Mine: Towards continuous depth mpi with nerf for novel view synthesis.” ICCV, 2021.

[4] Niklaus, Simon, et al. “3d ken burns effect from a single image.” ACM Transactions on Graphics (TOG) 38.6 (2019): 1-15.

Notes

1. use nemiver to debug.

   • gcc/g++ -g hello.c -o hello.o #-g for debug
   • nemiver hello #bin file

2. The comment character # does not introduce a make comment in the text of commands.

3. Wildcards: . expands to all the files containing a period. A question mark represents any single character, and […] represents a character class.

4. .PHONY: clean

5. Automatic Variables:

   • $@ The name of the current target.
   • $% The filename element of an archive member specification.
   • $< The name of the first prerequisite.
   • $? The names of all prerequisites that are newer than the target, separated by spaces.
   • $^ The names of all the prerequisites, separated by spaces. This list has duplicate names removed since for most uses, such as compiling, copying, etc., duplicates are not wanted.
   • $+ The names of all the prerequisites separated by spaces, including duplicates. This variable was created for specific situations such as arguments to linkers where duplicate values have meaning.
   • $* The stem of the target filename. A stem is typically a filename without its suffix. Its use outside of pattern rules is discouraged.

6. run makefile with —just-print option to view the execution process

How to write Makefile

1. single C-file

   1 2	hello: hello.c gcc -g hello.c -o hello</code></pre>

2. multiple C-files

   1 2 3 4 5 6 7 8

   count_words: count_words.o lexer.o -lfl gcc count_words.o lexer.o -lfl -ocount_words count_words.o: count_words.c gcc -g -c count_words.c lexer.o: lexer.c gcc -g -c lexer.c lexer.c: lexer.l flex -t lexer.l > lexer.c

3. set VPATH and CPPFLAGS in implicit rules

   1 2 3 4 5 6 7

   VPATH = src include CPPFLAGS = -I include count_words: counter.o lexer.o -lfl count_words.o: counter.h counter.o: counter.h lexer.h lexer.o: lexer.h

   VPATH can be used in a more advanced fashion as follows,

   1 2 3

   vpath %.c src vpath %.l src vpath %.h include

4. Use library .a. pack .o files into .a, similar as .lib in Windows.

   1 2 3 4 5

   libcounter.a: libcounter.a(lexer.o) libcounter.a(counter.o) libcounter.a(lexer.o): lexer.o $(AR) $(ARFLAGS) $@ $< libcounter.a(counter.o): counter.o $(AR) $(ARFLAGS) $@ $<

• in-source make
• out-of-source make

Write Basic CMakeLists.txt

• common head

  1 2	PROJECT(projectname [CXX] [C] [Java]) # project name cmake_minimum_required(VERSION 2.8.12) # minimum cmake version

• variable assignment

  1	SET(VAR [VALUE] [CACHE TYPE DOCSTRING [FORCE]])

  variable should be used by ${VAR} except for IF condition.
  commonly used path variables are listed as follows:

  1 2 3 4

  <projectname>_BINARY_DIR = PROJECT_BINARY_DIR=CMAKE_BINARY_DIR <projectname>_SOURCE_DIR = PROJECT__SOURCE_DIR=CMAKE_SOURCE_DIR SET(EXECUTABLE_OUTPUT_PATH ${PROJECT_BINARY_DIR}/bin) SET(LIBRARY_OUTPUT_PATH ${PROJECT_BINARY_DIR}/lib)

• display message

  1	MESSAGE([SEND_ERROR | STATUS | FATAL_ERROR] "message to display")

• generate output binary/library

  1 2 3 4 5 6 7 8

  ADD_EXECUTABLE(hello ${SRC_LIST}) ADD_LIBRARY(libname [SHARED|STATIC|MODULE] [EXCLUDE_FROM_ALL] source1 source2 ... sourceN) TARGET_LINK_LIBRARIES(target library1 <debug | optimized> library2 ...) /#target-specific // change the name, version, and so on of the output (*e.g.*, library, binary) SET_TARGET_PROPERTIES(target1 target2 ... PROPERTIES prop1 value1 prop2 value2 ...)

  Finding

• search source

  1 2	INCLUDE_DIRECTORIES([AFTER|BEFORE] [SYSTEM] dir1 dir2 ...) #include files LINK_DIRECTORIES(directory1 directory2 ...)

  CMAKE_INCLUDE_PATH, CMAKE_LIBRARY_PATH, CMAKE_MODULE_PATH are environment variables instead of CMake variables. when using FIND_***, CMAKE_INCLUDE_PATH, the above paths will be searched.

  1 2 3 4

  FIND_PATH(myHeader hello.h) IF(myHeader) INCLUDE_DIRECTORIES(${myHeader}) ENDIF(myHeader)

• search commands

  1 2 3 4 5

  FIND_FILE(<VAR> name1 path1 path2 ...) FIND_LIBRARY(<VAR> name1 path1 path2 ...) FIND_PATH(<VAR> name1 path1 path2 ...) FIND_PROGRAM(<VAR> name1 path1 path2 ...) FIND_PACKAGE(<name> [major.minor] [QUIET] [NO_MODULE] [[REQUIRED|COMPONENTS] [componets...]]) # find \*.cmake

  Compactness

• include the content of other files

  1	include(FILE) # load the content of FILE

• hierarchical binary tree

  1	ADD_SUBDIRECTORY(source_dir [binary_dir][EXCLUDE_FROM_ALL])

• macro definition

  1 2 3 4

  MACRO(add_example name) ADD_EXECUTABLE(${name} ${name}.cpp) TARGET_LINK_LIBRARIES(${name} dlib::dlib ) ENDMACRO()

  CMake Module

• define FindHELLO.cmake module

  1 2 3 4 5 6 7 8

  FIND_PATH(HELLO_INCLUDE_DIR hello.h /usr/include/hello /usr/local/include/hello) FIND_LIBRARY(HELLO_LIBRARY NAMES hello PATH /usr/lib/usr/local/lib) IF (HELLO_INCLUDE_DIR AND HELLO_LIBRARY) SET(HELLO_FOUND TRUE) ENDIF (HELLO_INCLUDE_DIR AND HELLO_LIBRARY) IF (HELLO_FOUND) IF (NOT HELLO_FIND_QUIETLY) MESSAGE(STATUS "Found Hello: ${HELLO_LIBRARY}")

  Installation

  cmake -DCMAKE_INSTALL_PREFIX=/usr # the default install target is /usr/local

• install library and binary

  1 2 3 4 5 6 7 8 9

  INSTALL(TARGETS targets... [[ARCHIVE|LIBRARY|RUNTIME] [DESTINATION <dir>] [PERMISSIONS permissions...] [CONFIGURATIONS [Debug|Release|...]] [COMPONENT <component>] [OPTIONAL] ] [...])

  ARCHIVE is static library *.a; LIBRARY is dynamic library *.so; RUNTIME is executable binary

• install regular file

  1 2 3 4 5

  INSTALL(FILES files... DESTINATION <dir> [PERMISSIONS permissions...] [CONFIGURATIONS [Debug|Release|...]] [COMPONENT <component>] [RENAME <name>] [OPTIONAL])

• install script file (e.g., *.sh), almost the same with installing files except for permission

  1 2 3 4 5

  INSTALL(PROGRAMS files... DESTINATION <dir> [PERMISSIONS permissions...] [CONFIGURATIONS [Debug|Release|...]] [COMPONENT <component>] [RENAME <name>] [OPTIONAL])

• install folders

  1 2 3 4 5 6 7 8

  INSTALL(DIRECTORY dirs... DESTINATION <dir> [FILE_PERMISSIONS permissions...] [DIRECTORY_PERMISSIONS permissions...] [USE_SOURCE_PERMISSIONS] [CONFIGURATIONS [Debug|Release|...]] [COMPONENT <component>] [[PATTERN <pattern> | REGEX <regex>] [EXCLUDE] [PERMISSIONS permissions...]] [...])

• install *.cmake

  1	INSTALL([[SCRIPT <file>] [CODE <code>]] [...])

Testing

1
2

ADD_TEST(mytest ${PROJECT_BINARY_DIR}/bin/main)
ENABLE_TESTING()

After generating Makefile, run make test

Cmake supports CDT4 and higher versions.

1

cmake -help  # check the supported generator

1. Install CDT to Eclipse: http://www.eclipse.org/cdt/downloads.php

2. The eclipse build directory should be sibling directory of the source directory.

   1 2 3

   mkdir eclipse cd eclipse cmake -G "Eclipse CDT4 - Unix Makefiles" -D CMAKE_BUILD_TYPE=Debug ../src_folder # note that CMAKE_BUILD_TYPE can be set as Debug or Release

cmake --build . --config Release is equivalent to make

cmake --build . --target install --config Release is equivalent to make install