1. Use the first few layers to simulate neuron activation in human brain [1]

2. Use the attention learnt by network to mimick human attention [2]

Reference

[1] Dapello, Joel, et al. “Simulating a primary visual cortex at the front of CNNs improves robustness to image perturbations.” BioRxiv (2020).

[2] Linsley, Drew, et al. “Learning what and where to attend.” arXiv preprint arXiv:1805.08819 (2018).

Attention in CNN:

According to [4], attention can be categorized into bottom-up attention (visual saliency, unsupervised) and top-down attention (task-driven, supervised).

According to [5], attention can be categorized into forward attention, post-hoc attention, and query-based attention.

• forward attention: spatial attention [16], channel attention [10] [17] [18], full attention [11], Deformable CNN v1 [8] v2 [9],

• post-hoc attention: CAM [6], GradCAM [7], scoreCAM [14], trainable CAM [20][21]

• query-based attention: [5]

• erase attention: [19] [12] [13]

• high-order attention [15]

Attention in RNN:

survey paper: survey on the attention based RNN model and its applications in computer vision [1]

• soft/hard attention: binary weight or soft weight

• item-wise/location-wise attention: location-wise attention is to convert an image to a sequence of local regions, which is essentially item-wise.

Earliest papers [2] [3] are basically the same except design specs of RNN unit.

Reference

[1] Wang, Feng, and David MJ Tax. “Survey on the attention based RNN model and its applications in computer vision.” arXiv preprint arXiv:1601.06823 (2016).

[2] Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. “Neural machine translation by jointly learning to align and translate.” arXiv preprint arXiv:1409.0473 (2014).

[3] Vinyals, Oriol, et al. “Grammar as a foreign language.” NIPS, 2015.

[4] Drew Linsley, Dan Shiebler, Sven Eberhardt, Thomas Serre: Learning what and where to attend. ICLR, 2019.

[5] Saumya Jetley, Nicholas A. Lord, Namhoon Lee, Philip H. S. Torr: Learn to Pay Attention. ICLR, 2018.

[6] Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, Antonio Torralba: Learning Deep Features for Discriminative Localization. CVPR 2016.

[7] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra:
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. ICCV 2017.

[8] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei:
Deformable Convolutional Networks. ICCV 2017.

[9] Xizhou Zhu, Han Hu, Stephen Lin, Jifeng Dai: Deformable ConvNets v2: More Deformable, Better Results. CoRR abs/1811.11168 (2018)

[10] Wei Li, Xiatian Zhu, Shaogang Gong: Harmonious Attention Network for Person Re-Identification. CVPR 2018.

[11] Cheng Wang, Qian Zhang, Chang Huang, Wenyu Liu, Xinggang Wang: Mancs: A Multi-task Attentional Network with Curriculum Sampling for Person Re-Identification. ECCV (4) 2018.

[12] Zintgraf, Luisa M., et al. “Visualizing deep neural network decisions: Prediction difference analysis.” arXiv preprint arXiv:1702.04595 (2017).

[13] Fong, Ruth C., and Andrea Vedaldi. “Interpretable explanations of black boxes by meaningful perturbation.” ICCV, 2017.

[14] Wang, Haofan, et al. “Score-CAM: Improved Visual Explanations Via Score-Weighted Class Activation Mapping.” arXiv preprint arXiv:1910.01279 (2019).

[15] Chen, Binghui, Weihong Deng, and Jiani Hu. “Mixed high-order attention network for person re-identification.” Proceedings of the IEEE International Conference on Computer Vision. 2019.

[16] Zhu, Xizhou, et al. “An empirical study of spatial attention mechanisms in deep networks.” ICCV, 2019.

[17] Wang, Qilong, et al. “ECA-net: Efficient channel attention for deep convolutional neural networks.” CVPR, 2020.

[18] Qin, Zequn, et al. “FcaNet: Frequency Channel Attention Networks.” arXiv preprint arXiv:2012.11879 (2020).

[19] Zhang, Xiaolin, et al. “Adversarial complementary learning for weakly supervised object localization.” CVPR, 2018.

[20] Jo, Sanhyun, and In-Jae Yu. “Puzzle-CAM: Improved localization via matching partial and full features.” arXiv preprint arXiv:2101.11253 (2021).

[21] Araslanov, Nikita, and Stefan Roth. “Single-stage semantic segmentation from image labels.” CVPR, 2020.

Resources

• Awesome-Aesthetic-Evaluation-and-Cropping

Focus on certain aspects:

• image composition: CADB
• color: ICAA17K

A comprehensive survey can be found here.

Terminology:

• black-box/white-box attack: the adversarial example is generated with or without knowing the prior knowledge of the target model.

• targeted/non-targeted attack: whether predicting a specific label for the adversarial example.

• universal perturbation: fool a given model on any image with high probability.

Attack

1. Backward Update

   • add imperceptible distortion and increase the classification loss

   • universal adversarial perturbation: learn a residual perturbation that works on most clean images

2. Forward Update

   • one-pixel attack: use differential evolution algorithm

   • Adversarial Transformation Networks: learn a network to translate clean image to adversarial example.

Defense

1. Use modified training samples during training or modified test samples during testing

2. Modify network: model parameters regularization, add a layer/module

3. Adversarial example detector: classify an example as adversarial or clean based on certain statistics

New perspective

Adversarial examples are not bugs, they are features.

1. Layered Depth Image (LDI): [1]

2. MultiPlane Image (MPI): [2] [3]

3. Point cloud: [4]

Reference

[1] Shih, Meng-Li, et al. “3d photography using context-aware layered depth inpainting.” CVPR, 2020.

[2] Tucker, Richard, and Noah Snavely. “Single-view view synthesis with multiplane images.” CVPR, 2020.

[3] Li, Jiaxin, et al. “Mine: Towards continuous depth mpi with nerf for novel view synthesis.” ICCV, 2021.

[4] Niklaus, Simon, et al. “3d ken burns effect from a single image.” ACM Transactions on Graphics (TOG) 38.6 (2019): 1-15.

Notes

1. use nemiver to debug.

   • gcc/g++ -g hello.c -o hello.o #-g for debug
   • nemiver hello #bin file

2. The comment character # does not introduce a make comment in the text of commands.

3. Wildcards: . expands to all the files containing a period. A question mark represents any single character, and […] represents a character class.

4. .PHONY: clean

5. Automatic Variables:

   • $@ The name of the current target.
   • $% The filename element of an archive member specification.
   • $< The name of the first prerequisite.
   • $? The names of all prerequisites that are newer than the target, separated by spaces.
   • $^ The names of all the prerequisites, separated by spaces. This list has duplicate names removed since for most uses, such as compiling, copying, etc., duplicates are not wanted.
   • $+ The names of all the prerequisites separated by spaces, including duplicates. This variable was created for specific situations such as arguments to linkers where duplicate values have meaning.
   • $* The stem of the target filename. A stem is typically a filename without its suffix. Its use outside of pattern rules is discouraged.

6. run makefile with —just-print option to view the execution process

How to write Makefile

1. single C-file

   1 2	hello: hello.c gcc -g hello.c -o hello</code></pre>

2. multiple C-files

   1 2 3 4 5 6 7 8

   count_words: count_words.o lexer.o -lfl gcc count_words.o lexer.o -lfl -ocount_words count_words.o: count_words.c gcc -g -c count_words.c lexer.o: lexer.c gcc -g -c lexer.c lexer.c: lexer.l flex -t lexer.l > lexer.c

3. set VPATH and CPPFLAGS in implicit rules

   1 2 3 4 5 6 7

   VPATH = src include CPPFLAGS = -I include count_words: counter.o lexer.o -lfl count_words.o: counter.h counter.o: counter.h lexer.h lexer.o: lexer.h

   VPATH can be used in a more advanced fashion as follows,

   1 2 3

   vpath %.c src vpath %.l src vpath %.h include

4. Use library .a. pack .o files into .a, similar as .lib in Windows.

   1 2 3 4 5

   libcounter.a: libcounter.a(lexer.o) libcounter.a(counter.o) libcounter.a(lexer.o): lexer.o $(AR) $(ARFLAGS) $@ $< libcounter.a(counter.o): counter.o $(AR) $(ARFLAGS) $@ $<

• in-source make
• out-of-source make

Write Basic CMakeLists.txt

• common head

  1 2	PROJECT(projectname [CXX] [C] [Java]) # project name cmake_minimum_required(VERSION 2.8.12) # minimum cmake version

• variable assignment

  1	SET(VAR [VALUE] [CACHE TYPE DOCSTRING [FORCE]])

  variable should be used by ${VAR} except for IF condition.
  commonly used path variables are listed as follows:

  1 2 3 4

  <projectname>_BINARY_DIR = PROJECT_BINARY_DIR=CMAKE_BINARY_DIR <projectname>_SOURCE_DIR = PROJECT__SOURCE_DIR=CMAKE_SOURCE_DIR SET(EXECUTABLE_OUTPUT_PATH ${PROJECT_BINARY_DIR}/bin) SET(LIBRARY_OUTPUT_PATH ${PROJECT_BINARY_DIR}/lib)

• display message

  1	MESSAGE([SEND_ERROR | STATUS | FATAL_ERROR] "message to display")

• generate output binary/library

  1 2 3 4 5 6 7 8

  ADD_EXECUTABLE(hello ${SRC_LIST}) ADD_LIBRARY(libname [SHARED|STATIC|MODULE] [EXCLUDE_FROM_ALL] source1 source2 ... sourceN) TARGET_LINK_LIBRARIES(target library1 <debug | optimized> library2 ...) /#target-specific // change the name, version, and so on of the output (*e.g.*, library, binary) SET_TARGET_PROPERTIES(target1 target2 ... PROPERTIES prop1 value1 prop2 value2 ...)

Finding

• search source

  1 2	INCLUDE_DIRECTORIES([AFTER|BEFORE] [SYSTEM] dir1 dir2 ...) #include files LINK_DIRECTORIES(directory1 directory2 ...)

  CMAKE_INCLUDE_PATH, CMAKE_LIBRARY_PATH, CMAKE_MODULE_PATH are environment variables instead of CMake variables. when using FIND_***, CMAKE_INCLUDE_PATH, the above paths will be searched.

  1 2 3 4

  FIND_PATH(myHeader hello.h) IF(myHeader) INCLUDE_DIRECTORIES(${myHeader}) ENDIF(myHeader)

• search commands

  1 2 3 4 5

  FIND_FILE(<VAR> name1 path1 path2 ...) FIND_LIBRARY(<VAR> name1 path1 path2 ...) FIND_PATH(<VAR> name1 path1 path2 ...) FIND_PROGRAM(<VAR> name1 path1 path2 ...) FIND_PACKAGE(<name> [major.minor] [QUIET] [NO_MODULE] [[REQUIRED|COMPONENTS] [componets...]]) # find \*.cmake

Compactness

• include the content of other files

  1	include(FILE) # load the content of FILE

• hierarchical binary tree

  1	ADD_SUBDIRECTORY(source_dir [binary_dir][EXCLUDE_FROM_ALL])

• macro definition

  1 2 3 4

  MACRO(add_example name) ADD_EXECUTABLE(${name} ${name}.cpp) TARGET_LINK_LIBRARIES(${name} dlib::dlib ) ENDMACRO()

CMake Module

• define FindHELLO.cmake module

  1 2 3 4 5 6 7 8

  FIND_PATH(HELLO_INCLUDE_DIR hello.h /usr/include/hello /usr/local/include/hello) FIND_LIBRARY(HELLO_LIBRARY NAMES hello PATH /usr/lib/usr/local/lib) IF (HELLO_INCLUDE_DIR AND HELLO_LIBRARY) SET(HELLO_FOUND TRUE) ENDIF (HELLO_INCLUDE_DIR AND HELLO_LIBRARY) IF (HELLO_FOUND) IF (NOT HELLO_FIND_QUIETLY) MESSAGE(STATUS "Found Hello: ${HELLO_LIBRARY}")

Installation

cmake -DCMAKE_INSTALL_PREFIX=/usr # the default install target is /usr/local

• install library and binary

  1 2 3 4 5 6 7 8 9

  INSTALL(TARGETS targets... [[ARCHIVE|LIBRARY|RUNTIME] [DESTINATION <dir>] [PERMISSIONS permissions...] [CONFIGURATIONS [Debug|Release|...]] [COMPONENT <component>] [OPTIONAL] ] [...])

  ARCHIVE is static library *.a; LIBRARY is dynamic library *.so; RUNTIME is executable binary

• install regular file

  1 2 3 4 5

  INSTALL(FILES files... DESTINATION <dir> [PERMISSIONS permissions...] [CONFIGURATIONS [Debug|Release|...]] [COMPONENT <component>] [RENAME <name>] [OPTIONAL])

• install script file (e.g., *.sh), almost the same with installing files except for permission

  1 2 3 4 5

  INSTALL(PROGRAMS files... DESTINATION <dir> [PERMISSIONS permissions...] [CONFIGURATIONS [Debug|Release|...]] [COMPONENT <component>] [RENAME <name>] [OPTIONAL])

• install folders

  1 2 3 4 5 6 7 8

  INSTALL(DIRECTORY dirs... DESTINATION <dir> [FILE_PERMISSIONS permissions...] [DIRECTORY_PERMISSIONS permissions...] [USE_SOURCE_PERMISSIONS] [CONFIGURATIONS [Debug|Release|...]] [COMPONENT <component>] [[PATTERN <pattern> | REGEX <regex>] [EXCLUDE] [PERMISSIONS permissions...]] [...])

• install *.cmake

  1	INSTALL([[SCRIPT <file>] [CODE <code>]] [...])

Testing

1
2

ADD_TEST(mytest ${PROJECT_BINARY_DIR}/bin/main)
ENABLE_TESTING()

After generating Makefile, run make test

Cmake supports CDT4 and higher versions.

1

cmake -help  # check the supported generator

1. Install CDT to Eclipse: http://www.eclipse.org/cdt/downloads.php

2. The eclipse build directory should be sibling directory of the source directory.

   1 2 3

   mkdir eclipse cd eclipse cmake -G "Eclipse CDT4 - Unix Makefiles" -D CMAKE_BUILD_TYPE=Debug ../src_folder # note that CMAKE_BUILD_TYPE can be set as Debug or Release

cmake --build . --config Release is equivalent to make

cmake --build . --target install --config Release is equivalent to make install

White balance: White balance is the process of removing unrealistic color casts, so that objects which appear white in person eyes are shown white in the image and this is more relevant to the settings of digital cameras (auto white balance). Since human eyes are very good at judging what is white under different light sources, if an
white object is captured in a wrong or mismatched color temperature, the realism could be significantly degraded.

datasets: https://cvil.eecs.yorku.ca/projects/public_html/sRGB_WB_correction/dataset.html

Color constancy: Color constancy is the ability to perceive color of objects, invariant to the color of the light source and it’s quite related to the human visual system. Existing computational color constancy methods
address this problem by first estimating the color of the light source and then correcting the input images pixel
to pixel to make it as taken under a white light source.

datasets: http://colorconstancy.com/evaluation/datasets/index.html