Newly Blog

Regulating latent variables or latent features can improve the generalizability of classifier and lower the error bound.

Regulating latent variables is essentially decrease the entropy of latent variables. There are some common tricks to decrease the entropy of latent variables, for example,

1. dropout
2. weight decay
3. add random noise to the latent variables in VAE and GAN.
4. add random perturbation to model parameters

For theoretical proof, please refer to here.

1. label smoothing: [1] interpolating ground-truth label and uniform label

2. bootstrapping: [2] interpolate noisy label and label from previous iteration

3. noisy data+clean data: [3] interpolate noisy label and distilled label

[1] Szegedy, Christian, et al. “Rethinking the inception architecture for computer vision.” CVPR, 2016.

[2] Reed, Scott, et al. “Training deep neural networks on noisy labels with bootstrapping.” arXiv preprint arXiv:1412.6596 (2014).

[3] Li, Yuncheng, et al. “Learning from noisy labels with distillation.” ICCV, 2017.

Definition: entities, attributes, and relationships

Two ways to construct knowledge graph:

1. probabilistic models (graphical model/random walk)

2. embedding based models

1) Approximate incremental SVM: pass through the dataset many times

• Pegasos: select a training batch in each iteration

  • python: https://github.com/ejlb/pegasos

       https://github.com/avaitla/Pegasos
    

  • C: https://www.cs.huji.ac.il/~shais/code/index.html
  • matlab: https://www.mathworks.com/matlabcentral/fileexchange/31401-pegasos-primal-estimated-sub-gradient-solver-for-svm?focused=5188208&tab=function

• sklearn.linear_model: SGD

  1 2 3 4 5 6

  clf= sklearn.linear_model.SGDClassifier(learning_rate = 'constant', eta0 = 0.1, shuffle = False, n_iter = 1) # get x1, y1 as a new instance clf.partial_fit(x1, y1) # get x2, y2 # update accuracy if needed clf.partial_fit(x2, y2)

2) Exact incremental or decremental SVM: only pass through the dataset once

• Incremental and Decremental Support Vector Machine Learning
  http://www.isn.ucsd.edu/svm/incremental

• SVM Incremental Learning, Adaptation and Optimization: extend the work above
  matlab: https://github.com/diehl/Incremental-SVM-Learning-in-MATLAB

• Incremental and decremental training for linear classification: extension of liblinear focusing on linear problem
  http://www.csie.ntu.edu.tw/~cjlin/papers/ws/index.html

1. simulates an infinite-depth network by fixed point iteration $h=f_{\theta}(h;x)$, in which $x$ is initial input, $\theta$ is the model parameter of one-time transformation. After infinite times of transformations, $x$ will approach the fixed point $h$. DEQ[1], MDEQ[2], iFPN[3]

Reference:

1. Bai, Shaojie, J. Zico Kolter, and Vladlen Koltun. “Deep equilibrium models.” Advances in Neural Information Processing Systems. 2019.
2. Bai, Shaojie, Vladlen Koltun, and J. Zico Kolter. “Multiscale deep equilibrium models.” arXiv preprint arXiv:2006.08656 (2020).
3. Wang, Tiancai, Xiangyu Zhang, and Jian Sun. “Implicit Feature Pyramid Network for Object Detection.” arXiv preprint arXiv:2012.13563 (2020).

Survey

• Image Matching from Handcrafted to Deep Features: A Survey

Deep learning methods

• Correlation tensor: [1] [2] [3]

Reference

1. Rocco, Ignacio, et al. “Neighbourhood consensus networks.” arXiv preprint arXiv:1810.10510 (2018).
2. Rocco, Ignacio, Relja Arandjelovic, and Josef Sivic. “Convolutional neural network architecture for geometric matching.” CVPR, 2017.
3. Chen, Jianchun, et al. “Arbicon-net: Arbitrary continuous geometric transformation networks for image registration.” NIPS, 2019.

Image proposals:

• Selective search [1]: hierarchical grouping based on different similarity metrics [code]

• Salient object detection [2]: identify the segment which is easy to compose from itself but hard from remaining parts of the image.

• EdgeBox [3]: identify the boxes that tightly enclose a set of edges are likely to contain an object.

• ACF detector [4]: compute gradient histograms on image pyramids

• Region Proposal Network (RPN) from faster-RCNN [5]

Video proposals:

• Video edgebox [1]: an extension of EdgeBox

• RC3D [2]: an extension of RPN

[1] Zhu, Wangjiang, et al. “A key volume mining deep framework for action recognition.” CVPR. 2016.

[2] Xu, Huijuan, Abir Das, and Kate Saenko. “R-c3d: Region convolutional 3d network for temporal activity detection.” ICCV, 2017.

Reference

Representative methods

Tutorial slides on AAAI2019

• Graph Convolutional Network (GCN) [6]
• Graph Attention Network [7]
• GraphSAGE(SAmple and aggreGatE) [8]
• Transformer [9]: transformer can be deemed as a type of GNN.

Avoid oversmoothing and go deeper

• Initial residual and Identity mapping [12]

• GCN and PageRank [13]

Graph similarity/matching

A survey on graph similarity [4]

Graph transformation:

pooling/unpooling [5]

Dynamic Graph:

• Pointer Graph Network [11]

Application

• GNN for zero-shot learning [1][2]: treat each category as a graph node

• GNN for multi-view learning [3]: treat each view as a graph node

• GNN for clustering [10]

Reference:

1. Wang, Xiaolong, Yufei Ye, and Abhinav Gupta. “Zero-shot recognition via semantic embeddings and knowledge graphs.” CVPR, 2018.

2. Lee, Chung-Wei, et al. “Multi-label zero-shot learning with structured knowledge graphs.” CVPR, 2018.

3. Wang, Dongang, et al. “Dividing and aggregating network for multi-view action recognition.” ECCV, 2018.

4. Ma, Guixiang, et al. “Deep Graph Similarity Learning: A Survey.” arXiv preprint arXiv:1912.11615 (2019).

5. Hongyang Gao, Shuiwang Ji: Graph U-Nets. CoRR abs/1905.05178 (2019)

6. Kipf, Thomas N., and Max Welling. “Semi-supervised classification with graph convolutional networks.” arXiv preprint arXiv:1609.02907 (2016).

7. Veličković, Petar, et al. “Graph attention networks.” arXiv preprint arXiv:1710.10903 (2017).

8. Hamilton, Will, Zhitao Ying, and Jure Leskovec. “Inductive representation learning on large graphs.” NeurIPS, 2017.

9. Vaswani, Ashish, et al. “Attention is all you need.” NeurIPS, 2017.

10. Bo, Deyu, et al. “Structural Deep Clustering Network.” Proceedings of The Web Conference 2020. 2020.

11. Veličković, Petar, et al. “Pointer Graph Networks.” arXiv preprint arXiv:2006.06380 (2020).

12. Chen, Ming, et al. “Simple and deep graph convolutional networks.” arXiv preprint arXiv:2007.02133 (2020).

13. Klicpera, Johannes, Aleksandar Bojchevski, and Stephan Günnemann. “Predict then propagate: Graph neural networks meet personalized pagerank.” arXiv preprint arXiv:1810.05997 (2018).

14. Dosovitskiy, Alexey, et al. “An image is worth 16x16 words: Transformers for image recognition at scale.” arXiv preprint arXiv:2010.11929 (2020).

15. Carion, Nicolas, et al. “End-to-End Object Detection with Transformers.” arXiv preprint arXiv:2005.12872 (2020).

16. Chen, Hanting, et al. “Pre-Trained Image Processing Transformer.” arXiv preprint arXiv:2012.00364 (2020).

17. Chefer, Hila, Shir Gur, and Lior Wolf. “Transformer Interpretability Beyond Attention Visualization.” arXiv preprint arXiv:2012.09838 (2020).

Objective Evaluation:

An empirical study on evaluation metrics of generative adversarial networks [1] with code.

1. Inception Score (IS): classification score using the InceptionNet pretrained on ImageNet

   $$IS=\exp\{E_x[KL(p_M(y|x)||p_M(y))]\}$$

   in which $p_M(y)$ is the marginal distribution of $p_M(y|x)$. Expect $p_M(y)$ to be of low entropy while $p_M(y|x)$ to be of high entropy. The higher, the better.

2. Mode score: extension of Inception score

3. Kernel MMD: MMD distance between two data distributions

4. Wasserstein distance: Wasserstein distance (Earth mover’s distance) between two data distributions.

5. Fréchet Inception Distance (FID): extract InceptionNet features and measure the data distribution distance. The lower, the better.

   $$FID=\|\mu_r-\mu_g\|+trace(\Sigma_r+\Sigma_g-2(\Sigma_r\Sigma_g)^{\frac{1}{2}})$$

6. KNN score: treat true data as positive and generated data as negative. Calculate the leave-one-out (LOO) accuracy based on 1-NN classifier.

7. Learned Perceptual Image Patch Similarity (LPIPS): $d(x,x_0)=\sum_l \frac{1}{H_l W_l}\sum_{h,w}\|w_l\circ (\hat{y}_{hw}^l-\hat{y}^l_{0hw})\|^2$ [3] [code]

Subjective Evaluation:

1. Each user sees two randomly selected results at a time and is asked to choose the one that looks more realistic. After obtaining all the pairwise results, Bradley-Terry model (B-T model) is used to calculate the global ranking score for each method. [2]

Reference

1. Xu, Qiantong, et al. “An empirical study on evaluation metrics of generative adversarial networks.” arXiv preprint arXiv:1806.07755 (2018).
2. Tsai, Yi-Hsuan, et al. “Deep image harmonization.” CVPR, 2017.
3. Zhang, Richard, et al. “The unreasonable effectiveness of deep features as a perceptual metric.” CVPR, 2018.

1. Kaggle datasets

2. Amazon datasets

3. UCI datasets

4. Google dataset search

5. Microsoft datasets

6. Awesome public datasets

7. Computer vision datasets

8. Graviti