Soft Loss

Given the predicted softmax logits $p_i$, ground-truth softmax logits or free-form weights $w_i$.

1. weighted softmax loss: $-\sum_{i} w_i \log p_i$

2. EMD softmax loss: $-\sum_{i} w_i p_i$

3. softmax loss after label flip layer: $-\log{\sum_{i} w_i p_i}$

4. knowledge distillation: $\sum_{i} (p_i-w_i)$