Combine different components: [1] [2]

References

1. Frühstück, Anna, et al. “Insetgan for full-body image generation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

2. Huang, Zehuan, et al. “From Parts to Whole: A Unified Reference Framework for Controllable Human Image Generation.” arXiv preprint arXiv:2404.15267 (2024).

The first paper: [1]

SwitchTransformer: [2]

Reference

When server is Linux, client is Windows, using Python.

In the python file:

add the following code

1
2

import pydevd
pydevd.settrace('202.120.38.30', port=5678)

in which the IP address is the Windows IP and the port is the default Eclipse debugger port.

Under Linux:

1. pip install pydevd
2. In the file lib/python2.7/site-packages/pydevd_file_utils.py, modify PATHS_FROM_ECLIPSE_TO_PYTHON = [(r'L:\test', r'/home/niuli/test')], in which the former is Windows path and the latter is Linux path.

Under Windows:

1. Install Eclipse IDE and PyDev plugin for Eclipse.
2. Pydev->Start Debug Server
3. Open the Debug perspective and watch Debug Server.

After the above preparation, run python code under Linux and the debugging process will jump to the debug server under Windows. For interactive debugging, open PyDev Debug Console.

arXiv: http://xxx.itp.ac.cn/......pdf

hugging face: https://hf-mirror.com

export HF_ENDPOINT=https://hf-mirror.com

from huggingface_hub import snapshot_download

snapshot_download(repo_id='Qwen/Qwen-7B',
                  repo_type='model',
                  local_dir='./model_dir',
                  resume_download=True)

modelscope: https://modelscope.cn/home

from modelscope.hub.snapshot_download import snapshot_download

model_dir = snapshot_download('qwen/Qwen-7B', 
                              cache_dir='./model_dir', 
                              revision='master')

[1] MLSD

Reference

[1] Gu, Geonmo, et al. “Towards light-weight and real-time line segment detection.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 1. 2022.

Simplify model architecture: [3]

Reduce sampling times: DDIM [1], PLMS [2]

Step distillation: [3] [4] [5] [6]

Adversarial distillation: [7]

References

[1] Song, Jiaming, Chenlin Meng, and Stefano Ermon. “Denoising diffusion implicit models.” arXiv preprint arXiv:2010.02502 (2020).

[2] Liu, Luping, et al. “Pseudo numerical methods for diffusion models on manifolds.” ICLR, (2022).

[3] Li, Yanyu, et al. “SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Seconds.” NeurIPS(2023).

[4] Salimans, Tim, and Jonathan Ho. “Progressive distillation for fast sampling of diffusion models.” ICLR, 2022.

[5] Luhman, Eric, and Troy Luhman. “Knowledge distillation in iterative generative models for improved sampling speed.” arXiv preprint arXiv:2101.02388 (2021).

[6] Meng, Chenlin, et al. “On distillation of guided diffusion models.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[7] https://stability.ai/research/adversarial-diffusion-distillation

text-to-sketch

• [1][SVG]: a) use text-to-image model to generate an image. Align image and sketch through CLIP. b) Align text and sketch through diffusion model.
• [2][SVG]: a) Align text and sketch through CLIP
• [3][SVG]: a) Align text and sketch through diffusion model

image-to-sketch

• [4][SVG]: Align image and sketch. Use MLP to predict offsets from initial points.
• [5][SVG]: Align image and sketch. Use saliency for initialization.

SVG: Scalable Vector Graphics
SDS: score distillation sampling

References

[1] Xing, Ximing, et al. “DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models.” arXiv preprint arXiv:2306.14685 (2023).

[2] Frans, Kevin, Lisa Soros, and Olaf Witkowski. “Clipdraw: Exploring text-to-drawing synthesis through language-image encoders.” Advances in Neural Information Processing Systems 35 (2022): 5207-5218.

[3] Jain, Ajay, Amber Xie, and Pieter Abbeel. “Vectorfusion: Text-to-svg by abstracting pixel-based diffusion models.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[4] Vinker, Yael, et al. “Clipascene: Scene sketching with different types and levels of abstraction.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[5] Vinker, Yael, et al. “Clipasso: Semantically-aware object sketching.” ACM Transactions on Graphics (TOG) 41.4 (2022): 1-11.

[1] The edited previous frame serves as condition for editing the next frame.

References

[1] Chai, Wenhao, et al. “StableVideo: Text-driven Consistency-aware Diffusion Video Editing.” arXiv preprint arXiv:2308.09592 (2023).

• tutorial: [1]

Difference between VM and Docker

VM: guest OS -> BINS&LIBS -> App
Docker: BINS*LIBS -> App

Installation

https://docs.docker.com/engine/install/ubuntu/

Uninstallation

• Uninstall the Docker Engine, CLI, containerd, and Docker Compose packages: sudo apt-get purge docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras

• Images, containers, volumes, or custom configuration files on your host aren’t automatically removed. To delete all images, containers, and volumes: sudo rm -rf /var/lib/docker, sudo rm -rf /var/lib/containerd

Commands

image

docker image pull $image_name:$version

docker image tag $image_name:version $registryIP:port/username/image_name:version

docker image push $registryIP:port/username/image_name:version

docker image build -t $image_name .

docker image ls

docker image rm $image_name

docker image save $image_name > $filename

docker load < $filename

container

docker container create $image_name

docker container start $container_ID

docker container run $image_name # run is equal to create and start

docker container run -it $image_name /bin/bash

docker container ls, docker container ls --a

docker container pause $container_ID

docker container unpause $container_ID

docker container stop $container_ID

docker container kill $container_ID # the difference between stop and kill is that stop may do some clean-up before killing the container

docker container rm $container_ID

docker container prune # remove all the exit containers

docker container exec -it $containe_ID /bin/bash

docker container cp $container_ID:$file_path .

docker container commit $container_ID $image_name:$version

Dockerfile

FROM python:3.7
WORKDIR ./docker_demo
ADD . .
RUN pip install -r requirements.txt
CMD ["python", "./src/main.py"]

Tutorial: [1]

Use GPU in docker

1. Install nvidia-docker https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

2. docker run --gpus all -it $image_name