GitHub - nerdyrodent/VQGAN-CLIP: Just playing with getting VQGAN+CLIP running lo...
source link: https://github.com/nerdyrodent/VQGAN-CLIP
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
VQGAN-CLIP Overview
A repo for running VQGAN+CLIP locally. This started out as a Katherine Crowson VQGAN+CLIP derived Google colab notebook.
Original notebook:
Some example images:
Environment:
- Tested on Ubuntu 20.04
- GPU: Nvidia RTX 3090
- Typical VRAM requirements:
- 24 GB for a 900x900 image
- 10 GB for a 512x512 image
- 8 GB for a 380x380 image
You may also be interested in CLIP Guided Diffusion
Set up
This example uses Anaconda to manage virtual Python environments.
Create a new virtual Python environment for VQGAN-CLIP:
conda create --name vqgan python=3.9 conda activate vqgan
Install Pytorch in the new enviroment:
Note: This installs the CUDA version of Pytorch, if you want to use an AMD graphics card, read the AMD section below.
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
Install other required Python packages:
pip install ftfy regex tqdm omegaconf pytorch-lightning IPython kornia imageio imageio-ffmpeg einops torch_optimizer
Clone required repositories:
git clone 'https://github.com/nerdyrodent/VQGAN-CLIP' cd VQGAN-CLIP git clone 'https://github.com/openai/CLIP' git clone 'https://github.com/CompVis/taming-transformers'
Note: In my development environment both CLIP and taming-transformers are present in the local directory, and so aren't present in the requirements.txt
or vqgan.yml
files.
As an alternative, you can also pip install taming-transformers and CLIP.
You will also need at least 1 VQGAN pretrained model. E.g.
mkdir checkpoints curl -L -o checkpoints/vqgan_imagenet_f16_16384.yaml -C - 'https://heibox.uni-heidelberg.de/d/a7530b09fed84f80a887/files/?p=%2Fconfigs%2Fmodel.yaml&dl=1' #ImageNet 16384 curl -L -o checkpoints/vqgan_imagenet_f16_16384.ckpt -C - 'https://heibox.uni-heidelberg.de/d/a7530b09fed84f80a887/files/?p=%2Fckpts%2Flast.ckpt&dl=1' #ImageNet 16384
The download_models.sh
script is an optional way to download a number of models. By default, it will download just 1 model.
See https://github.com/CompVis/taming-transformers#overview-of-pretrained-models for more information about VQGAN pre-trained models, including download links.
By default, the model .yaml and .ckpt files are expected in the checkpoints
directory.
See https://github.com/CompVis/taming-transformers for more information on datasets and models.
Video guides are also available:
Using an AMD graphics card
Note: This hasn't been tested yet.
ROCm can be used for AMD graphics cards instead of CUDA. You can check if your card is supported here: https://github.com/RadeonOpenCompute/ROCm#supported-gpus
Install ROCm accordng to the instructions and don't forget to add the user to the video group: https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html
The usage and set up instructions above are the same, except for the line where you install Pytorch.
Instead of pip install torch==1.9.0+cu111 ...
, use the one or two lines which are displayed here (select Pip -> Python-> ROCm):
https://pytorch.org/get-started/locally/
Using the CPU
If no graphics card can be found, the CPU is automatically used and a warning displayed.
Regardless of an available graphics card, the CPU can also be used by adding this command line argument: -cd cpu
This works with the CUDA version of Pytorch, even without CUDA drivers installed, but doesn't seem to work with ROCm as of now.
Uninstalling
Remove the Python enviroment:
conda remove --name vqgan --all
and delete the VQGAN-CLIP
directory.
To generate images from text, specify your text prompt as shown in the example below:
python generate.py -p "A painting of an apple in a fruit bowl"
Multiple prompts
Text and image prompts can be split using the pipe symbol in order to allow multiple prompts. You can also use a colon followed by a number to set a weight for that prompt. For example:
python generate.py -p "A painting of an apple in a fruit bowl | psychedelic | surreal:0.5 | weird:0.25"
Image prompts can be split in the same way. For example:
python generate.py -p "A picture of a bedroom with a portrait of Van Gogh" -ip "samples/VanGogh.jpg | samples/Bedroom.png"
Story mode
Sets of text prompts can be created using the caret symbol, in order to generate a sort of story mode. For example:
python generate.py -p "A painting of a sunflower|photo:-1 ^ a painting of a rose ^ a painting of a tulip ^ a painting of a daisy flower ^ a photograph of daffodil" -cpe 1500 -zvid -i 6000 -zse 10 -vl 20 -zsc 1.005 -opt Adagrad -lr 0.15 -se 6000
"Style Transfer"
An input image with style text and a low number of iterations can be used create a sort of "style transfer" effect. For example:
python generate.py -p "A painting in the style of Picasso" -ii samples/VanGogh.jpg -i 80 -se 10 -opt AdamW -lr 0.25
Output Style
Picasso
Sketch
Psychedelic
A video style transfer effect can be achived by specifying a directory of video frames in video_style_dir
. Output will be saved in the steps directory, using the original video frame filenames. You can also use this as a sort of "batch mode" if you have a directory of images you want to apply a style to. This can also be combined with Story Mode if you don't wish to apply the same style to every images, but instead roll through a list of styles.
Feedback example
By feeding back the generated images and making slight changes, some interesting effects can be created.
The example zoom.sh
shows this by applying a zoom and rotate to generated images, before feeding them back in again.
To use zoom.sh
, specifying a text prompt, output filename and number of frames. E.g.
./zoom.sh "A painting of a red telephone box spinning through a time vortex" Telephone.png 150
There is also a simple zoom video creation option available. For example:
python generate.py -p "The inside of a sphere" -zvid -i 4500 -zse 20 -vl 10 -zsc 0.97 -opt Adagrad -lr 0.15 -se 4500
Random text example
Use random.sh
to make a batch of images from random text. Edit the text and number of generated images to your taste!
./random.sh
Advanced options
To view the available options, use "-h".
python generate.py -h
usage: generate.py [-h] [-p PROMPTS] [-ip IMAGE_PROMPTS] [-i MAX_ITERATIONS] [-se DISPLAY_FREQ] [-s SIZE SIZE] [-ii INIT_IMAGE] [-in INIT_NOISE] [-iw INIT_WEIGHT] [-m CLIP_MODEL] [-conf VQGAN_CONFIG] [-ckpt VQGAN_CHECKPOINT] [-nps [NOISE_PROMPT_SEEDS ...]] [-npw [NOISE_PROMPT_WEIGHTS ...]] [-lr STEP_SIZE] [-cuts CUTN] [-cutp CUT_POW] [-sd SEED] [-opt {Adam,AdamW,Adagrad,Adamax,DiffGrad,AdamP,RAdam,RMSprop}] [-o OUTPUT] [-vid] [-zvid] [-zs ZOOM_START] [-zse ZOOM_FREQUENCY] [-zsc ZOOM_SCALE] [-cpe PROMPT_FREQUENCY] [-vl VIDEO_LENGTH] [-ofps OUTPUT_VIDEO_FPS] [-ifps INPUT_VIDEO_FPS] [-d] [-aug {Ji,Sh,Gn,Pe,Ro,Af,Et,Ts,Cr,Er,Re} [{Ji,Sh,Gn,Pe,Ro,Af,Et,Ts,Cr,Er,Re} ...]] [-cd CUDA_DEVICE]
optional arguments: -h, --help show this help message and exit -p PROMPTS, --prompts PROMPTS Text prompts -ip IMAGE_PROMPTS, --image_prompts IMAGE_PROMPTS Image prompts / target image -i MAX_ITERATIONS, --iterations MAX_ITERATIONS Number of iterations -se DISPLAY_FREQ, --save_every DISPLAY_FREQ Save image iterations -s SIZE SIZE, --size SIZE SIZE Image size (width height) (default: [512, 512]) -ii INIT_IMAGE, --init_image INIT_IMAGE Initial image -in INIT_NOISE, --init_noise INIT_NOISE Initial noise image (pixels or gradient) -iw INIT_WEIGHT, --init_weight INIT_WEIGHT Initial weight -m CLIP_MODEL, --clip_model CLIP_MODEL CLIP model (e.g. ViT-B/32, ViT-B/16) -conf VQGAN_CONFIG, --vqgan_config VQGAN_CONFIG VQGAN config -ckpt VQGAN_CHECKPOINT, --vqgan_checkpoint VQGAN_CHECKPOINT VQGAN checkpoint -nps [NOISE_PROMPT_SEEDS ...], --noise_prompt_seeds [NOISE_PROMPT_SEEDS ...] Noise prompt seeds -npw [NOISE_PROMPT_WEIGHTS ...], --noise_prompt_weights [NOISE_PROMPT_WEIGHTS ...] Noise prompt weights -lr STEP_SIZE, --learning_rate STEP_SIZE Learning rate -cuts CUTN, --num_cuts CUTN Number of cuts -cutp CUT_POW, --cut_power CUT_POW Cut power -sd SEED, --seed SEED Seed -opt, --optimiser {Adam,AdamW,Adagrad,Adamax,DiffGrad,AdamP,RAdam,RMSprop} Optimiser -o OUTPUT, --output OUTPUT Output file -vid, --video Create video frames? -zvid, --zoom_video Create zoom video? -zs ZOOM_START, --zoom_start ZOOM_START Zoom start iteration -zse ZOOM_FREQUENCY, --zoom_save_every ZOOM_FREQUENCY Save zoom image iterations -zsc ZOOM_SCALE, --zoom_scale ZOOM_SCALE Zoom scale -cpe PROMPT_FREQUENCY, --change_prompt_every PROMPT_FREQUENCY Prompt change frequency -vl VIDEO_LENGTH, --video_length VIDEO_LENGTH Video length in seconds -ofps OUTPUT_VIDEO_FPS, --output_video_fps OUTPUT_VIDEO_FPS Create an interpolated video (Nvidia GPU only) with this fps (min 10. best set to 30 or 60) -ifps INPUT_VIDEO_FPS, --input_video_fps INPUT_VIDEO_FPS When creating an interpolated video, use this as the input fps to interpolate from (>0 & <ofps) -d, --deterministic Enable cudnn.deterministic? -aug, --augments {Ji,Sh,Gn,Pe,Ro,Af,Et,Ts,Cr,Er,Re} [{Ji,Sh,Gn,Pe,Ro,Af,Et,Ts,Cr,Er,Re} ...] Enabled augments -cd CUDA_DEVICE, --cuda_device CUDA_DEVICE Cuda device to use
Troubleshooting
CUSOLVER_STATUS_INTERNAL_ERROR
For example:
RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling cusolverDnCreate(handle)
Make sure you have specified the correct size for the image. For more information please refer to #6
RuntimeError: CUDA out of memory
For example:
RuntimeError: CUDA out of memory. Tried to allocate 150.00 MiB (GPU 0; 23.70 GiB total capacity; 21.31 GiB already allocated; 78.56 MiB free; 21.70 GiB reserved in total by PyTorch)
Your request doesn't fit into your GPU's VRAM. Reduce the image size and/or number of cuts.
Citations
@misc{unpublished2021clip, title = {CLIP: Connecting Text and Images}, author = {Alec Radford, Ilya Sutskever, Jong Wook Kim, Gretchen Krueger, Sandhini Agarwal}, year = {2021} }
@misc{esser2020taming, title={Taming Transformers for High-Resolution Image Synthesis}, author={Patrick Esser and Robin Rombach and Björn Ommer}, year={2020}, eprint={2012.09841}, archivePrefix={arXiv}, primaryClass={cs.CV} }
Katherine Crowson - https://github.com/crowsonkb
Public Domain images from Open Access Images at the Art Institute of Chicago - https://www.artic.edu/open-access/open-access-images
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK