I’ve been in a VQGAN+CLIP craze lately, so here’s a list of all VQGAN+CLIP implementations I found on the internet (The symbol 🔰 means perfect for non-programmers alike. If you don’t know where to start, you can start with these):

VQGAN+CLIP implementations

Name Author Description / Features VQGAN+CLIP (codebook sampling method) @RiversHaveWings The original VQGAN+CLIP notebook of Katherine Crowson (@RiversHaveWings). AI Art Machine @hillelogram 🔰 Very accessible Colab notebook. Has advanced options that are explained in a beginner-friendly level. Create realistic AI-Generated Images with VQGAN+CLIP @minimaxir 🔰 Has good UI affordances and more descriptive explanation of parameters. Have options for deterministic output by using icon-based input/target images. VQGAN+CLIP (with pooling and quantize method) @ak92501 Has an optional Gradio demo for a more streamlined experience. Zoetrope 5 @classpectanon Has advanced parameters for more controlled AI art generation. I haven’t tried this yet, but it may be good to flesh your artwork more. VQGAN+CLIP Python command-line interface @nerdyrodent Not a Google Colab notebook, but a Github repo that you can fork and run locally. Provides a command-line interface to generate AI-art on the fly. VQGAN+CLIP (z+quantize method with augmentations) @somewheresy It seems to be the first English-translated notebook of Katherine Crowson (@RiversHaveWings). CLIPIT PixelDraw @dribnet A very interesting fork of the VQGAN+CLIP notebooks that uses PixelDraw to generate pixel art given a prompt. Nightcafe Studio NightCafe Studio Not a Colab notebook, but rather a managed service where you need to setup an account. I can’t comment how different the outputs are compared to the Colab notebooks. Kapwing AI Video Generator Kapwing A web-hosted version of CLIP VQGAN. Generates videos after processing. It’s not as customizable, but the processing time is relatively fast!

CLIP-guided art generators

These aren’t necessarily VQGAN implementations, but can produce AI art nonetheless:

Name Author Description / Features The Big Sleep: BigGAN x CLIP @advadnoun Uses a CLIP-guided BigGAN generator. I can’t comment on the quality of the outputs, but this is exciting to try as well! Aleph-Image @advadnoun Uses a CLIP-guided DALL-E decoder. Try it out for more interesting results! CLIP Guided Diffusion HQ 512x512 @RiversHaveWings Uses OpenAI’s 512x512 class-conditional ImageNet diffusion model with CLIP. It is fixed at 512x512, but it also has a 256x256 version.

The common denominator across these works is that they are guided by OpenAI’s CLIP so that the image matches the text description. For more CLIP-guided projects, check out this Reddit post from February.

Resources

If you wish to learn more about VQGAN and CLIP, I suggest reading the following:

• Alien Dreams: An Emerging Art Scene by Charlie Snell: gives a good overview and history of the recent AI Art scene. Traces its roots from the introduction of CLIP and its pairing of VQGAN today.
• The Illustrated VQGAN: by yours truly, here I tried to explain how VQGAN works in a conceptual level. It starts with how images are “perceived” then ends with the whole VQGAN system.

Of course, nothing beats reading the original papers themselves:

1. Esser, P., Rombach, R. and Ommer, B., 2021. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12873-12883).
2. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J. and Krueger, G., 2021. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020.

Did I miss anything? Just comment below!

Changelog

• 08-22-2021: Added Kapwing and PixelDraw
• 08-21-2021: This blogpost was featured in Comet’s newsletter!