BlobGAN: A BIG step for GANs

May 14th 2022 new story

0

BlobGAN allows for unreal manipulation of images, made super easily controlling simple blobs. All these small blobs represent an object, and you can move them around or make them bigger, smaller, or even remove them, and it will have the same effect on the object it represents in the image. You can even create novel images by duplicating blobs, creating unseen images in the dataset like a room with two ceiling fans. Learn more in the video! Watch the video here!

BlobGAN allows for unreal manipulation of images, made super easily controlling simple blobs. All these small blobs represent an object, and you can move them around or make them bigger, smaller, or even remove them, and it will have the same effect on the object it represents in the image. This is so cool!

As the authors shared in their results, you can even create novel images by duplicating blobs, creating unseen images in the dataset like a room with two ceiling fans! Correct me if I’m wrong, but I believe it is one of, if not the first, paper to make the modification of images as simple as moving blobs around and allowing for edits that were unseen in the training dataset. 

And you can actually play with this one compared to some companies we all know! They shared their code publicly and a Colab Demo you can try right away. Even more exciting is how BlobGAN works. Learn more in the video!

Watch the video

References

►Read the full article: https://www.louisbouchard.ai/blobgan/
►Epstein, D., Park, T., Zhang, R., Shechtman, E. and Efros, A.A., 2022.
BlobGAN: Spatially Disentangled Scene Representations. arXiv preprint
arXiv:2205.02837.
►Project link: https://dave.ml/blobgan/
►Code: https://github.com/dave-epstein/blobgan
►Colab Demo: https://colab.research.google.com/drive/1clvh28Yds5CvKsYYENGLS3iIIrlZK4xO?usp=sharing#scrollTo=0QuVIyVplOKu
►My Newsletter (A new AI application explained weekly to your emails!): https://www.louisbouchard.ai/newsletter/

Video Transcript

if you think that the progress with guns

was over you couldn't be more wrong

here's blob gun and this new paper is

just incredible blob gun allows for

unreal manipulation of images made super

easily controlling simple blobs all

these small blobs represent an object

and you can move them around make them

bigger smaller or even remove them and

it will have the same effect on the

object it represents in the image this

is so cool as the authors shared in

their results you can even create novel

images by duplicating blubs creating

unseen images in the data set like this

room with two ceiling fans correct me if

i'm wrong but i believe it's one of if

not the first paper to make the

modification of images as simple as

moving blobs around and allowing for

edits that were unseen in the training

dataset and you can actually play with

this one compared to other companies we

all know they shared are called publicly

and a collab demo you can try right away

even more exciting is how bloggian works

which we'll dive into in a few seconds

to publish an excellent paper like

blobgun the researchers needed to run

many experiments on multiple machines

those who played with guns know how long

and painful this process can be plus

their code is available on github and

google collab this means their code has

to be reproducible funnily enough this

is also a really strong point of this

episode's sponsor weights and biases

weights and biases change my life as a

researcher it tracks everything you need

for your code to be reproducible the

hyper parameters the github commit

hardware usage metrics and the python

version leaving you without headaches ok

some might still appear because of

deadlines or bugs but none from trying

to reproduce experiments weights and

biases is also super helpful when

sharing your experiment results with

your colleagues a great tool for that is

reports they can act as dashboards for

supervisor pis or managers to check how

experimentation is going meaning more

time for research while improving your

feedback's quality please don't be like

most researchers who keep their code a

secret and try using weights and biases

with the first link below

now let's get back to our paper blub gun

spatially disentangled scene

representations the title says it ital

blovkian uses blobs to disentangle

objects in a scene meaning that the

model learns to associate each blob with

a specific object in the scene like a

bed window or ceiling fan once trained

you can move the blobs and objects

around individually make them bigger or

smaller duplicate them or even remove

them from the picture of course the

results are not entirely realistic but

as a great person would say just imagine

the potential of this approach two more

papers down the line

what's even cooler is that this training

occurs in an unsupervised scheme this

means that you do not need every single

image example to train it as you would

in supervised learning a quick example

is that supervised training will require

you to have all the desired

manipulations in your image that are set

to teach blobs to learn those

transformations whereas in unsupervised

learning you do not need this extensive

data and the model will learn to achieve

this task by itself associating bluffs

to objects on its own without explicit

labels we train the model with a

generator and a discriminator in a gun

fashion i will simply do a quick

overview as i've covered guns in

numerous videos before as always in guns

the discriminator's responsibility is to

train the generator to create realistic

images the most important part of the

architecture is the generator with our

blobs and a style gun 2 like decoder i

also covered style gun based generators

in other videos if you are curious about

how it works but in short we first

create our blobs this is done by taking

random noise as in most generator

networks and mapping it into blobs using

a first neural network this will be

learned during training then you need to

do the impossible take this blob

representation and create a real image

out of it this is where the gan magic

happens since you are still listening

please consider subscribing to the

channel and liking the video it means a

lot and supports my work for free also

we have a community called learn ai

together on discord to learn exchange

with fellow ai enthusiasts i'm convinced

you'll love it there and i will be glad

to meet you

we need a star gun like architecture to

create our images from these blobs of

course we added the architecture to take

the blobs we just created as inputs

instead of the usual random noise

then we turn our model using the

discriminator to learn to generate

realistic images once we have good

enough results it means our model can

take on blob representation instead of

noise and generate images but we still

have a problem how can we disentangle

those blobs and make them match objects

well this is the beauty of our

unsupervised approach the model will

iteratively improve and create realistic

results while also learning how to

represent these images in the form of a

fixed number of blobs you can see here

how blubs are often used to represent

the same objects or very similar objects

in the scene here you can also see how

the same gloves are used to represent

either a window or a painting which

makes a lot of sense likewise you can

see that light is almost always

represented in the fort blub similarly

you can see how blubs are often

representing the same regions in the

scene most certainly leads you to the

similarities of images in the dataset

used for this experiment and voila this

is how blobgan learns to manipulate

scenes using a very intuitive blob

representation i'm excited to see the

realism of the results improve keeping a

similar approach using such a technique

we could design simple interactive apps

to allow designers and anyone to

manipulate images easily which is quite

exciting of course this was just an

overview of this new paper and i

strongly recommend reading their paper

for a better understanding and a lot

more detail on their approach

implementation and tests they did as i

said earlier in the video they also

shared their code publicly and a color

demo you can try right away all the

links are in the description below

thank you for watching until the end and

i will see you next week with another

amazing paper

[Music]

0