Make your own AI Art with Deep Neural Networks

I’m seriously rubbish at drawing and painting, but who needs arty fingers when you can whip up your own AI to make fine art for you?

Back in 2010, when The Looking Glass Club was published, my then-agent had complained that the technology in 2035 seemed too advanced for just a few decades hence, but despite justifying this with The Singularity, even I’m surprised by just how much AI has advanced in the five years since. I gave a talk about AI at Imperial a few months ago, and demo-ed how Deep Neural Networks are starting to get scarily good at things that, until now, have remained firmly in the realm of human capability, like learning to play Breakout better than you without knowing anything about the game beforehand, and imagining trippy works of art.

Yesterday, I came across @DeepForger (a Twitter bot by @alexjc) that paints impressionistic versions of your photos by copying the style and techniques of famous artists. How’s that for living in the future?  It painted this amazing portrait of me in the style of Picasso:

Okay, it’s not quite Picasso, but it’s really quite remarkable. It’s understood the stylistic essence of the art piece it was given and created something in a similar vein!

This extraordinary AI is an implementation of a neural art algorithm that was only published at the end of August!

Unsurprisingly @DeepForger has a long queue of commissions so I decided to have a go at creating my own AI Artist that I could whip into making art at will. If you want to do the same, here’s how I did it, and what I learned in the process, so that hopefully you can avoid some of the mis-steps I took.

I’m working on a setup that can handle bigger files as I write this, but here’s an Escher-inspired memorial of mum’s cat Misty, who sadly passed away a few days ago:

How to create an instance of “Neural-Style” on an Ubuntu VM

Neural-style is one of several implementations of Leon Gatys et al‘s neural art algorithm on github. If you know what you’re doing, and already have an Ubuntu server or VM, that link should be enough to get you started.

I’ve been experimenting with three different setup flavours:

1. An Ubuntu VM running on VirtualBox on my laptop
2. A more powerful Ubuntu VM created as a droplet on DigitalOcean
3. An AWS GPU instance

I’ll give detailed steps here for the first two configurations, and you can find steps for setting up an AWS GPU instance nicely documented here. (UPDATE: the AWS+docker implementation caused me too many issues, but I finally got a GPU-accelerated instance up and producing amazing results; I’ll post at the end) First though, here are some important things that can save you some pain and why I tried all three:

Learnings/Gotchas – read these first!

The two main causes of pain were lack of memory and directory permissions.

Neural-style needs at least 4GB of RAM on the Ubuntu instance if you use the default settings. You can easily brick your VM if you don’t have enough (VirtualBox hangs and goes into Guru Meditation mode you have to start over). It has a mode which only needs 1GB RAM but it doesn’t give good results AFAICT.

You can get it to work with less RAM by setting the image_size switch to either 256 or 128 pixels (default is 512) but then the images are too small to be good.

If you’re using VirtualBox you can probably give the VM about 45% of the host RAM safely – more than this and you’ll run into trouble. This is why I ended up trying DigitalOcean and AWS although I did get eventually get a version working on my laptop too.

If you use Vagrant to set up your VM then make sure you run all the commands using sudo to avoid permission errors which can be arcane and hard to figure out.

The default mode of neural-style is optimised to use a GPU but there’s a switch to force CPU.

I found that VMs with lots of cores (whether CPU or GPUs) didn’t make the code run any faster so there’s no point renting a cloud VM with loads (e.g. AWS g2.2xlarge is enough). GPUs run the code substantially than CPUs though (because matrix ops are done directly on the chip).

Setting up a VM on DigitalOcean is far simpler than AWS but they don’t have a GPU virtualisation so they are quite slow, no matter how many CPUs you go for.

Step 1 – Spin up a fresh Ubuntu VM

If you use Vagrant and VirtualBox:

1. On your host create a new directory for your project and then in powershell or cmd run vagrant init ubuntu/trusty64
2. vagrant up
3. You can either increase the memory before step 2, using the Virtual Manager, or poweroff and do it after checking your VM works.

Alternatively, if your home machine isn’t powerful enough, you can use DigitalOcean to create a cloud VM. Just create a new droplet with at least 4GB (preferably 8GB) based on Ubuntu 14.04 and SSH in. (Remember to use sudo for everything, if you’re not root which you won’t be in a VirtualBox machine.)

Then, follow these commented steps in turn to configure your VM (you have run each step separately, this isn’t a script but if anyone fancies knocking up a Chef recipe to do this, do share!)

# you'll need git
sudo apt-get install git

# 1. install lua package manager
sudo apt-get install luarocks

# 2. install lua just-in-time compiler 
sudo apt-get install luajit

# 3. install torch - a scientific computing framework for luajit
curl -s https://raw.githubusercontent.com/torch/ezinstall/master/install-all | bash

# 4. protocol buffers - google's data exchange format
sudo apt-get install libprotobuf-dev protobuf-compiler

# 5. install caffe - a deep learning framework
sudo luarocks install loadcaffe

# 6. to handle various image formats
sudo luarocks install image

# 7. a neural network library
sudo luarocks install nn

# 8. and finally, get "neural-style" from github repo
sudo git clone https://github.com/jcjohnson/neural-style.git

# 9. go into the directory you've just cloned
cd neural-style

# 10. Download the neural models that contain the art genius - this step may take a while
sudo sh models/download_models.sh 

You’re should be almost ready to make some art. You may also want to set up another shared folder for image resources, but I was lazy and just the vagrant folder on my host which was shared by default and copied stuff in/out of this directory as needed.

You will need a source art image for the style (in fact, you can use more than one but start simple) and a photo that you want to artify.

A headsup

Before you start, here’s what to expect if everything runs smoothly: we’ll use a verbose flag (-print_iter 1) so you should see steps as they happen, if things go quiet for more than 30s or so then check the state of your VM using Vbox manager to make sure it’s alive – if not, it’s probably run out of memory.

When neural-style is running, it starts by creating convolution layers which you’ll see in the output, and then after a few more steps, it should start producing image iterations. You need between 1,000 and 2,000 iterations for a good image. If you don’t see output like this with iterations every few seconds then something has gone wrong:

conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv3_4: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv4_4: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
conv5_4: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8: 1 1 4096 1000
Setting up style layer          2       :       relu1_1
Setting up style layer          7       :       relu2_1
Setting up style layer          12      :       relu3_1
Setting up style layer          21      :       relu4_1
Setting up content layer        23      :       relu4_2
Setting up style layer          30      :       relu5_1
WARNING: Skipping content loss
Running optimization with L-BFGS
Iteration 1 / 1000
  Content 1 loss: 2094767.343750
  Style 1 loss: 26350.109863
  Style 2 loss: 4467230.468750
  Style 3 loss: 1869238.085938
  Style 4 loss: 90886093.750000
  Style 5 loss: 12086.410522
  Total loss: 99355766.168823
   creating recyclable direction/step/history buffers
Iteration 2 / 1000
  Content 1 loss: 2094763.906250
  Style 1 loss: 26350.109863
  Style 2 loss: 4467230.468750
  Style 3 loss: 1869238.085938
  Style 4 loss: 90886087.500000
  Style 5 loss: 12086.410522
  Total loss: 99355756.481323

Finally: let’s make make some AI art!

If you want to do a small test run with safe options, try this (replacing the two image filenames with your real files):

# small fast images
sudo th neural_style.lua -style_image ORIGINALARTISTIMAGE.jpg -content_image YOURPHOTO.jpg -gpu -1 -image_size 128 -print_iter 1

If that works, then you can try something larger:

# default image size is 512 anyway, takes more memory the larger the file
sudo th neural_style.lua -style_image ORIGINALARTISTIMAGE.jpg -content_image YOURPHOTO.jpg -gpu -1 -image_size 512 -print_iter 1

If you absolutely have to work with less than 3GB available then you can change the optimiser but to get good results you’ll need to experiment A LOT more, and normalising gradients is recommended:

sudo th neural_style.lua -style_image ORIGINALARTISTIMAGE.jpg -content_image YOURPHOTO.jpg -gpu -1 -image_size 512 -print_iter 1 -optimizer adam -normalize_gradients 

UPDATE – AWS GPU-accelerated version with CUDA:

Docker seemed to complicate everything for me, so I went back and unpicked the dependences so I could you the plain install with AWS.

Here are the modified steps:

# follow these steps for AWS/GPU 
# http://tleyden.github.io/blog/2015/11/22/cuda-7-dot-5-on-aws-gpu-instance-running-ubuntu-14-dot-04/

# then follow ONLY the mount GPU steps here (not the docker stuff)
# http://tleyden.github.io/blog/2015/11/22/running-neural-style-on-an-aws-gpu-instance/
# and then the steps later marked optional

# you'll need git
sudo apt-get install git

# install lua package manager
sudo apt-get install luarocks

# install lua just-in-time compiler 
sudo apt-get install luajit

# install torch - a scientific computing framework for luajit
# (takes quite a while!)
curl -s https://raw.githubusercontent.com/torch/ezinstall/master/install-all | bash

# protocol buffers - google's data exchange format
sudo apt-get install libprotobuf-dev protobuf-compiler

# install caffe - a deep learning framework
sudo luarocks install loadcaffe

# to handle various image formats
sudo luarocks install image

# a neural network library
sudo luarocks install nn

# and finally, get "neural-style" from github repo
sudo git clone https://github.com/jcjohnson/neural-style.git

# (optional for GPU/AWS) install cutorch - a CUDA backend for torch 
# see https://en.wikipedia.org/wiki/CUDA
sudo luarocks install cutorch
sudo luarocks install cunn

cd neural-style

sudo sh models/download_models.sh

# you may need SFTP set up too - I assume you can work that out

# now you can paint FAST!
sudo th neural_style.lua -style_image style.jpg -content_image subject.jpg -print_iter 1

And this is the first result I got! Stunning!

And a much better Escher-cat:

Have fun! And share your art!

Gruff