Robin Percy

Building a Batch Image Uploader in Elixir, Phoenix, Ruby and Iron

Building a Batch Image Uploader in Elixir, Phoenix, Ruby and Iron

In the modern world of social media, many of us have online albums full of photos. For us developers who build the infrastructure for these photo albums, there are many existing options available to us. However, when it comes to Elixir and Phoenix, there aren’t any ready-made options available. Before we dive into our app, allow me to introduce our chosen technologies, for those unfamiliar.

Elixir is a functional, dynamically-typed language, that sits atop of the Erlang VM. Elixir was built with the aims of delivering low-latency, fault tolerant, distributed systems, and allows for huge vertical scaling as it runs its code inside incredibly lightweight threads of execution (similar to Goroutines in Golang, and threads in Crystal.) These threads are all entirely isolated, to allow for independent garbage collection and reduction of system-wide pauses. This uses our machine’s resources more efficiently. Elixir also allows for massive horizontal scaling, as those threads can communicate with other process threads running on separate machines.

If you would like to learn more Elixir, I would recommend the Programming Elixir book available from the Pragmatic Bookshelf. I purchased this book a few years ago and it really did aid my Elixir education, making the learning process simple, and offering practical guides and programming tasks that show-off the features of the language really rather well.

Phoenix brings back the simplicity and joy in writing modern web applications by mixing tried and true technologies with a fresh twist of functional ideas. Phoenix leverages the Erlang VM’s ability to handle millions of connections, alongside Elixir’s beautiful syntax and productive tooling for building fault-tolerant systems. I have used Phoenix for every web app I’ve built in the last couple of years, and I’ve had an absolute blast in doing so!

In our application today, we’re going to use Phoenix to build a web application in which people can create albums of photos that are uploaded to Amazon S3 buckets, which Iron then processes — resizing and colouring the images.

This example app demonstrates a multiple-image uploader in Phoenix, which as yet, there are no definitive tutorials on. It also demonstrates Iron scheduling the batch processing of those images inside user’s albums.

By building our batch image processor app using Ruby* *and IronWorker, I am emulating the very-real-life situation of having a polyglot SOA (Service Oriented Architecture), in which we rely on multiple smaller applications, commonly in a variety of languages and technologies, to concurrently perform the tasks of an equivalent monolith application.

Before we get into our application build, there are a few things we need to install, namely: Elixir, Phoenix, Ruby, Iron CLI and Postgres. In my opinion, Ruby is best managed using Rbenv, and all of the other technologies can be installed with Homebrew or an equivalent package/app manager. I won’t go into details on installing them here, however, you can find tutorials for each at the following locations:

With those installed, let’s crack on!

Note: The completed Repos for this sample application can be found here:

  • Imagey — Phoenix Web Application

  • ImageProc — Iron.io / Ruby Image Processor

Creating our Phoenix App

In your Terminal, run the following command:

$ mix phx.new imagey --no-brunch

In this command, I’ve included the --no-brunch flag, as I personally don’t like using the JS brunch app to manage assets. Certainly, in this sample application it would be overkill. (Personal preference - you can totally use it if you choose to!) With our Phoenix* *app generated, open up your mix.exs file, and add the following to your dependencies:

Embedded content: https://gist.github.com/rbnpercy/4da5d6efa793d2e88f2c4f21c65ad302#file-deps-exs

And in the same file, update your application func with the following:

Embedded content: https://gist.github.com/rbnpercy/f6ecb8e4c9f36b3f638512d8d1349b7e#file-mix-exs

Now go ahead and bundle your dependencies by running:

$ mix deps.get

The next thing we need to do is update our config.exs file with our S3 and Database* *credentials:

Embedded content: https://gist.github.com/rbnpercy/0fca293c62b491a9e5d341ae5d4bf557#file-config-exs

If you haven’t already created the database in Postgres, you can do so by running the following mix command:

$ mix ecto.create

You can now run iex -S mix phx.server to start the Pheonix server within an interactive window, and preview your application.

With your database now connected, and configuration ready, our next move is to generate the structuring of our Album and Image resources.

Generating Album & Photo Resources

Let’s start off with our Album resource. Run the following generator:

$ mix phx.gen.html Images Album albums name:string --binary-id

This will generate all of the relevant resources for our Album objects including model, view, controller, and templates, all contained within the Image context. Running that command will return the following instructions, which you should now follow:

Add the resource to your browser scope in lib/imagey_web/router.ex:

Embedded content: https://gist.github.com/rbnpercy/c5835c55a37779511ac89cba2510e1de#file-router-ex

Remember to update your repository by running migrations:

$ mix ecto.migrate

With that, we have our Album resources created, and ready for the Photos resources that are to be contained within. Let’s run the generator again, this time for our Photo resources:

$ mix phx.gen.html Images Photo photos image:string album_id:string --binary-id

Once again, we get the following instructions returned, so go ahead and implement them too:

Add the resource to your browser scope in lib/imagey_web/router.ex:

Embedded content: https://gist.github.com/rbnpercy/06af342fa3859601ebaab29b1638532e#file-router-ex

Once again, remember to update your repository by running migrations:

$ mix ecto.migrate

Create Frontend Image Uploader

To allow image uploads, we need to enable multipart form submissions for our Photo resources. Open up templates/photo/form.html.eex and change it to the following:

Embedded content: https://gist.github.com/rbnpercy/ce5043d6477de6b63b5ac2ca402199dd#file-form-html-eex

Setting up Phoenix Associations

Now with our Albums and Photo resources in, we need to set up the associations between them. Phoenix and Ecto* *provide great association protocols and make it easy for us to implement these. Open up lib/imagey/images/album.ex and add the following into the model:

Embedded content: https://gist.github.com/rbnpercy/448fb98280d1cd18e01aa4ad9123235a#file-album-ex

This has_many protocol references our Photo objects automatically using referential ID’s. Phoenix handles this in the backend without any further effort from us. Sweet. That’s not it though; we need to tell our Photo* objects to reference Albums *too.

Open up lib/imagey/images/photo.ex and add the following:

Embedded content: https://gist.github.com/rbnpercy/5acac053522abff78bc27d9474b31271#file-photo-ex

Alongside this, keeping in photo.ex, ensure your changeset func reflects the following, to include the Album ID in photo creation:

def changeset(photo, attrs) do
  photo
  |> cast(attrs, [:album_id, :image])
  |> validate_required([:image])
**end**

Now once again, go ahead and run:

$ mix ecto.migrate

At this point, if you haven’t had Phoenix running throughout this tutorial, you can test it out by running iex -S mix phx.server. Browse to *http://localhost:4000/photos* to test out the uploading functionality so far. Note that at the moment, you can only upload 1 photo at a time. We will move onto our Phoenix Multiple Image uploader shortly.

When you view the album/photos index here, you may notice that you get the following error:

**#Ecto.Association.NotLoaded<association :photos is not loaded>**

To solve this, we need to tell Phoenix to preload our Photo association into our Albums and Photos respectively. Open up lib/imagey/images/images.ex:

Change your get_album!() func like this:

Embedded content: https://gist.github.com/rbnpercy/0c1a325f4d6a140ce4681dd41891c6b7#file-images-ex

And in the same images.ex change your list_photos()and get_photo!() funcs to reflect the following:

Embedded content: https://gist.github.com/rbnpercy/fcfcd9116e585d6af5be79809f342095#file-images-ex

Create Phoenix S3 Image Uploader

Moving onto the big guns of our application, it’s time to create our S3 Image Uploader. Open up controllers/photo_controller.ex, and change your create() function to reflect this:

Embedded content: https://gist.github.com/rbnpercy/93701213011c3aa7d1ff526a1955c4e0#file-photo_controller-ex

Then we need to create the following upload function, that create() calls:

Embedded content: https://gist.github.com/rbnpercy/127a1788dfe3bcf846e8d7dc8cfb41ad#file-photo_controller-ex

Let’s break this function down! The first thing to note, is that our upload function takes an argument. This argument is the map of image_params passed to our create() function when the online form is submitted with the photo. From that map of params, we extract the file extension (jpg/png etc.) from the upload.filename. We then use Ecto’s in-build UUID generator to create a UUID for our image for S3 storage. We create a flnm string built up from the new UUID and file extension.

For our S3 upload, we then read the binary data of the image into a file_binary variable via the tuple pattern matching. We then call the ExAws.S3 module with our stated S3 bucket, providing the filename to upload to, and the image binary data to assign to said filename. And that’s that! The image is uploaded to our S3 bucket, and will appear in the S3 control panel:

So we have our Phoenix application uploading a Photo* *that belongs to an Album. For the sake of this sample application, realistically, albums are not assigned to a user, as we have not implemented User management — that is another blog for another day. If however, you would like to implement User management I would recommend using the Coherence library. Coherence is to Phoenix what Devise is to Ruby on Rails.

If you’re coming from a Rails background, it will prove very familiar territory. There is also a Coherence Demo Project available to help guide you. As well as this, take note that when we upload photos, we have to enter the Album ID ourselves. In the real world, this would be automatically inferred.

Anyway, regardless of Album assignment for demo purposes, our application still only allows singular image uploads. We need to fix that!

Multiple Image Uploader in Phoenix

The first change we need to make is to our html form template. Open up templates/photo/form.html.eex and change the file input for the image, as follows:

Embedded content: https://gist.github.com/rbnpercy/a910a0cbe9d869ea26fd2ca609987ada#file-form-html-eex

Now that we are allowing multiple selection of images in our frontend form, we need to enumerate over these images and process them accordingly. Open up controllers/photo_controller.ex, and change the create() function:

Embedded content: https://gist.github.com/rbnpercy/f1856b39562bb87df817355b5d77de57#file-photo_controller-ex

Firstly, we are grabbing our Album ID from the submitted photo_params. With that Album ID , we create a new child S3 Bucket, so each user’s photo album has its own S3 container. Then, to process each image uploaded, we are Enumerating over them, mapping the params to img for access. The code follows our earlier S3 upload code, but then calls a store() function at the end, passing in the Plug conn* *, the created filename, and the album id of the image. We now need to create that store() function, so let’s do so, just above our create() function:

Embedded content: https://gist.github.com/rbnpercy/d5b75b67b9ff3634e15648393dbe7747#file-photo_controller-ex

You might note that this store() function is actually our original create() function! The only difference being, that on successful upload and creation, we redirect the user to the Album of pictures, instead of the former singular photo created.

If you boot your Phoenix app now, and select multiple images for the file uploader, the successful logs in your terminal will look like the following:

And your S3 bucket will contain the multiple images you just uploaded:

Now that we have implemented our S3 uploader, we need to change our frontend Album view to display our photos, from their S3 routes. Open up templates/album/show.html.eex and have it reflect:

Embedded content: https://gist.github.com/rbnpercy/b50f113fed52ff5f847150afca081fe2#file-show-html-eex

Note that in this template, the S3 bucket and containing folder are referenced to imagey_elixir, and you must enter your own specific Bucket in the image source. The child bucket, however, reflects the bucket created for the user’s photo album.

We have now successfully implemented a multiple image uploader in Phoenix! As stated in the intro of this article, these images are also processed once uploaded to S3, to make them all uniformly sized, and applied with colouration. This is done by an Iron.io scheduled process that we will implement now.

Ruby Iron.io Image Processing

The first thing to do, for our image processing app, is to create a skeleton app structure, and include some files to be completed. Create the directory imageprocand touch files: process.rb, Gemfile, iron.json, and Dockerfile, then complete them as follows:

Dockerfile:

FROM iron/ruby

WORKDIR /app
ADD . /app

ENTRYPOINT ["ruby", "process.rb"]

iron.json:

{
  "token": "YOUR_IRON_TOKEN",
  "project_id": "PROJ_ID"
}

Gemfile:

source 'https://rubygems.org'

gem 'iron_worker'
gem 'aws-sdk'
gem "mini_magick"

Finally, let’s create the main Ruby script to enumerate over uploaded photos, and apply the resizing & colour normalising.

imageproc.rb:

Embedded content: https://gist.github.com/rbnpercy/7f6ad9564b5d863beea0d13adcd6fe66#file-imageproc-rb

The first thing to note about this Ruby script, is that we use the MiniMagick Gem to interface with ImageMagick. We pass the MiniMagick gem the dimensions we wish our photos to be resized to, using our Iron Payload.

We also pass the details of the S3 Bucket (photo album) to our processing script from our Iron JSON payload.

Overall, this script alone isn’t much use — and especially if it’s run without receiving a JSON payload from Iron. So, the way to fix this, is to turn this Ruby script into an actual, useful application. We do this by bundling it up, and shipping it into Iron.io.

In my previous article, I covered this process in details, so will cover it slightly more briefly here. The first thing to do (as Iron run their infrastructure on Docker), is to package our script up into a Docker image. Run the following, from the script’s directory:

$ docker build -t USERNAME/imageproc:1.0.0 .

On success, we then need to push that image upto Docker as follows:

$ docker push USERNAME/imageproc:1.0.0

And with that, we need to register the application with Iron:

$ iron register USERNAME/imageproc:1.0.0

Once this is done, we are ready for our Iron.io Ruby application to be called from our Elixir Phoenix application. We can do this by calling a HTTPotion POST request from our photo store() function:

Embedded content: https://gist.github.com/rbnpercy/79530193be12c701354fe844d4ad45a7#file-photo_controller-ex

And with that, we have a multiple image uploader in Phoenix/Elixir, that runs an automatic photo editor using Ruby and Iron. For the image processing, you actually have multiple options with regards to running the script. Naturally, running it upon album upload as in this app, is the obvious route to take. However, because of Iron’s scheduling you could perhaps choose to have the processing run once a day, including every uploaded album.

Allowing multiple image uploads through Phoenix was a topic I could find absolutely no fully-complete tutorials on, so I do hope that this article is of great help to anyone looking to implement such a feature, without having to turn to further external packages etc.

I’d like to thank you for going along with this long article, and please feel free to leave any comments here on this article, or alternatively, reach out any time — robin@percy.pw!

Happy Hacking!

@rbin

IronWorker
IronWorkerHighly available and scalable task queue / worker service. Starting at $26/mo.
Try Nowarrow_right
Stratus Background
StratusUpdate

Sign up for the Stratus Update newsletter

With our monthly newsletter, we’ll keep you up to date with a curated selection of the latest cloud services, projects and best practices.
Click here to read the latest issue.