Elixir vs Ruby Showdown - Phoenix vs Rails

Phoenix vs Rails

This is the second post in our Elixir vs Ruby Showdown series. In this latest installment, we’re exploring the performance of the Phoenix and Rails web frameworks when put up against the same task. Before we get into code samples and benchmark results, let’s answer a few common questions about these kinds of tests:

tl;dr Phoenix showed 10.63x more throughput over Rails when performing the same task, with a fraction of CPU load

FAQ

Isn’t this apples to oranges?

No. These tests are a direct comparison of our favorite aspects of Ruby and Rails with Elixir and Phoenix. Elixir has the promise to provide the things we love most about Ruby: productivity, metaprogramming, elegant APIs, and DSLs, but much faster, with a battle-tested concurrency and distribution model. The goals of this post are to explore how Elixir can match or exceed our favorite aspects of Ruby without sacrificing elegant APIs and the productive nature of the web frameworks we use.

Are benchmarks meaningful?

Benchmarks are only as meaningful as the work you do upfront to make your results as reliable as possible for the programs being tested. Even then, benchmarks only provide a “good idea” of performance. Moral of the story: never trust benchmarks, always measure yourself.

What are we comparing?

Elixir Phoenix Framework

Phoenix 0.3.1
Cowboy webserver (single Elixir node)
Erlang 17.1

Ruby on Rails

Rails 4.0.4
Puma webserver (4 workers - 1 per cpu core)
MRI Ruby 2.1.0

We’re measuring the throughput of an “equivalent” Phoenix and Rails app where specific tasks have been as isolated as possible to best compare features and performance. Here’s what we are measuring:

Match a request from the webserver and route it to a controller action, merging any named parameters from the route
In the controller action, render a view based on the request Accept header, contained within a rendered parent layout
Within the view, render a collection of partial views from data provided by the controller
Views are rendered with a pure language templating engine (ERB, EEx)
Return the response to the client

That’s it. We’re testing a standard route matching, view rendering stack that goes beyond a Hello World example. Both apps render a layout, view, and collection of partials to tests real-world throughput of a general web framework task. No view caching was used and request logging was disabled in both apps to prevent IO overhead. The wrk benchmarking tool was used for all tests, both against localhost, and remotely against heroku dynos to rule out wrk overhead on localhost. Enough talk, let’s take a look at some code.

Routers

Phoenix

defmodule Benchmarker.Router do
  use Phoenix.Router
  alias Benchmarker.Controllers

  get "/:title", Controllers.Pages, :index, as: :page
end

Rails

Benchmarker::Application.routes.draw do
  root to: "pages#index"
  get "/:title", to: "pages#index", as: :page
end

Controllers

Phoenix (request parameters can be pattern-matched directly in the second argument)

defmodule Benchmarker.Controllers.Pages do
  use Phoenix.Controller

  def index(conn, %{"title" => title}) do
    render conn, "index", title: title, members: [
      %{name: "Chris McCord"},
      %{name: "Matt Sears"},
      %{name: "David Stump"},
      %{name: "Ricardo Thompson"}
    ]
  end
end

Rails

class PagesController < ApplicationController

  def index
    @title = params[:title]
    @members = [
      {name: "Chris McCord"},
      {name: "Matt Sears"},
      {name: "David Stump"},
      {name: "Ricardo Thompson"}
    ]
    render "index"
  end
end

Views

Phoenix (EEx)

...
    <h4>Team Members</h4>
    <ul>
      <%= for member <- @members do %>
        <li>
          <%= render "bio.html", member: member %>
        </li>
      <% end %>
    </ul>
...

<b>Name:</b> <%= @member.name %>

Rails (ERB)

...
    <h4>Team Members</h4>
    <ul>
      <% for member in @members do %>
        <li>
          <%= render partial: "bio.html", locals: {member: member} %>
        </li>
      <% end %>
    </ul>
...

<b>Name:</b> <%= member[:name] %>

Localhost Results

Phoenix showed 10.63x more throughput, with a much more consistent standard deviation of latency. Elixir’s concurrency model really shines in these results. A single Elixir node is able to use all CPU/memory resources it requires, while our puma webserver must start a Rails process for each of our CPU cores to achieve councurrency.

Phoenix:
  req/s: 12,120.00
  Stdev: 3.35ms
  Max latency: 43.30ms

Rails:
  req/s: 1,140.53
  Stdev: 18.96ms
  Max latency: 159.43ms

Phoenix

$ mix do deps.get, compile
$ MIX_ENV=prod mix compile.protocols
$ MIX_ENV=prod elixir -pa _build/prod/consolidated -S mix phoenix.start
Running Elixir.Benchmarker.Router with Cowboy on port 4000

$ wrk -t4 -c100 -d30S --timeout 2000 "http://127.0.0.1:4000/showdown"
Running 10s test @ http://127.0.0.1:4000/showdown
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.31ms    3.53ms  43.30ms   79.38%
    Req/Sec     3.11k   376.89     4.73k    79.83%
  121202 requests in 10.00s, 254.29MB read
Requests/sec:  12120.94
Transfer/sec:     25.43MB

Rails

$ bundle
$ RACK_ENV=production bundle exec puma -w 4
[13057] Puma starting in cluster mode...
[13057] * Version 2.8.2 (ruby 2.1.0-p0), codename: Sir Edmund Percival Hillary
[13057] * Min threads: 0, max threads: 16
[13057] * Environment: production
[13057] * Process workers: 4
[13057] * Phased restart available
[13185] * Listening on tcp://0.0.0.0:9292

$ wrk -t4 -c100 -d30S --timeout 2000 "http://127.0.0.1:9292/showdown"
Running 10s test @ http://127.0.0.1:9292/showdown
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    21.67ms   18.96ms 159.43ms   85.53%
    Req/Sec   449.74    413.36     1.10k    63.82%
  11414 requests in 10.01s, 25.50MB read
Requests/sec:   1140.53
Transfer/sec:      2.55MB

Heroku Results (1 Dyno)

Phoenix showed 8.94x more throughput, again with a much more consistent standard deviation of latency and with 3.74x less CPU load. We ran out of available socket connections when trying to push the Phoenix dyno harder to match the CPU load seen by the Rails dyno. It’s possible the Phoenix app could have more throughput available if our client network links had higher capacity. The standard deviation is particularly important here against a remote host. The Rails app struggled to maintain consistent response times, hitting 8+ second latency as a result. In real world terms, a Phoenix app should respond much more consistently under load than a Rails app.

Phoenix:
  req/s: 2,691.03
  Stdev: 139.92ms
  Max latency: 1.39s

Rails:
  req/s: 301.36
  Stdev: 2.06s
  Max latency: 8.36s

Phoenix (Cold)

$ ./wrk -t12 -c800 -d30S --timeout 2000 "http://tranquil-brushlands-6459.herokuapp.com/showdown"
Running 30s test @ http://tranquil-brushlands-6459.herokuapp.com/showdown
  12 threads and 800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   317.15ms  139.55ms 970.43ms   81.12%
    Req/Sec   231.43     66.07   382.00     63.92%
  83240 requests in 30.00s, 174.65MB read
  Socket errors: connect 0, read 1, write 0, timeout 0
Requests/sec:   2774.59
Transfer/sec:      5.82MB

Phoenix (Warm)

$ ./wrk -t12 -c800 -d180S --timeout 2000 "http://tranquil-brushlands-6459.herokuapp.com/showdown"
Running 3m test @ http://tranquil-brushlands-6459.herokuapp.com/showdown
  12 threads and 800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   318.52ms  139.92ms   1.39s    82.03%
    Req/Sec   224.42     57.23   368.00     68.50%
  484444 requests in 3.00m, 0.99GB read
  Socket errors: connect 0, read 9, write 0, timeout 0
Requests/sec:   2691.03
Transfer/sec:      5.65MB

Load

load_avg_1m=2.78

sample#memory_total=34.69MB
sample#memory_rss=33.57MB
sample#memory_cache=0.09MB
sample#memory_swap=1.03MB
sample#memory_pgpgin=204996pages sample#memory_pgpgout=196379pages

Rails (Cold)

$ ./wrk -t12 -c800 -d30S --timeout 2000 "http://dry-ocean-9525.herokuapp.com/showdown"
Running 30s test @ http://dry-ocean-9525.herokuapp.com/showdown
  12 threads and 800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.85s     1.33s    5.75s    65.73%
    Req/Sec    22.68      7.18    61.00     69.71%
  8276 requests in 30.03s, 18.70MB read
Requests/sec:    275.64
Transfer/sec:    637.86KB

Rails (Warm)

$ ./wrk -t12 -c800 -d180S --timeout 2000 "http://dry-ocean-9525.herokuapp.com/showdown"
Running 3m test @ http://dry-ocean-9525.herokuapp.com/showdown
  12 threads and 800 connections
    Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.07s     2.06s    8.36s    70.39%
    Req/Sec    24.65      9.97    63.00     67.10%
  54256 requests in 3.00m, 122.50MB read
  Socket errors: connect 0, read 1, write 0, timeout 0
Requests/sec:    301.36
Transfer/sec:    696.77KB

Load

sample#load_avg_1m=10.40

sample#memory_total=235.37MB
sample#memory_rss=235.35MB
sample#memory_cache=0.02MB
sample#memory_swap=0.00MB
sample#memory_pgpgin=66703pages
sample#memory_pgpgout=6449pages

Summary

Elixir provides the joy and productivity of Ruby with the concurrency and fault-tolerance of Erlang. We’ve shown we can have the best of both worlds with Elixir and I encourage you to get involved with Phoenix. There’s much work to do for Phoenix to match the robust ecosystem of Rails, but we’re just getting started and have very big plans this year.

Both applications are available on Github if want to recreate the benchmarks. We would love to see results on different hardware, particularly hardware that can put greater load on the Phoenix app.

Shoutout to Jason Stiebs for his help getting the Heroku applications setup and remotely benchmarked!

FAQ

Isn’t this apples to oranges?

Are benchmarks meaningful?

What are we comparing?

Elixir Phoenix Framework

Ruby on Rails

Routers

Controllers

Views

Localhost Results

Phoenix

Rails

Heroku Results (1 Dyno)

Phoenix (Cold)

Phoenix (Warm)

Load

Rails (Cold)

Rails (Warm)

Load

Summary

Sign up to receive updates on all things Littlelines