Reactive Systems: It’s Actors All the Way Down

Much of what we write about on the Icon blog is instant payments and instant payment adoption around the world. This blog is part of a series, which aims to bring a bit of a change of pace, where I present some of the technical learnings that we’ve picked up while developing the Instant Payments Framework (IPF) – our elastic, high-availability, always-on instant payments solution.

You may have heard of the Reactive Manifesto – in a nutshell, it’s about building flexible, scalable, fault tolerant systems that are loosely coupled. These are all nice words that make us feel warm and fuzzy inside, but:

• what does flexibility, scalability and fault tolerance actually mean in the real world? And…
• what benefits do we get from those?

In this post I’ll try to highlight the benefits of building a reactive system with some examples and benchmarks, and, hopefully, learn a thing or two along the way.

One of the core libraries that we use in developing IPF is Akka – it’s an open source implementation of the actor model for the JDK, and is at the heart of most of the components that make up the IPF product family. The actor model exhibits a lot of reactive features by being fault-tolerant (self-healing), massively scalable (because an actor does not a thread make), and super performant. The main mantra of those in the Akka community is “let it crash” – this comes from the telecom world where dropping a single link shouldn’t make the entire system die in a heap. The system should be designed in an atomic way such that a single unit of work failing to complete shouldn’t cause the entire system to judder to a halt – rather, report the problem and move on with your life.

We’ll leave the fault tolerance point for another blog post but, to test the performance and scalability point, let’s devise a simple experiment to pit Akka and the actor model against everybody’s favourite tried-and-true Spring Framework.

The Experiment

We’re going to create a REST service which will calculate and return (as a JSON array) the first n primes, where n is grabbed via a path variable at /primes/{n}. A weird use case and definitely not something that you’ll ever implement, but hey – who’s writing this blog?!

We’ll have one implementation that does the actual prime calculation, but with two endpoint implementations: One with a Spring Web MVC Controller – a classic choice. The second will be backed by an Akka actor system with Akka HTTP at the front. If you’re interested in the source, I’ve linked to the GitHub repo at the end of this post.

The Test

We’ll create a JUnit test that will use OkHttp by Square to make 1000 requests concurrently against each endpoint implementation. We’ll request the first 20 prime numbers over and over again, and gather the following stats afterwards:

• Minimum latency
• Maximum latency
• Mean latency
• 95th percentile
• Requests processed per second

We’ll also ignore the first test run as a “warm-up” and only count results for the second test run (i.e. requests 1001-2000, or 1000-1999 if you’re a developer).

The hardware

This bit is awkward. The laptop I’m doing this on isn’t exactly dernier cri – so you may have to shield your eyes for the next bit while I list the specs:

• CPU: Intel i5-6200U @ 2.40 GHz (dual core)
• RAM: 8GB

Ouch. Let’s move on.

In the red (Spring) corner…

Let’s first try out our Spring Framework endpoint implementation. We ran 1000 requests concurrently against it, and…

OK! That’s pretty good, and definitely faster than if I’d tried to work out the first 20 primes a thousand times myself. And what’s up with that 1872 ms response? Anyway, let’s see how the other contender does…

In the blue (Akka) corner…

Right, let’s see how Akka does:

Wow! So it’s processing nearly 100 messages more per second than the default Spring MVC implementation. Also:

• the fastest response time was 2 ms faster than Spring’s fastest. No biggie when you’re calculating primes, but it definitely can make a difference in other applications.
• The slowest request took 567 ms, as opposed to 1872 in the Spring corner
• 95% of requests completed in 368 ms (as opposed to 757 with Spring)

I actually suspect that the difference wouldn’t be so stark on a more powerful machine (or a server), but doing this on a low-spec machine kind of highlights the fact that Akka is making better use of system resources.

Here are some “fancy” graphs that I came up with using sophisticated data analytics software (Excel) that put the text in a more visual way, if you’re that way inclined:

Fig. 1: Spring

Clearly there’s a couple of huge spikes near the end (well, there’s not really an “end” since all the requests were sent at once), also there are frequent blips at around the 600 ms mark.

Fig. 2: Akka HTTP

Much smoother with Akka HTTP! Not as many anomalous requests and a tighter shape overall.

Conclusion

So what have we learned? We’ve talked about Akka and how it’s a really good starting point for implementing reactive systems if you’re heading on that type of journey. We also demonstrated that in a fairly common use case (an HTTP endpoint that’s serving up JSON), Akka HTTP performs better than Spring MVC (the current default choice for such a thing) by a not-insignificant margin. This is probably because each request to Akka HTTP starts up a temporary (ephemeral) actor which is responsible for handling only that request. There’s a dispatcher which matches the requested route, and once a route is found, a new actor is spun up to deal with the request and we can move on to processing another request. There’s definitely an air of pattern matching going on here – which is probably derived from Akka’s Erlang spirit.

Akka HTTP has a much smaller user base than Spring, and an even smaller user base that uses the Java bindings, so it’s a bit tough to find community support when you come up short. Thankfully, the Akka docs are excellent and were a great resource when we were getting up to speed with Akka and Akka HTTP.

There’s a bit of a learning curve because of the programming model. It’s a lot more functional than what you might call “traditional” Java. Everything’s a lambda and as a result you need to get your head around how to grab a path variable or marshal/unmarshal a request body, for example. But again the docs are a great example of how to guide a user through unfamiliar territory, and how to write software documentation in general.

Also a bit of full disclosure: after doing some research into the topic, I have discovered that you can actually make a Spring controller do deferred processing and do things like complete a request later with a CompletableFuture  – which could bring its performance up to an Akka HTTP level, but after this many words I think I’ll make that an exercise for the reader.

References

To create the head2head application, I used Spring Boot and my knowledge of Spring @RequestMappings to get up and running, and for Akka HTTP I pretty much copy-and-pasted…er…reused the source from the intro in the docs which you can find here, and also referred to some of the Akka HTTP code that we have in the IPF source to grab the primeCount path variable.

You can find the source for the head2head application at http://github.com/paltaie/akka-spring-head2head – feel free to fork it or issue a PR if you have any improvements you’d like to make. The Akka application runnable is here, and the Spring one is here. Enjoy!

This blog is part of a series, read the next one here.