What we learned @dotScale conf:
Managing the unmanageable
As modern software has been growing and has become ever more complex, much of our information systems evolution was slowed down. For a few years now, new solutions have been emerging to shift paradigms and solve the sheer complexity of managing the enormous quantity of data we’re all working on.
DotScale is the conference exploring these issues and the 2016 edition showed more than ever, that the scaling ecosystem is seething with excitement.
Check out our notes below to see what noteworthy trends are going to influence large scale systems engineering in the upcoming years. Some technologies are definitely worth watching over or trying!
1. About Erlang with Mickaël Rémond
Mickaël Rémond presents the Erlang language which is a surprising good answer to current challenges.
For those of you who do not know much about Erlang, it is a somewhat old and niche language which was started in the mid 80’s to address the development of telephony applications by Ericsson.
It was built to robustly enable vertical and horizontal scaling, with a “let it crash” philosophy. It is particularly interesting to note that thirty years later, this niche issue has become widespread among startups. The multiplicity of machines we use makes the Erlang language and concepts perfectly adequate to answer current challenges.
Erlang implements the Actor model which provides a simple & strong abstraction for concurrency and distribution. It makes writing large scale distributed applications easier, by avoiding lower level concepts such as multi-threading. The concepts has since then been applied to many languages like in Akka for the JVM which we use at Logmatic.io. Languages like Elixir give a friendlier face to the Erlang language with syntax updates. Some other projects like Ejabberd have already been very successful used in the scaling behind WhatsApp (check out the full article here).
→ As our own stack entails distributed systems with important performance and resistance challenges, we’re keeping an eye on Erlang language developments. If like us, you work on highly distributed, low latency, fault tolerant applications, we strongly advise you check the underlying concepts out. They will be invaluable tools in your distributed engineering toolbelt.
2. About encapsulation and containers with Eric Brewer and Sandeepan Banerjee
As information system became ever more complex, slicing responsibilities into ever smaller services became an answer to the bugs and the slowed innovation pace of these convoluted and tangled systems. But dealing with micro services meant dealing with matrices of services that were barely hand-manageable. It was complex and difficult to implement within our infrastructures, especially if your team was not growing as rapidly as your system was. Docker and other related tools such as Kubernetes are pretty useful. Higher levels of code encapsulation allow teams to just describe deployments and thus ensure easy scaling with:
- A reduced stickiness to your infra provider
- Environment reproducibility, from dev to staging to prod
- Utilizing your machines to their full capacity for better rentability of resources consumption
Containers is the catalyst that will allow the en masse entrance of microservices in the tech world, which in turns is key to scaling.
The containers phenomenon allows for the consumption of infrastructure. It enables easy app portability, in scenarios where you can run pieces of your infra on Amazon or anywhere else. An obstacle remains though: while moving stateless services is easy, moving stateful services (ie with attached data like databases) is still troublesome. In a distributed environment, when moving a container (like an Elasticsearch node), we also want its content to move with it (i.e. data in this case), and data is often huge and difficult to move. But things are moving on in this area too. Tools like Flocker already address the stateful container case, while Kubernetes is going to tackle it on upcoming iterations.
Stateless containerized apps are efficiently and easily managed but stateful app (databases for example) remain a challenge.
→ As we’re confronted on a daily basis with infrastructure scaling, working on machine resources, deployment and monitoring, the cloud has become critical to us. We’re looking into tools that help us manage our growing number of services and machines. In light of the recent DCOS open-sourcing, frequent improvements to Kubernetes or Docker Swarm and also new commercial offerings such Azure Container Service, this is a space you definitely want to keep an eye on!
3. About hacking a business with Backblaze’s CEO Gleb Budman
Gleb Budman tells us a story of finding a strong focus and pursuing it all along while hacking solutions. He explains how he defined his Backblaze focus as simple, unlimited backup storage for consumers for 5$/a month. Having a clear and focused purpose allowed them later on to disregard features such as speed performance or nearly perfect uptime, as both were not needed for what they wanted to achieve.
Later down the line, he refused to pay what he called the “IT tax” for power supplies, where changing the shape of its containing box is billed at high rates to companies.
→ Awesome presentation that gives your thrills of excitement and pushes you to reconsider the walls on your path that you’ve become accustomed to, and push through ahead. Though being basically a hardware story, it can directly be applied to software. We recommend watching it just to get the day going with positive energy and excitement (video available on the dotConferences website.)!