24 Jul 2017, 10:05

Deploying a real persistent/durable etcd cluster inside kubernetes

When using an etcd cluster to store important key value data you’ll probably prefer data persistency over availability. If more than half of your etcd cluster members goes down you’ll prefer to wait for them to return back accepting a loss of availability instead of recreating a new etcd cluster restoring a backup of your data that will probably be an old version of the data at the moment of the disaster.

As an example in stolon the stolon cluster data is saved inside a store like etcd or consul. Restoring the stolon cluster data from a backup could lead to bad behaviors since the stolon cluster state (that contains different information where the primary one is which postgres is the master/primary) won’t be in sync with the restore cluster data.

Stolon was architected to be seamlessy deployed inside a k8s cluster so it becomes logical to also deploy the store (etcd or consul) inside k8s.

Today there’re different ways to deploy an etcd cluster inside k8s but, as I’m going to explain in this post, these don’t meet the above requirement:

Why these above options fail to meet our requirements?

28 Dec 2016, 13:51

Building an atreus keyboard

I always felt traditional keyboards not really good for touch typing since the staggered letters felt very innatural and my wrists were forced to be angled causing some stress on them.

Since I use the mouse as less as possible preferring all the possible keyboard shortcuts, trying to reach CTRL/ALT/SUPER without moving my hands from the home row position is not possible and time consuming. Hitting “esc” or the various punctuation required a lot of stretch of the pinkies. Also moving from the VIM “hjkl” to the default arrows requires too much movement. Additionally I always wanted to be able to configure the keyboard layout with my preferences using open source firmwares like tmk or qmk so I finally decided to build my own keyboard.

So I started looking for some more ergonomical keyboards. These were my primary requisites:

  • Use better key placement than staggered rows
  • Be able to use my thumbs not only for hitting a long spacebar
  • Use multiple layers to avoid moving my hands as much as possible and so have less keys
  • Be able to use it as primary keyboard also when moving.

Some of the most famous DIY keyboards that could partly satisfy my needs were the planck keyboard and the infinity ergodox keyboard. The planck has an “ortholinear” key placement while the ergodox has staggered columns and a thumb cluster. At a first sight the ergodox looked perfect for my needs with the exception of the portability.

Then I found the atreus and I was fashinated by its layout since it seemed to fit all my needs and, being my first self built keyboard, it looked quite easy and cheaper than other models.

I honestly don’t like the woodish case and I decided to build it using acrylic and, since the matrix is quite simple and I have a bit of experience at soldering, I don’t see the need for a pcb but choosed to manually solder it.

Finished look

This is the (almost) final (needs some more cleanup) result:

01 Mar 2016, 18:08

Immutable infrastructure with terraform and rolling upgrades of stateful services

Note: A simplified but complete way to see and test all of this in action is available on github.

Since we are always promoting new methodologies and technologies to our customers, the last years were full of projects based on IaaS (public and private) managed using Infrastructure as Code. Today, with the experience we gained in these years using different solution and technologies, I’d like to describe one way to manage an immutable infrastructure.

To make the below description the simplest I’m going to skip all the various parts related to application containers and their orchestration (that we are deeply using) (since also this is immutable infrastructure) and their way to do upgrade of stateless services (rolling vs blue/green deployments) but I’m going to just focus on the IaaS immutable infrastructure and in particular on how to handle rolling upgrades of stateful services.

I’m not going to explain here (it deserves a dedicated post) all the rationales that brought us to this solution and why some technologies over other (and this will probably change in future).

First a brief list of the primary requirements that this solution tries to satisfy:

  • We want to be able to create from scratch in minutes complete test environments as much as possible similar to the production environment. In this way we can run reliable integrations tests for every pull request (but also making the developers run the tests by themself).
  • We want to avoid the management problems and the uncertain of the upgrade consistency of our infrastructures using classic provisioning.
  • We want to create our server images and test the new infrastructure creation and upgrade before doing this in production.
  • We have to manage high available stateful services (databases etc…) that needs to be upgraded without downtime.
  • Manage independent block of the full infrastructure and handle the dependencies between them.

To achieve this primary requirements we ended up creating an IaaS Immutable infrastructure using packer for image creation and terraform for infrastructure creation/upgrade all integrated to achieve rolling upgrades of stateful services. Everything fully testable.

29 Jan 2016, 14:54

An introduction to stolon: cloud native PostgreSQL high availability

At sorintlab, in the old days and with customers with “old school” infrastructures we had different ways to make an high available PostgreSQL. We used heartbeat, heartbeat+crm, rhcs, coreosync+pacemaker with a shared storage and a correctly (!!!) configured fencing/stonith.

When moving our infrastructures and customers to cloud based infrastructures (being it a private or a public one), the way to design the infrastructures drastically (and fortunately) changed. Finally our customers were forced to think about the various parts of the infrastructure like something that can break in every moment and that’s also ephemeral. This has always been the case on every infrastructure, but previously the customers tried to ignore this fact and prefered to pay tons of money for something claimed to be “unbreakable”.

In the cloud the “stateful” layers like databases are the most difficult the get right. You can completely offload it to your cloud provider DBaaS offer (like Amazon RDS) or you can implement it by yourself inside the IaaS.

We are more and more focusing on noSQL databases that already provide, given their nature, a shared nothing, sharded and replicated architecture.

But sometimes a relational database is still the best choice or is needed by some products (for example some Hadoop components need a PostgreSQL instance).

21 Nov 2014, 15:43

go and ldap password policies
In the last days, I started working on an oauth2 and openid connect server (thanks to osin). As the project is starting as a corporate Single Sign On based on an ldap, the users needs to know when their password is expiring and be able to change it during the login phase. So some ldap client features were needed: Password policies (fedora 389 and other netscape derivatives are using this draft rfc while openldap ppolicy overlay uses this other draft rfc Password Modify Extended operation (rfc 3062) StartTLS I noticed that there were a lot of go ldap clients forks.