After a few years of putting together and playing with all kinds of services, one question started getting more annoying every time it came up: Do we really need another VM for that or can we fit this on a existing one?
At first, it might seems like a good idea to creating a virtual machine for every single app or service. Just to keep everything well structured and avoid a dependency nightmare. But even with templating and vm cloning, you need to maintain a full operating system for each service, which creates a lot of work if you want to keep everything up to date. Additionally you are wasting memory with running a full Linux instance for each service.
So we ended up put similar things together,.. like all php apps on one webserver. But the more we put on top of that pile of webapps the more fragile and unstable the hold thing grew,.. up to a point where we started to schedule regular reboot to keep things running.
Now we started moving in a new direction. Docker container allows us to build and maintain our service units rapidly bud in a clean and by design reproducible way. With a 5-node CoreOS cluster we can deploy the services flexible and elegantly. For Storage we are using a 2-node Gluster and a 3-node MariaDB Galera (MySQL) cluster. Together with the CARP based pfSense HA-Firewall in front we had for awhile now, we are slowly moving towards full Fault-Tolerance.