I’ve been ranting about Terraformalot recently, and I’ve shown some pretty
neat examples of building a VPC – however before now I haven’t got anything actually
useful running.
In this post, we’re gonna change that, and launch ourselves a publicly
available and scalable PAAS (Platform As A Service), built around Apache Mesos and Marathon
with nginx and Amazon’s ELB for load balancing and Route53 for DNS.
Terraform 0.4.0 is out, with the remote modules feature!
As per my last post,
I’ve been playing around with this already, and I thought it was worth showing
off my simple example of this feature actually being used.
It’s wrapped up in a way that you can fork my 2 repositories and play with it
yourself, and I encourage you to do so
I’ve been playing around with Terraform a bunch recently, and I’m pretty excited about 0.4.0.
However at the moment, the examples leave quite a lot to be desired. So for my own learning and entertainment, I’ve created a set of example Terraform modules,
and a simple example that you should be able to clone and run.
I’ve been a massive fail at blogging recently, as I’ve been way too busy doing cool stuff :)
However, I have talked at a whole bunch of conferences and tech meetups in the last 12 months,
here’s a quick run-down of where’s I’ve been for posterity:
I’ve been playing around with Docker a load recently, and I’ve developed a nice
little pattern for sharing sockets that I’m gonna pass on. (By sockets, I specifically
mean Unix domain sockets) for the purposes
of this post.
A lot of applications can either use a TCP socket, or a Unix domain socket to communicate – for
example FastCGI and http://www.mysql.com/ both
allow you to use either mode (and in the case of mysql, both).
I have a small home server, and a want to run a bunch of applications that inside containers
(for security and management reasons), and these applications need to speak to a mysql server
(via a unix domain socket – which just appears to be a file on the filesystem.
I also want to run the mysql server inside a container – so the mechanics of getting a socket
shared between them are a little non-trivial.
Lets go through a worked example of how I’ve solved this for the dovecot imap and pop3 server
talking to mysql. This is an especially fun example, as dovecot runs as root (within it’s container)
so if someone hacked into the server through an exploit in dovecot – they can rm -rf anything
I share with that container…
This is the mysql data directory, which will be mounted in the mysql container as /var/lib/mysql
mysql_socket vg0 -wi-ao 4.00m
This is going to get mounted at /socket/mysql inside containers, and will hold the mysql Unix domain socket
vmail vg0 -wi-ao 20.00g
And this is my mail spool for dovecot with all the emails in it.
The immediate problem with this is that if the mysql socket volume is shared between multiple other containers
(as dovecot won’t be the only app using this mysql instance), as dovecot runs as ‘root’, then if it gets hacked,
the hacker can delete the mysql socket, and any other programs trying to connect to mysql through that
socket will be affected.
We can’t allow that – that would defeat the entire point of using containers for application isolation!
The trick is that you only need read access to use a unix domain socket that some other program has created.
Therefore, we can use bind mounts to fix this – by re-binding a readonly
(-o ro) copy of the file system, and giving that to dovecot would stop these issues, easy…
So, our partitions get mounted like this:
/dev/mapper/vg0-vmail on /mnt/volumes/vmail type ext3 (rw,noatime)
/dev/mapper/vg0-mysql on /mnt/volumes/mysql type ext3 (rw,noatime)
/dev/mapper/vg0-mysql_socket on /mnt/volumes/mysql_socket type ext3 (rw,noatime)
/mnt/volumes/mysql_socket on /mnt/volumes_ro/mysql_socket type none (ro,bind)
The last line involves a little trick. When I tried this, to my dismay, it didn’t work.
But as you can see from the Googles,
read only bind mounts are a bit tricky – you have to re-mount them a second time to make them
read-only.
The trick I use here is twofold – one, I only create them with puppet (mostly using the excelent
mounts module (which I only had to patch a little bit),
the associated code looks like this:
1234567891011
mounts {"Mount volume ${name} bind to ro":
ensure=> present,
source=> "/mnt/volumes/${name}",
dest=> "/mnt/volumes_ro/${name}",
type=> 'none',
opts=> 'bind,ro', # Note bind mount ignores ro till you remount, thus the trickery below} ->
exec{"/bin/mount -o remount,ro '/mnt/volumes_ro/${name}'":
onlyif=> "/bin/mount -l | /bin/grep '/mnt/volumes/${name} on /mnt/volumes_ro/${name} type none (rw,bind)'"}
This could be notify rather than just ordering – but I like to be paranoiad and check these are ok
every puppet run..
However, if the machine is freshly rebooted, then the read-only file systems won’t be remounted yet, ergo
the second piece of cunning is an upstart script to make sure things as kosher after a reboot:
1234567891011
# /etc/init/remount-ro-bind-mounts.conf - Dirty hack to remount ro bind mounts properlydescription "Remount RO bind mount filesystems on boot"start on local-filesystems
console output
script
for fs in $(cat /etc/fstab | awk '{ print $2 " " $4 }' | grep 'bind,ro$' | cut -d' ' -f1); do mount | grep rw,bind | awk '{ print $3 }' | grep $fs >/dev/null; if[$? -eq 0 ]; then mount -o remount,ro $fs; fi; doneend script
And there we go – all done(-ish). I just rsync the vmail spool and mysql data from another machine, and start it all up!
The containers look like this:
1234
$ sudo docker ps
ID IMAGE COMMAND CREATED STATUS PORTS
04a9da050ba8 t0m/dovecot:latest /start run 25 minutes ago Up 25 minutes 110->110, 143->143, 993->993, 995->995
20b3a7a65966 t0m/mysql:latest /start run About an hour ago Up About an hour
This seemingly isn’t a well known term, and today I’m irritated by people
thinking I perform ‘magic’ by using grep, so here’s a rant about what I call
the “Star Wars” ™ methodology of problem solving:
To put it quite simply:
Use the Source Luke
Oftentimes grepping through the source code of the thing that’s giving you
trouble (even if it’s written in a language you don’t speak) will turn up
gold.
Given a basic model of OO languages, it’s easily possible to infer what’s
going on (or what may be going on) most of the time given basic observations
and a few print statements, or even a file and line number.
Getting a full backtrace out of mostdynamiclanguages is real easy too, even in extreme cases..
Even if you don’t know for sure, some source diving will at least
help give you more concrete ideas about how the code is built, which
will allow you to frame the problem better when you do peresent it
to someone with the relevant language and/or domain knowledge.
Even if your assumptions were totally wrong – at at least look more
initiative (and tried to go deeper) than your average Joe, so
the expert is more likely to want to help you, as you’re already
demonstrated a provable desire to be taught to fish, rather than
just be thrown them.