At Emerging Technology Advisors (ETA) we help a lot of companies convert their IT infrastructure from traditional deployment scenarios to Docker container based deployments. The first topic of discussion is often how to convert their existing, potentially legacy applications to run in a Docker container.
While this is normally straightforward, one of the items that can cause major problems is dockerizing applications that exclusively use the
syslog(3) facility. This is a common best practice for daemon-based applications especially those built in the C/C++ programming language.
From a quick search, most solutions require you to run a sidecar syslog aggregator that will
bind-mount (or for later versions of Docker, using
volumes-from)a volume to extract the logs from out of the main container. This has been the traditional manner for log aggregation-as-a-service components like Loggly, LogEntries, and others. As of Docker version 1.3, the recommended solution is to use
docker-exec and execute a
tail -f /var/log/syslog against the running container.
The problem with these proposed solutions are that they require the container to be operated as a snowflake from the rest of your container operations. Specifically they require special attention to either run or debug which prevents them from being managed as cattle and will undoubtedly lead scenarios of frustration and confusion later down the road. The ideal scenario is to make a syslog-based daemon/application look and act like a normal container without any special efforts or external “support” containers running.
When presented with this problem, our team looked to solve this in a manner that would not create a snowflake inside our container world and also abide by the Principle of Least Astonishment for developers using the container. Said differently, the container, regardless of the application, should just work. In order to accomplish this, our team worked through a variety of known unix tricks and hacks from symbolic link redirection to embedded rsyslog, but none of the solutions fit the expectations we had set. So we dug into the playbook of arcane unix trickery and pulled out the multipurpose relay,
socat(1), AKA the network swiss army knife, in concert with some unbuffering and
sed magic. Since we did not find this documented anywhere else, we decided to publish what we did.
To take a containerized daemon process that uses syslog exclusively and convert it into a unremarkable docker container (not a snowflake), one needs only install
socat with your package manager and add the redirection instruction to either your run script OR your docker entrypoint script. For our use, we have standardized on the
docker-entrypoint.sh model, so our additions are as follows:
This will create a unidirectional socket relay on
/dev/log, the socket location for syslog, and redirect any input to that socket over to
STDOUT. The raw data from
socat does not include any separator, like a newline, because that is left to syslog to handle as it processes the log stream. Due to this, the data stream is piped into a series of unbuffered (thanks
stdbuf) transformers to chunk the data and finally format and present the logs by way of
/dev/fd/1 (standard out).
With these in place, you can now treat a daemonized, syslog based service just as you would any other container based application. At ETA we use this technique regularly to containerize system level and/or legacy applications. This reduces the complexity of a given containerization effort because it reduces the influence that application decisions have upon operational execution. The result is all ephemeral containers can be treated the same, in development and in production, reducing the likelihood of accidental misuse.
If you need assistance with modernizing or containerizing your IT infrastructure, please contact [email protected]