Simple Rotating Log Capture
This post concerns simple text-dump logs, not “data logs” that are sent to, say, Kafka for structured treatment.
I agree with the 12-factor app that programs should simply print logs to the standard output, and should not worry about what file to save the logs to, let alone file size capping and rotation, etc. Capture, storage, and consumption of the log are concerns of the execution environment.
Suppose our program
myapp logs to
stderr periodically, simply do
myapp 2>&1 | multilog s1000000 n10 /path-to-log/myapp
This will save logs in the directory
The currently active log file is named
Each log file is size-capped at 1000000 bytes (via the argument
s1000000), i.e. 1 MB.
Older log files are named after the timestamp of the start (or maybe end?)
of the content of the file.
At most 10 (most recent) log files, including
current, are kept in the directory,
and older ones are discarded; this is controlled by the argument
If we want to observe the terminal printout in the meantime, we can do
myapp 2>&1 | tee >(multilog s1000000 n10 /path-to-log/myapp)
There are a number of ways to customize the behavior of
For example, it can selectively capture lines by pattern matching.
It can also insert a timestamp at the beginning of each line if you so wish.
Some useful resources:
Several alternatives appear to be similar and equally adequate, including
I chose the
daemontools because this is a package (called
daemontools) that is
apt-get installable on
If your program is in Python, I had some
tips on Python logging here
that will work well with such
If you want to observe the terminal printout as well as capture it in files, you may be tempted to use
python myapp.py 2>&1 | tee >(multilog s1000000 n10 /path-to-log/myapp)
To your surprise, you won’t see printouts. Instead, you need to use
python -u myapp.py 2>&1 | tee >(multilog s1000000 n10 /path-to-log/myapp)
This has to do with buffering.