Today I have finished Open Sourcing my log monitoring app. It started as a weekend project to find an alternative to real-time logging services and other apps. There are lots of alternatives but many of them did not satisfy my requirements or were a bit expensive to use. I discussed these topics in my previous article about logging.
The source code and documentation of the app can be found on GitHub. There is not much code. All heavy lifting is done by rsyslog and MySQL. I just put together a simple web app with some nice Node.js packages for quering the database and sending mail. How it relates to the system architecture is shown on the following diagram.
At the moment I have this setup running for 8 machines and about 30 apps. Throught the remote rsyslog instances it is monitoring about 200 different log sources (files, scripts, apps) in total. The performance is OK, log insertion and querying are fast (5000 lines/sec is not an issue on a low-end dual-core CPU with 5400 rpm Western Digital Red drives). However, my original plan to purge logs by selective queries turned out to be too slow. This is the reason why the final version of the code only supports table purge by log line count and has no selective purge filters.