Real-time logging with Papertrail
This week I finished migration to Papertrail. Papertrail is an online real-time log aggregator and monitoring service. I am mostly using it to monitor error logs of web applications and correlate the errors with request logs of frontend web servers. It also has mail alerts that notify me when a certain pattern (like an error message) appears in a log.
I previously used Logentries which used to offer a very generous low-end tier with lots of features. However, they have now removed the features from my account that I really needed: email alerts and aggregated live logs (multiple correlated realtime logs). These features are now in a much higher tier that costs a lot more money. I host at Linode and at the cost of 99$/month (Pro tier) it would rather make sense to go with Linode Managed (100$/month) which not only includes logging but also 24/7 incident management by Linode's own team. Besides, the Logentires Pro tier data volume retention (60GB over 30 days) does not match my usage volume. The busiest server that I maintain has generated less than 20GB of logs over its entire existence (2-3 years).
Anyway, migration to Papertrail was relatively easy. It uses the remote syslog protocol for gathering logs. They maintain an Open Source daemon daemon that reads local log files and turns them into a stream of syslog events. I'm using the daemon to gather the app logs. The daemon is written in Go which means deploying a single statically linked binary. An init script, systemd, upstart and supervisord service descriptors are also provided to start it as a system or a container service.
Alternatives
I have looked into alternatives, including self-hosted solutions and standalone log watch/monitor utilities. Self-hosted solutions are flexible but take lots of time to configure and you need a separate machine to run them. Remote syslog aggregators have no web-accesible UI and miss alerting features. Utilities like logwatch are mostly meant for logging standard services such as Apache web server, Postfix mail server, SSH daemon, etc. and take some configuration magic to get working for custom apps. Neither they provide remote monitoring and time-based correlation of logs. The current most popular online logging services seem to be Logentries, Papertrail and Loggly. While all of them provide similar features, Papertrail was currently the most cost effective service for my use case.