Oct 19 2008

Network / System Monitoring Smorgasbord

Published by at 10:51 am under Technology   

At one of my firms (a Software as a Service provider), we have a Zabbix installation in place to monitor our piles of mostly Linux servers. Recently we look a closer look at it and and found ample opportunities to monitor more aspects, of more machines and device, more thoroughly. The prospect of increased investment in monitoring led me to look around at the various tools available.

The striking thing about network monitoring tools is that there are so many from which to choose. Wikipedia offers a good list, and the comments on a Rich Lafferty blog post include a short introduction from several of the players. (Update – Jane Curry offers a long and detailed analysis of network / system monitoring and some of these tools (PDF).)

For OS level monitoring (CPU load, disk wait time, # of processes waiting for disk, etc.), Linux exposes extensive information with “top”, “vmstat”, “iostat”, etc. I was disappointed to not find any of these tools conveniently presenting / aggregating / graphing the data therein. From my short look, some of the tools offer small subsets of that data; for details, they offer the ability for me to go in and figure out myself what data I want in and how to get it. Thanks.

Network monitoring is a strange marketplace; many of the players have a very similar open source business model, something close to this:

  • core app is open source
  • low tier commercial offering with just a few closed source addons, and support
  • high tier commercial offering with more closed source addons, and more support

I wonder if any of them are making any money.

Some of these tools are agent-based, others are agent-less. I have not worked with network monitoring in enough depth to offer an informed opinion on which design is better; however, I have worked with network equipment enough to know that it’s silly not to leverage SNMP.
I spent yesterday looking around at some of the products on the Wikipedia list, in varying levels of depth. Here I offer first impressions and comments; please don’t expect this to be comprehensive, nor in any particular order.

Zabbix

Our old installation is Zabbix 1.4; I test-drove Zabbix 1.6 (advertised on the Zabbix site as “New look, New touch, New features”. The look seemed very similar to 1.4, but the new feature list is nice.

We most run Ubuntu 8.04, which offers a package for Zabbix 1.4. Happily, 8.04 packages for Zabbix 1.6 are available at http://oss.travelping.com/trac.

The Zabbix agent is delightfully small and lightweight, easily installing with a Ubuntu package. In its one configuration file, you can tell it how to retrieve additional kinds of data. It also offers a “sender”, a very small executable that transmits a piece of application-provided data to your Zabbix server.

I am reasonably happy with Zabbix’s capabilities, but I have the GUI design to be pretty weak, with lots of clicking to get through each bit of configuration. I built far better GUIs in the mid-90s with far inferior tools to what we have today.  Don’t take this as an attack on Zabbix in particular though; I have the same complaint about most of the other tools here.

We run PostgreSQL; Zabbix doesn’t offer any PG monitoring in the box, but I was able to follow the tips at http://www.zabbix.com/wiki/doku.php?id=howto:postgresql and get it running. This monitoring described there is quite high-level and unimpressive, though.

Hyperic

I was favorably impressed by the Hyperic server installation, which got two very important things right:

  1. It included its own PostgreSQL 8.2, in its own directory, which it used in a way that did not interfere with my existing PG on the machine.
  2. It needed a setting changed (shmmax), which can only be adjusted by root. Most companies faced with this need would simply insist the installer run as root. Hyperic instead emitted a short script file to make the change, and asked me to run that script as root. This greatly increased my inclination to trust Hyperic.

Compared to Zabbix, the Hyperic agent is very large: a 50 MB tar file, which expands out to 100 MB and includes a JRE. Hyperic’s web site says “The agent’s implementation is designed to have a compact memory and CPU utilization footprint”, a description so silly that it undoes the trust built up above. It would be more honest and useful of them to describe their agent as very featureful and therefore relatively large, while providing some statistics to (hopefully) show that even its largish footprint is not significant on most modern servers.

Setting all that aside, I found Hyperic effective out-of-the-box, with useful auto-discovery of services (such as specific disk volumes and software packages) worth monitoring, it is far ahead of Zabbix in this regard.

For PostgreSQL, Hyperic shows limited data. It offers table and index level data for PG up through 8.3, though I was unable to get this to work, and had to rely on the documentation instead for evaluation. This is more impressive at first glance than what Zabbix offers, but is still nowhere near sufficiently good for a substantial production database system.

Ganglia

Unlike the other tools here, Ganglia comes from the world of high-performance cluster computing. It is nonetheless apparently quite suitable nowadays for typical pile of servers. Ganglia aims to efficiently gather extensive, high-rate data from many PCs, using efficient on-the-wire data representation (XDR) and networking (UDP, including multicast). While the other tools typically gather data at increments of once per minute, per 5 minutes, per 10 minutes, Ganglia is comfortable gathering many data points, for many servers, every second.

The Ganglia packages available in Ubuntu 8.04 are quite obsolete, but there are useful instructions here to help with a manual install.

Nagios

I used Nagios briefly a long time ago, but I wasn’t involved in the configuration. As I read about all these tools, I see many comments about the complexity of configuring Nagios, and I get the general impression that it is drifting in to history. However, I also get the impression that its community is vast, with Nagios-compatible data gathering tools for any imaginable purpose.

Others

Zenoss

Groundwork

Munin

Cacti

How Many Monitoring Systems Does One Company Need?

It is tempting to use more than one monitoring system, to quickly get the logical union of their good features. I don’t recommend this, though; it takes a lot of work and discipline to set up and operate a monitoring system well, and dividing your energy across more than one system will likely lead to poor use of all of them.

On the contrary, there is enormous benefit to integrated, comprehensive monitoring, so much so that it makes sense to me to replace application-specific monitors with data feeds in to an integrated system. For example, in our project we might discard some code that populates RRD files with history information and published graphs, and instead feed this data in to a central monitoring system, using its off-the-shelf features for storage and graphing.

A flip side of the above is that as far as I can tell, none of these systems offers detailed DBA-grade database performance monitoring. For our PostgreSQL systems, something like pgFouine is worth a look.

Conclusion

I plan to keep looking and learning, especially about Zenoss and Ganglia. For the moment though, our existing Zabbix, upgraded to the current version, seems like a reasonable choice.

Comments are welcome, in particular from anyone who can offer comparative information based on substantial experience with more than one of these tools.

If you found this post useful, please link to it from your web site, mention it online, or mention it to a colleague.

4 responses so far

4 Responses to “Network / System Monitoring Smorgasbord”

  1. Karl Katzke says:

    I had an impossible time getting Zenoss up and running properly. It’s configuration is halfway to impossible — and I’m coming from the Nagios/Cacti world here.

    I definitely want to see further reviews of ganglia and zabbix, though.

  2. Jonas B. says:

    Nagios graphs the data you want with the nagiosgraph plugin, You use nrpe to gather information form Nagios plugins remotely. It’s really very easy to set up, but you need to familiarize yourself with its architecture before you dive in. A host has services and contacts, a service has contacts, etc. When it comes to monitoring you really want mature products. Give it a chance, it’s well worth it!

  3. Matt Ray says:

    Zenoss is remote agentless monitoring, and we’re making a push into more OS monitoring via SSH in our next release: http://forums.zenoss.com/viewtopic.php?t=8137

    There is a community ZenPack for PostgreSQL, (http://www.zenoss.com/community/projects/zenpacks/postgresql) and I’m working on opening up Community ZenPack efforts to more collaboration very soon. I’d be more than happy to get a more robust PostgreSQL ZenPack worked on.

    Karl: Sorry to hear you had issues with Zenoss’ installation, was this recent? We’ve done a lot of installer work and documentation around this recently. Feel free to email me for followup.

    Thanks,
    Matt Ray
    Zenoss Community Manager
    mray@zenoss.com

  4. po says:

    I’ve eval’d all of the aforementioned tools and find some are more suited to their intended roles: trending and others monitoring.
    Sure, the lines have been blurred (nagios plugins for ganglia, etc), but personally I think they better left as two distinct roles (until the next one comes along…).

    Nagios has addons to chart performance data, but it’s first and foremost a monitoring & alerting system. It can scale to large numbers of clients by the use of nrpe or even nsca to push results (and even distributed nagios instances, etc). I prefer to push the result back to avoid holding a client up , rather than polling via snmp, etc. Alerting is flexible. I wish it was simpler to setup and use, but saying it’s the least worst of a bad bunch does it a disservice.

    Ganglia, which I also use is ideal for trending. Basically you can chart anything provided you can supply a value to the gmond program (pick your language). It’s great for seeing whether MySQL setting change really did the trick than the one two weeks ago and whether disk i/o dropped. This is very useful for capturing business/app metrics in addition to the typical lamp stack metrics. If deployed with multicast, then you have some resilience for your data/gui (provided you have more than one host running gmetad – i.e more than one host storing the rrds). I find it useful for capacity planning.

    It’s also pretty easy if you’re not using snmp to get custom data into Nagios / Ganglia – I’m sure the other have similar provisions, but it was trickier imo.

    Others:
    I’ve used hyperic (ent demo), zenoss, zabbix, cacti, but I found them all to be restrictive. I settled on a Nagios / Ganglia combination
    Hyperic – (I tested 3.0, but is now v4 I believe) was feature limited in the demo, but I found the agents bulky.
    Zenoss – install was awkward, but the interface was tricky to use.
    Cacti – only found it useful for small installs. Painful if you need to do repetitive tasks (templates in cacti can be tricky). Used it with cactid, otherwise polling could take a while.
    Zabbix – disliked for similar reasons to cacti, largely related to admin/ config tasks.
    I also tried OpenNMS – didn’t eval in as much depth (suffered from memory issues even after tuning the server)

    Another one which may be of interest: http://www.collectd.org – it also support multicast, and is purely intended to be a stats collector (in so far that you choose your own tool to display the resulting rrds), but very small footprint. (A collectd > ganglia plugin would be nice!)