Unix Tip: Scanning your messages file for warnings

Send in your Unix questions today! | See additional Unix tips and tricks

Too often we sysadmins look through our messages files only when something has gone wrong on our systems and we need some clues that might help us piece together what has happened. Routine scanning or, better yet, intelligent extraction and summarization of important notices and warnings can help alert us to problems when they make their first appearance and sometimes avoid the worst repercussions of a problem on your systems.

In this week's column, we look at a script that scans for warning messages, collects the text and frequency of appearance of these warnings and presents this information in digested form to the user.

I call this script "look4warnings" and run it interactively when I want to see what kind of warnings are being generated on a particular system and through cron with output sent to me via email if I want to be kept abreast of developing problems on systems I log into infrequently.

The output of the look4warnings script looks like this:

# /usr/local/bin/look4warnings types of warnings: 1 WARNINGS & Counts ================= WARNING: processor level 4 interrupt not serviced: 2 Notice that the script tells me how many different types of messages have appeared in the messages file being checked as well as how any times each particular warning error has appeared. The script reads the /var/adm/messages file and then examines each line in the file looking for the word "WARNING" whether in all uppercase as it normally appears, lowercase or some mix of uppercase and lowercase. Using perl, we can elect to match text regardless of whether the text is in upper or lower case by adding a lowercase "i" after the match phrase (e.g., m/WARNING/i;). If we find a match, we want to save the text of the warning message, but only if we have not already done so. To accomplish this, the script uses a hash to keep track of each warning and the number of occurrences we have seen. For example, we might have an element in the array which records that the number of times we have seen the "processor level 4 interrupt not serviced" warning message is 2. When the first warning message is encountered, we add it to the two data structures that we are using to keep track of errors. The assignments would look like this: $warnmsg[0]="processor level 4 interrupt not serviced"; $warnings{processor level 4 interrupt not serviced}=1; In other words, the array that we are using to keep track of the warning messages we have seen so far assigns the text of the message to element 0 while the hash remembers that we have seen that particular warning message once. The next time we encounter the same warning message, the $warnmsg array will not get a new element since it has already recorded the text of this message, but the hash for the message is incremented to reflect that the warning has now appeared twice. The "if ( ! $warnings{$msg} )" statement asks whether the warning has not yet been included in hash. If $warnings{processor level 4 interrupt not serviced} does not exist, it will be recorded in both the $warnmsg array and the hash. Otherwise, the hash will simply be incremented. #!/usr/bin/perl -w # look through messages file for warnings, show summaries $msgs="/var/adm/messages"; open (MSGS,"<$msgs") || die "Cannot open $msgs"; while ( <MSGS> ) { if ( /WARNING/i ) { ($msg)=/WARNING:\s*(.*)/i; if ( ! $warnings{$msg} ) { $warnmsg[++$#warnmsg]=$msg; $warnings{$msg}=1; } else { $warnings{$msg}++; } } } By the time we have processed every line in the messages file, we are ready to display the results. If we have seen any warnings, the size of the $warnmsg array ($#warnings) will be 0 or larger; Perl arrays report a size of -1 when they are empty. If the $warnmsg array is empty, we print "No warning messages found" and exit. Otherwise, we print the number of distinct warnings we have seen followed by a list of the warning messages and the number of times each has been seen. # display numbers of unique warnings seen if ( $#warnmsg >= 0 ) { $num=$#warnmsg + 1; print "\ntypes of warnings: $num\n\n"; } else { print "\nNo warnings messages found\n\n"; exit; } # display count for each warning seen print "WARNINGS & Counts\n"; print "=================\n"; foreach $msg ( @warnmsg ) { print "$msg: "; print "$warnings{$msg}\n"; } print "\n"; While this example may be a bit far-fetched, I decided that the best way to test this script was to include a large variety of warnings in a sample messages file and then verify that the script found all of them and reported the correct number of appearances of each. The output from my test run of look4warnings, therefore, includes a much larger variety of warnings than you are likely to see on a single system: # ./look4warnings types of warnings: 14 WARNINGS & Counts ================= add_spec: No major number for sf: 2 cachefs:invalid cache version: 4 Clock gained 4 days-- CHECK AND RESET THE DATE!: 1 Could not find matching rule in rules.ok: 13 FAN FAILURE check if fans are still spinning: 8 FAN FAILURE still sensed: 8 No network locking on string: contact admin to install server change: 22 processor level 4 interrupt not serviced: 25 /tmp: File system full, swap space limit exceeded: 3 TOD clock not initialized-- CHECK AND RESET THE DATE!: 10 Unable to repair the / filesystem. Run fsck: 1 vxvm:vxio: Illegal vminor encountered: 31 /pci@0,0/pci1014,22@9/pcie11,4030@0/cmdk@1,0 (Disk1):: 52 you are staring too intently at your screen: 1 On this topic Unix Tip: Reassembling Very Large "split" Files Unix Tip: Experimenting with Rsync Unix Tip: Shuffling file systems with rsync Building a Monitoring Infrastructure with Nagios Get practical tips, IT news, how-tos, and the best in tech humor.   Sandra Henry-Stocker has been administering Unix systems for more than 18 years. She describes herself as "USL" (Unix as a second language) but remembers enough English to write books and buy groceries. She currently works for TeleCommunication Systems, a wireless communications company, in Annapolis, Maryland, where no one else necessarily shares any of her opinions. She lives with her second family on a small farm on Maryland's Eastern Shore. Send comments and suggestions to bugfarm@gmail.com.