Summary: log file and mail message filtering programs
Robert E. Van Cleef
vancleef at nas.nasa.gov
Tue Apr 2 02:01:08 AEST 1991
Thanks to those who replied...
To supply context, here was the original question:
--------------------------------------------------------------------------
> From: vancleef at nas.nasa.gov (Robert E. Van Cleef)
> Subject: log file and mail message filtering programs
> Date: Fri, 22 Mar 91 15:12:40 GMT
> Organization: NASA/Ames Research Center
> Keywords: message filter logfiles errors system administration
>
> One of the major problems with the administration of a large number
> of systems is the large volume of information that is generated everyday
> by the systems.
>
> There is a massive amount of information that is available in the system
> log files or system mail messages that the system administrator is forced
> to ignore, or may not even be aware of, because of the large amount of
> information and the enormous amount of noise.
>
> (It is almost as bad a trying to keep up with a USEnet newsgroup:)
>
> Has anyone done any work on developing intelligent filters that can monitor
> the information generated by a couple of hundred workstations, filter out the
> noise, and summarize the results?
>
> Any pointers would be welcome, and I will summarize any results that I receive.
>
> Bob
> --
> Bob Van Cleef vancleef at nas.nasa.gov
> NASA Ames Research Center (415) 604-4366
> ---
> Perception is reality...
--------------------------------------------------------------------------
Here is a summary of the replies. Apparently there is only one tool "watcher"
freely available and one commercial product "XRSA" ...
Look for a new book - "UNIX Tool Building" by Kenneth Ingham.
It includes a description a tool called "watcher", also by Kenneth
Ingham, which was also described in the paper:
> "Keeping Watch Over the Flocks by Night (and Day)"
> by Kenneth Ingham
> Proceedings of the Summer 1987 USENIX Technical Conference and Exhibition,
> Summer 1987, pp. 105-110.
Thanks to:
> From: smfedor at solar.lerc.nasa.gov (Gregory Fedor)
> From: Fuat C. Baran <fuat at cunixf.cc.columbia.edu>
> From: Scott Gasparian <gaspar at inf.ethz.ch>
I was also sent some small script examples (included below)
Thanks to:
> From: Dan Chaney <chaney at ms.uky.edu>
There is also a complany called XRSA, that provides a consulting/
system monitoring servic: (more below)
> From: dick at ccnext.ucsf.edu (Dick Karpinski)
> From: eci386!jmm at zoo.toronto.edu (John Macdonald)
--------------------------------------------------------------------------
Here is the full collection of replies: hope it helps... Bob
------------------------- full text follows ------------------------------
> From smfedor at solar.lerc.nasa.gov Fri Mar 22 09:44:15 1991
> Date: Fri, 22 Mar 91 12:43:56 EST
> From: smfedor at solar.lerc.nasa.gov (Gregory Fedor)
> Message-Id: <9103221743.AA00298 at solar.lerc.nasa.gov>
> To: vancleef%nas.nasa.gov at amelia.nas.nasa.gov
> Subject: Re: log file and mail message filtering programs
> Newsgroups: comp.unix.admin
> In-Reply-To: <1991Mar22.151240.6626 at nas.nasa.gov>
> Organization: NASA/Lewis Research Center, Cleveland
> Status: RO
>
> In article <1991Mar22.151240.6626 at nas.nasa.gov> you write:
> >Has anyone done any work on developing intelligent filters that can
> >monitor the information generated by a couple of hundred workstations,
> >filter out the noise, and summarize the results?
> >
> >Any pointers would be welcome, and I will summarize any results that I receive.
>
> Bob,
>
> I am currently reading a book title _UNIX Tool Building_ by Kenneth Ingham.
> In it he is walking the reader through the building of a utility called
> "watcher" that he helped create at the University of New Mexico for monitoring
> the status of many system. From what I've read so far, it sounds like this
> is what you are looking for.
>
> Also, from what I gather it's available in comp.source.unix on uunet.uu.net.
> I haven't had a chance yet to go check this out (I'm only on page 61 :). I
> plan on retrieving it though and trying it out here at Lewis as well as taking
> some concepts for a project I'm working on.
>
> I hope this helps. If you need any further information, drop me a line.
> I look forward to hearing what other answers you get.
>
> --
> ===============================================================================
> Gregory A. Fedor | Far from day, far from light \
> Sverdrup Technology Inc. | Out of time, out of sight \
> NASA Lewis Research Center | To a world, young and free \\-^-/___
> Cleveland, Ohio 44135-3191 | Weep no more, follow me |===[o]/ #o
> (216) 433-8468 | /VVV
> smfedor at lerc01.lerc.nasa.gov | Forever...Forever...Forever /
> (128.156.10.14) | Voyagers 1 & 2
> ===============================================================================
>
> From fuat at cunixf.cc.columbia.edu Fri Mar 22 17:05:08 1991
> Received: by cunixf.cc.columbia.edu (5.59/FCB)
> id AA10339; Fri, 22 Mar 91 20:04:57 EST
> Date: Fri, 22 Mar 91 20:04:57 EST
> From: Fuat C. Baran <fuat at cunixf.cc.columbia.edu>
> Message-Id: <9103230104.AA10339 at cunixf.cc.columbia.edu>
> To: vancleef at nas.nasa.gov
> Cc: fuat at cunixf.cc.columbia.edu
> Subject: Re: log file and mail message filtering programs
> Newsgroups: comp.unix.admin
> In-Reply-To: <1991Mar22.151240.6626 at nas.nasa.gov>
> Organization: Columbia University Center for Computing Activities
> Status: RO
>
> In article <1991Mar22.151240.6626 at nas.nasa.gov> you write:
> >Has anyone done any work on developing intelligent filters that can
> >monitor the information generated by a couple of hundred workstations,
> >filter out the noise, and summarize the results?
>
> Take a look at:
>
> "Keeping Watch Over the Flocks by Night (and Day)"
> by Kenneth Ingham
> Proceedings of the Summer 1987 USENIX Technical Conference and Exhibition,
> Summer 1987, pp. 105-110.
>
> Kenneth Ingham has also written a book based on this paper (I'm not
> sure of the name but it was something like "UNIX Tool Building." I
> have the book at home, and can get you the details if you want).
> --Fuat
> --
> Internet: fuat at columbia.edu U.S. MAIL: Columbia University
> BITNET: fuat at cunixc Center for Computing Activities
> UUCP: ...!rutgers!columbia!cunixf!fuat 712 Watson Labs, 612 W115th St.
> Phone: (212) 854-5128 Fax: (212) 662-6442 New York, NY 10025
> ---------------------------------------------------------------
> From @s.ms.uky.edu:chaney at ms.uky.edu Fri Mar 22 20:34:38 1991
> From: Dan Chaney <chaney at ms.uky.edu>
> Date: Fri, 22 Mar 1991 23:33:48 EST
> X-Mailer: Mail User's Shell (7.2.0 10/31/90)
> To: vancleef at nas.nasa.gov
> Subject: Re: log file and mail message filtering programs
> Message-Id: <9103222333.aa01669 at s.s.ms.uky.edu>
> Status: RO
>
> Newsgroups: comp.unix.admin
> References: <1991Mar22.151240.6626 at nas.nasa.gov>
>
> A lot of it depends on what sorts of things you want to keep up with, of
> course. I keep track of mail daemons and queues through scripts that
> know what 'normal' is and send mail when things don't quite match. That
> is helpful to maintain 'running' programs. Checking for the presence
> of TCP daemons is fairly simple if you assume the existence is proof
> enough of a daemon's state.
> echo "quit" | telnet mozart.ms.uky.edu 25
> if [ $? != 0 ]; then
> echo "Problem with the daemon"
> fi
> That tells me if the smtp daemon is running. Along with sendmail -bp's on
> other machines, I can usually catch a clogged mailer within an hour or
> so (these scripts run every 4 hours, but that is just because I like
> diligence in a major way) We also run MMDF on two machines and that
> makes for lots of log files. Clever greps and diffs on 'ok' log files
> brings my over all system mail down to a reasonable level. One helpful
> trick I use is running scripts that write a lot of info to a specific
> log - and overwrite the old data. This allows the full data to be at
> least accessible, without getting in your way under 'normal' circumstances.
>
> I guess the theme is to train scripts what is normal or just do diffs
> on a 'normal' output. I can provide you the scripts if you want. If
> you want some ugly scripts, I'll show you the archive-maintaining scripts
> that just tell me how things are and send nagging notes to all my archivers.
> A truly obnoxious piece of scripting :-)
>
> -dan
> ------------------------------------------------
> From gaspar at inf.ethz.ch Sat Mar 23 03:44:37 1991
> From: Scott Gasparian <gaspar at inf.ethz.ch>
> Message-Id: <9103231146.AA06449 at orion.inf.ethz.ch>
> Cc: gaspar at orville.nas.nasa.gov
> Subject: Re: log file and mail message filtering programs
> Status: RO
>
> Have you heard of the program called "watcher" ? It takes input
> from cron outputs, syslogs, msgs, etc, and compares them. If
> something changes past a certain parameter (say load goes over
> 20 or disk free goes over 90%), it mails a msg to set people. I
> will try and remmeber where we got ours and send you more info.
> I think it was U of New Mexico or something like that.
>
> very usefull little utility. Might be in *.sources.something.
>
> --gaspo.
>
> /----------------------------------------------------------------------------\
> | Scott "gaspo" Gasparian -- System Administrator | _>________ _<________ |
> | Dept. Informatik, Eidg. Techn. Hochschule, Zurich |/[][][][][]\/[][][][][]\|
> | ETH-Zentrum, CH-8092 Zurich. T# 01-01-254-7205 |`oo------oo'`oo------oo'|
> | gaspar at inf.ethz.ch | "Good friends we've had, or good friends we've lost, |
> | ..!ethz-inf!gaspar | along the way.In this proud land,you can't forget your|
> | gaspo at scri.fsu.edu | past,so dry your tears I say. No woman, No cry." -BMW |
> \----------------------------------------------------------------------------/
>
> From dick at ccnext.ucsf.EDU Mon Mar 25 17:16:07 1991
> From: dick at ccnext.ucsf.edu (Dick Karpinski)
> Message-Id: <9103260115.AA17508@ ccnext.ucsf.edu >
> To: vancleef at nas.nasa.gov
> Subject: XRSA does just that
> Status: RO
>
> There is a commercial product from a software house in Canada which
> does just that sort of thing. It's called eXpert Remote System
> Administrator and uses possibly some AIish software in the central
> host to reduce the data coming in to just the part that's most
> interesting to the human attendants. They seem to want $20k/yr to
> get into the game, so I'm interested in cheap clones. Many of us
> human administrators ought to be willing to collaborate on a public
> access package like that. PERL pops to mind as a useful tool for
> many of these tasks. I have lotsa stuff from the xrsa folks if
> that would interest you further. I'd like to pursue this matter
> to the point of having some tools and a continuing sysadmin mailing
> list for enhancements etc....
>
> Dick
>
> Dick Karpinski Minicomputer Manager, UCSF Information Technology Services
> Domain: dick at cca.ucsf.edu FAX: (415) 476-9537 (415) 476-4529 (11-7)
> BITNET: dick at ucsfcca or dick at ucsfvm (415) 658-6803 (Home)
> USPS: U-76 UCSF, San Francisco, CA 94143-0704 (415) 658-3797 (ans)
> --------------------------------------------------------------------------
> From eci386!jmm at zoo.toronto.edu Tue Mar 26 09:16:12 1991
> From: eci386!jmm at zoo.toronto.edu (John Macdonald)
> Date: Tue, 26 Mar 1991 11:53:38 EST
> Newsgroups: comp.unix.admin
> In-Reply-To: <1991Mar22.151240.6626 at nas.nasa.gov>
> Organization: Elegant Communications Inc.
> X-Mailer: Mail User's Shell (7.1.2 7/11/90)
> To: vancleef at nas.nasa.gov
> Subject: Re: log file and mail message filtering programs
> Message-Id: <9103261153.AA12599 at eci386.UUCP>
> Status: RO
>
> In article <1991Mar22.151240.6626 at nas.nasa.gov> you write:
> |One of the major problems with the administration of a large number
> |of systems is the large volume of information that is generated
> |everyday by the systems.
> |
> |There is a massive amount of information that is available in the system
> |log files or system mail messages that the system administrator is forced
> |to ignore, or may not even be aware of, because of the large amount of
> |information and the enormous amount of noise.
> |
> |(It is almost as bad a trying to keep up with a USEnet newsgroup:)
> |
> |Has anyone done any work on developing intelligent filters that can monitor
> |the information generated by a couple of hundred workstations, filter out
> |the noise, and summarize the results?
> |
> |Any pointers would be welcome, and I will summarize any results that I receive.
>
> Well, we have had some previous email discussions about
> XRSA - it can do much of this, and can be extended by us
> to add the rest as a consulting project to any degree of
> detail that you are willing to have us address.
>
> XRSA does a great deal of reduction and analysis of many
> log files already. The reports that it generates are of
> two major categories - daily and urgent. Daily reports
> show interesting details about the systems. Urgent
> reports only show indications of upcoming and current
> problems. We typically expect that sys admins will normally
> read urgents, and will read dailies only on a casual basis
> or to obtain detailed background info for an unusually puzzling
> urgent problem.
>
> There is a (very brief) summary provided for a group of
> systems (currently it just states whether logs were
> received, and whether there was an urgent condition, for
> each monitored system in the group) which could be easily
> extended to summarize any particular condition that you
> wished to oversee.
>
> Feel free to request additional info from me.
>
> --
> Cure the common code... | John Macdonald
> ...Ban Basic - Christine Linge | jmm at eci386
> ---------------------------------------------------------------------------
> From eci386!jmm at zoo.toronto.edu Wed Mar 27 06:05:13 1991
> From: eci386!jmm at zoo.toronto.edu (John Macdonald)
> Date: Wed, 27 Mar 1991 08:47:24 EST
> In-Reply-To: Message dated Tue Mar 26 10:19 from vancleef at garg.nas.nasa.gov
(Robert E. Van Cleef) Re: "Re: log file and mail message filtering programs"
> X-Mailer: Mail User's Shell (7.1.2 7/11/90)
> To: vancleef at garg.nas.nasa.gov (Robert E. Van Cleef)
> Subject: Re: log file and mail message filtering programs
> Message-Id: <9103270847.AA26712 at eci386.UUCP>
> Status: RO
>
> /===== Re: log file and mail message filtering programs =====
> || Quoting Robert E. Van Cleef, message dated Mar 26, 10:19
> |+-----
> || John;
> ||
> || Unfortunately, when I read the stuff you sent previously I interpreted
> || it as a consulting service setup. I will see if I can dig out the oldd
> || mail messages and re-read them...
> ||
> || Bob
> \=========================
>
> Hmm, our usual (almost constant) problem is that people
> try and treat XRSA as strictly a product. In fact, it
> is closer to being a consulting service than a product,
> but a major portion of the consulting activity is carried
> out automatically by software.
>
> Essentially, we provide consulting and licensed software
> to a service provider, who can then use this to provide
> sys admin service as a product to their customers. The
> service provider can be either an separate company that
> is providing service as a marketed product to its customers
> (currently we have marketing agreements of this sort with
> IBM and Bull and others of their ilk are close to signing),
> or it can be a central support department within a large
> organization that provides service to the rest of the
> organization.
>
> --
> Cure the common code... | John Macdonald
> ...Ban Basic - Christine Linge | jmm at eci386
>
--------------------- end of forwarded material -------------------
Bob Van Cleef - vancleef at nas.nasa.gov
RNS Distributed Systems
NASA Ames Research Center (415) 604-4366
Mail Stop 258-6 FTS 464-4366
Moffet Field, CA 94035-1000 FAX (415) 604-4377
__
"If you're not a liberal at 20, you have no heart, and
if you're not a conservative at 40, you have no head."
Winston Churchill
More information about the Comp.unix.admin
mailing list