Documentation Home
Help! Errors
Help! False Positives
Help! Spam Leakage
Installation Guides
Features
Procedures
SNF Community
Software
Technology
Tools
Direct Support
Glossary
Q&A

SNFServer

Log Files

Scan / Activity Logs

SNFMulti scan / activity logs come in two flavors. The newer XML based log format offers much more information in an efficient structured format. The older "classic" log format is also offered for backward compatibility.

XML Logs (V3+)

The XML logs produced by the SNFMulti engine are designed to be as efficient as possible, to make good use of XML, and to be easily readable in a plain old text editor -- with a little practice.

Using the logs as XML data

Due to the nature of log files (a string of continuously appended entries), you will need to do a tiny bit of work if you wish to convert SNF's XML logs into "proper" XML for import into databases and other XML systems. Simply add the top level element and document descriptor tags of your choice, close the top element at the bottom, and you should be ready to go. What the SNFMulti engine provides for you are the "guts" of an XML document for a given logging period. By themselves SNF's XML logs are not well formed XML in the strictest sense... but they are not far from it and the modifications required to create well formed XML from them are trivial.

Reading the XML logs in a text editor

The XML logs are formatted so that they will be easy to read in an ordinary text editor with just a little practice. This makes it practical for system administrators to quickly open a log file and find what they are looking for -- or even to tail the log and watch the activity go by (if you can read fast!). Consider these entries (a scan error followed by a normal scan entry):

<s u='20070508012348' m='/spool/msg0123456789.msg' code='69' error='ERROR_MSG_FILE'/>
<s u='20070508012349' m='/spool/msg1123456789.msg' s='48' r='1234567' >
  <m s='48' r='1234567' i='2394' e='2409' f='m'/>
  <m s='48' r='2234567' i='2501' e='2515' f='m'/>
  <m s='48' r='1234567' i='2394' e='2409' f='f'/>
  <p s='10' t='8' l='3294' d='84'/>
  <g o='1' i='101.201.31.04'  t='u'  c='0.12345' p='0.3342983'  r='Caution'/>
</s>

The first scan of message file /spool/msg0123456789.msg produced a message file error.

The second scan occurred at 2007 05 08, 12:34:49 UTC, produced a scan result code of 48 matching rule ID 1234567. Performance and GBUdb data about the scan were also provided showing that it took 10 ms to setup the scan (open, and read the message file) and 8 ms to perform the scan operation. The scan covered 3294 bytes and used a maximum of 84 evaulators. According to GBUdb the source ip for this message (101.201.31.04) was found in the second Received header (ordinal 1). The flag on the GBUdb record for this IP is "Ugly" (t='u'), the confidence figure is 0.12345 and the spam probability figure is 0.3342983 placing this source in the "Caution" range.

Below are descriptions of what each element and attribute mean:

<s/> - Scan Log Entry (always on if modes is not none)
s:u = UTC timestamp,
s:m = Message ID / file,
s:o = Overhead time in ms,
s:t = Scan time in ms,
s:s = Scan result code,
s:l = Scan length,
s:d = Scan Depth,
s:error = Error Description (empty if no error)

<s><m/></s> - Scan Match Entry (controlled by matches attribute)
m:s = Match Symbol
m:r = Match Rule ID
m:i = Match Index
m:e = Match Endex
m:f = Match flag

<s><p/></s> - Scan Performance Monitoring (performance='yes')
p:s = Setup time in milliseconds
p:t = Scan time in milliseconds
p:l = Scan length in bytes
p:d = Scan depth (peak evaluator count)

<s><g/></s> - GBUdb Activity For This Scan (gbudb='yes')
g:o = Ordinal of the source IP
g:i = IP address identified as source
g:t = IP Record type (g)ood, (b)ad, (u)gly
g:p = Spam probability
g:c = Confidence
g:r = Range: Unknown, White, Normal, New, Caution, Black, Truncate

<i/> - Information Message.
i:u = UTC Timestamp.
i:context = Where this message was sent from.
i:code = Numerical code for this message.
i:text = Text version of the message.

<e/> - Error Message.
e:u = UTC Timestamp.
e:context = Where this error occurred.
e:code = Numerical code for this error.
e:text = Text version of the error.

Systems using the SNFMulti engine can use the GBUdb directly to test IPs. When they do so they have the option to report that operation and the result of that test through the XML scan / activity log. That reporting format follows these conventions:

<t/> - IP test log.

t:u = UTC Timestamp.
t:ip = IP under test.
t:t = GBUdb record type.
t:g = GBUdb good event count.
t:b = GBUdb bad event count.
t:c = GBUdb confidence figure.
t:p = GBUdb probability figure.
t:r = GBUdb range.
t:a = Action taken.

Classic Logs (V1 - V2)

The SNFMulti engine can produce SNF version 2 log files for backward compatibility with existing log processing utilities. In fact, the engine can produce both types of log files simultaniously to provide an easier migration path to the new XML based log format.

A Classic (SNF 2) log file contains important statistics about the performance of the sniffer utility and its rulebase. Each line records a single match or clean response from the pattern matching engine. The fields on each line are separated by a tab. The fields meanings are as follows:

1 - License ID.

2 - Timestamp YYYYMMDDhhmmss (UTC).

3 - Scanned message file.

4 - Setup time in milliseconds (time required to load the rule set).

5 - Scan time in milliseconds (time required to load and scan the message).

6 - Clean / Match / White / Final result or error message.

7 - Matching Rule ID.

8 - Reported Group / Symbol ID.

9 - Index of the pattern match (approximate start position).

10 - Endex of the pattern match (approximate end position).

11 - Depth (Maximum number of evaluators alive during the scan).

Below is a segment of a typical log file for SNF 2.

snf2beta 20021204081147 Db8c1112.SMD 581 140 Match 13392 63 4212 4238  36
snf2beta 20021204081147 Db8c1112.SMD 581 140 Match 13392 63 4483 4509  36
snf2beta 20021204081147 Db8c1112.SMD 581 140 Match 13392 63 4930 4956  36
snf2beta 20021204081147 Db8c1112.SMD 581 140 Match 16995 63 5029 5044  36
snf2beta 20021204081147 Db8c1112.SMD 581 140 Match 13392 63 5287 5313  36
snf2beta 20021204081147 Db8c1112.SMD 581 140 Match 16995 63 5386 5401  36
snf2beta 20021204081147 Db8c1112.SMD 581 140 Match 15144 63 7458 7481  36
snf2beta 20021204081147 Db8c1112.SMD 581 140 Match 15144 63 7557 7580  36
snf2beta 20021204081147 Db8c1112.SMD 581 140 Final 13392 63 960  986   36
snf2beta 20021204081149 Db8c0118.SMD 250 871 Clean 0     0  0    31986 74
snf2beta 20021204081200 Db8cf02a.SMD 291 70  Match 17054 63 3417 3443  38
snf2beta 20021204081200 Db8cf02a.SMD 291 70  Match 17054 63 3842 3868  38
snf2beta 20021204081200 Db8cf02a.SMD 291 70  Final 17054 63 3417 3443  38
snf2beta 20021204081202 Db8cf112.SMD 251 40  Clean 0     0  0    3145  28
snf2beta 20021204081222 Db8e402a.SMD 240 110 Match 16688 63 2516 2527  39
snf2beta 20021204081222 Db8e402a.SMD 240 110 Match 16688 63 2516 2615  39
snf2beta 20021204081222 Db8e402a.SMD 240 110 Match 16688 63 2516 3998  39
snf2beta 20021204081222 Db8e402a.SMD 240 110 Match 16688 63 2516 4086  39
snf2beta 20021204081222 Db8e402a.SMD 240 110 Final 16688 63 2516 2527  39

Classic log files produced by SNF 3 and above may also include other activity entries and errors. Those entries are formatted in such a way that they maintain compatibility with existing SNF 2 logging utilities by re-using the message name field to indicate the context of an error or information message and the Clean / Match / Final field to indicate the message itself. This is similar to the way error messages would be produced in the original SNF V2 logs.

An information example might be:

snf2beta 20070521012345 -RELOADING- 0 0 Success 0 0 0 0 0

In this case, the context of the informational message was --Reloading-- indicating that the rulebase file and / or configuration was being reloaded, and the resulting message was Success.

An error example might be:

snf2beta 20070521012345 -RELOADING- 0 0 ERROR_RULE_FILE 0 67 0 0 0

Here again the context is --Reloading-- but the result indicates that there was a problem reading the rulebase file ( ERROR_RULE_FILE ). Note also that the error code associated with this condition is also reported as 67.

Please email support@armresearch.com with any questions.