[ba-ohs-talk] Spam filtering tools
http://razor.sourceforge.net/ (01)
"Vipul's Razor is a distributed, collaborative, spam detection and
filtering network. Through user contribution, Razor establishes a
distributed and constantly updating catalogue of spam in propagation that
is consulted by email clients to filter out known spam. Detection is done
with statistical and randomized signatures that efficiently spot mutating
spam content. User input is validated through reputation assignments based
on consensus on report and revoke assertions which in turn is used for
computing confidence values associated with individual signatures. " (02)
There is an enormous amount of technology in this project that, I think,
has broader implications and applicability in an OHS environment. For instance: (03)
"The Razor v2 protocol has been completely redesigned. The new protocol is
based on exchange of Structured Information Strings, that are similar to
URIs and can be parsed with URI decoding libraries. v2 protocol supports
Pipelining, which means Razor Agents can keep a connection open with server
to eliminate the latency introduced by TCP 3-way handshake and 4-way
breakdown for every connection. The new protocol semantics allow seamless
introduction of new signature schemes. Razor v2 protocol specification will
be available shortly" (04)
and (05)
"Nilsimsa is a fuzzy signature algorithm based on statistical models of
n-gram occurrence in a piece of text. Nilsimsa disregards small changes
(mutations) in text that are statistically irrelevant. Nilsimsa signatures
can be compared to determine the similarity (between 0 - 100%) in source
texts. Razor v2 includes support for Nilsimsa signatures. " (06)