Subject: [Asrg] "Uncaught spam" research project

From: Martijn Grooten <[email-masked]>
Date: Fri, 30 Apr 2010 15:37:18 +0100

I intend to do a little project where I send a lot of spam[1] through a large number of mostly commercial[2] spam-filters (which I'm doing anyway) and then look at differences between spam that's caught by all filters, spam that is misidentified by one filter and spam that is misidentified by more than, say, 25% of the filters. All with the purpose of finding where spam filters can be improved.

Things I want to look at include the location of sender's IP, the character se, the size of the body, the presence of an inline image (or attachment in general), SPF[3] and whether the message is caught when it is resent after an hour/day/week. (The latter to see if it's just a matter of signatures/blacklists not updating fast enough.) Feel free to suggest more things to look at, or make general suggestions for the project. I'm also happy to hear the suggestion not to run (or publish) the research at all. I am aware that this could also give spammers some insight in which techniques are more likely to evade filters.

Thanks.

Martijn.

[1] Spam in the context of this email is spam sent to spam traps. So the real, proper spam, not the perhaps-not-100%-CAN-SPAM-compliant spam.

[2] Several of these make use of open source filters (e.g. SpamAssassin), so it's fair to say that most filters are covered. The setup does exclude techniques such as TCP fingerprinting or greylisting though.

[3] I would love to include DKIM, but I can only distinguish between does have and does not have a DKIM-signature; the redacting of emails to hide the original recipient makes me unable to decide whether a present signature was actually valid.

Virus Bulletin Ltd, The Pentagon, Abingdon, OX14 3YP, England.
Company Reg No: 2388295. VAT Reg No: GB 532 5598 33.

---

From: John Leslie <[email-masked]>
Date: Fri, 30 Apr 2010 12:06:58 -0400

Martijn Grooten <[email-masked]> wrote:
> 
> I intend to do a little project where I send a lot of spam[1] through
> a large number of mostly commercial[2] spam-filters (which I'm doing
> anyway) and then look at differences between spam that's caught by
> all filters, spam that is misidentified by one filter and spam that
> is misidentified by more than, say, 25% of the filters. All with the
> purpose of finding where spam filters can be improved.
> 
> Things I want to look at include
> - the location of sender's IP,
> - the character se,
> - the size of the body,
> - the presence of an inline image (or attachment in general),
> - SPF[3]
> - and whether the message is caught when it is resent after an
>   hour/day/week. (The latter to see if it's just a matter of
>   signatures/blacklists not updating fast enough.)
>
> Feel free to suggest more things to look at,

   I'd definitely record the AS of the sender's IP.

> or make general suggestions for the project. I'm also happy to hear
> the suggestion not to run (or publish) the research at all.

   Oh, definitely run it... The question is how much to obscure when
you publish it.

> I am aware that this could also give spammers some insight in which
> techniques are more likely to evade filters.

   Filters, hopefully, are a moving target; so whatever you publish
will be of limited use a week later.

> [1] Spam in the context of this email is spam sent to spam traps.
> So the real, proper spam, not the perhaps-not-100%-CAN-SPAM-compliant
> spam.

   It will be necessary to at least sample the "interesting" cases,
since spamtraps do get some non-spam...

> [2] Several of these make use of open source filters (e.g.
> SpamAssassin), so it's fair to say that most filters are covered.
> The setup does exclude techniques such as TCP fingerprinting or
> greylisting though.

   That's OK, though it might be interesting to compare those
techniques. BTW are you saying that if a (commercial?) spam-filter
uses those techniques, your setup will exclude them?

> [3] I would love to include DKIM, but I can only distinguish between
> does have and does not have a DKIM-signature; the redacting of
> emails to hide the original recipient makes me unable to decide
> whether a present signature was actually valid.

   I would assume that the interesting datum is whether the DKIM
signature was valid when received, and that the DKIM signature
itself needs to be excised.

--
John Leslie <[email-masked]>

---

From: Martijn Grooten <[email-masked]>
Date: Fri, 30 Apr 2010 17:32:45 +0100

>    I'd definitely record the AS of the sender's IP.

That should be possible, but would that give information that's useful in a statistical way? I want to find data such as "7% of all spam is sent from Liechtenstein, whereas 12% of hard-to-filter spam is sent from there; we ought to focus more on emails coming from Liechtenstein's IP space". I expect the percentages of AS's found to be so low that the differences will hardly ever be significant.

> will be of limited use a week later.

I realise that. I do want to run the project over a longer period of time and I do hope the project to yield some general information on where to focus. To be honest, I'm not sure what to expect and whether I can expect the results to be useful, but that's one of my reasons for running the project.

> since spamtraps do get some non-spam...

Good point and yes, I will do that.

> uses those techniques, your setup will exclude them?

No. Just that these features are turned off.

Thanks

Martijn.

---

From: Aaron Wolfe <[email-masked]>
Date: Fri, 30 Apr 2010 13:05:35 -0400

> I intend to do a little project where I send a lot of spam[1] through a large number of mostly commercial[2] spam-filters (which I'm doing anyway) and then look at differences between spam that's c…

In my experience, you will find more variation in performance between
a properly configured / maintained spam filter and a system left at
defaults and forgotten about than you will find between different
vendors.  Filters will use a variety of tactics to detect spam but
most are common and unless a vendor has implemented them incorrectly
they should perform identically.  For instance RBLs, smtp syntax
checks, dns checks, IP connection characteristics are all going to be
common. There is variation in how the results of these things are
used, but this is often configurable and needs to be tweaked for a
particular type of site for best performance anyway.

Are you planning to compare these systems in their default
configurations?  If so your results may be more an indicator of which
vendor's defaults work best for your system than anything else.

> _______________________________________________
> Asrg mailing list
> [email-masked]
> http://www.irtf.org/mailman/listinfo/asrg
>

---

From: Bill Cole <[email-masked]>
Date: Fri, 30 Apr 2010 14:55:22 -0400

Martijn Grooten wrote, On 4/30/10 10:37 AM:
[...]
>
> [1] Spam in the context of this email is spam sent to spam traps. So the
> real, proper spam, not the perhaps-not-100%-CAN-SPAM-compliant spam.

That skews your sample quite a bit. A significant fraction of the hard cases 
these days are CAN-SPAM compliant campaigns sent by well-meaning originators 
using mostly-whitehat ESP's. The vast majority of spam is nothing like that, 
but the vast majority is also fairly easy to shun for all but the largest 
and cheapest providers. The spam that ends up in a real user's normal Inbox 
(where it is most annoying to them) is unlikely to ever hit a trap address. 
On the other side, traps vary in provenance so there is a lot of variation 
in what they get but as a direct corollary of the fact that you can call an 
address a trap, the mail it is sent will be coming from less careful and 
clueful spammers. That tilt won't be the same for every sort of trap, but it 
will be there for all traps to some degree.

That does not make your research pointless, but it is important to 
understand the skew and the subjective "cost" of different sorts of uncaught 
spam in applying whatever you find.

> [2] Several of these make use of open source filters (e.g. SpamAssassin),
> so it's fair to say that most filters are covered.

Not so much. One real world distinction between vendorware wrapping SA and 
SA deployed consciously is that the former is often bought as an alternative 
to employing skilled staff, while the latter is likely to be a tool that is 
constantly being adjusted and enhanced by skilled staff. Sites vary in what 
spam they get, what non-spam they get, and FP tolerance. Spammers adapt 
somewhat over time to filtering tactics, especially to SA because it is the 
dominant open source filtering tool. A commercial filter built around SA is 
likely to use an older version with its ornate configurability configured in 
a manner so that by default it is safe for any site and exposed to local 
adjustment in only the simplest ways, while a well-managed SA deployment is 
likely to be kept current and to have departures from the distributed config 
defaults that would be intolerable for other sites.

Again, that is an issue that speaks primarily to interpretation and 
application of your results, rather than to the whole plan. If you don't 
have a skilled SA wrangler handy you cannot test the results of a customized 
and tuned SA deployment, and that is one of the strong arguments against 
using straight SA in the real world.

> [3] I would love to include DKIM, but I can only distinguish between does
> have and does not have a DKIM-signature; the redacting of emails to hide
> the original recipient makes me unable to decide whether a present
> signature was actually valid.

Probably not a big loss in itself, as DKIM correlation with spamminess is 
very sensitive to the sort of mailstream one has, and in complex ways.

HOWEVER, this raises a very serious design pitfall. You need to make sure if 
you plan to feed redacted messages to filters that they can be made to 
ignore whatever redaction-spoor is in those messages. The simplest example 
is the one you give: broken signatures. A valid signature may correlate very 
poorly to validity of the mail while a bad signature may correlate quite 
well to invalidity. As a general rule, redaction for the purpose of hiding 
individual identity naturally tends to make all of the redacted messages a 
little bit more like each other in ways that may be obvious or may be 
subtle, and many filters are designed to look for patterns of similarity as 
evidence of spam.

Ultimately I think it is so hard to be sure that you are avoiding 
significant effects from redaction that researching filters with redacted 
inputs is a total waste of time. I don't really understand the point of 
redaction for this anyway, since the addresses are traps. The case for 
recipient redaction may be plausible when spam is being reported in public 
or to untrusted parties, i.e. to protect individual privacy or the effects 
of disclosing a trap address. For filter testing with trap spam, the only 
risk of disclosure would be if a filter works in some collaborative fashion 
akin to DCC or Razor/Pyzor. If you are afraid of that resulting in harmful 
disclosure of your trap addresses, then you have an intractable problem. The 
GIGO principle applies, and any modification of your inputs from what a 
filter would see in the real world makes your inputs garbage.