Subject: [Asrg] Re: whitelisting

From: [email-masked]
Date: Wed, 27 Apr 2005 00:22:01 +0000 (GMT)

>> The point of a whitelist (IMHO) is to toss out certain types of 
> incoming mail
> that you aren't willing to trust or accept from people you don't 
> recognize.

> I think we're getting into a bit of a semantical tiff here. 

No, not really.

> This is dependent on whether or not your site is practicing a default deny or 
default accept policy with regards to mail. 

No.

The default behavior (for unlisted senders) under my approach would depend upon 
the characteristics of the E-mail message.

A plain text E-mail, with no attachments, no HTML, and less than some specified 
size (probably 25K or 50K or something) would probably be set to default to 
"accept".  This would provide for "ordinary" unsolicited E-mails (say for a 
customer relations department of a consumer products company) to be delivered.  

People without an established contact history who send "large" E-mails, or 
attachments, or HTML, or scripting, etc etc, would find their mail either 
blocked or at least rerouted to quarantine for exceptional handling.

> Many sites would rather trust unknown senders and reject known poor senders 
rather than increase false positive rates and associated support costs.

That approach has led to the widespread conclusion that whitelisting approaches 
are not tolerable... especially in situations where users might ANTICIPATE 
receiving "unsolicited" E-mails (e.g. hearing from someone you gave a business 
card to, once upon a time).  Now, on my approach you could theoretically set it 
as you're suggesting (if you REALLY want to) by setting the defaults 
appropriately.

But my feeling is that someone you don't know who sends you an unsolicited 
E-mail containing encrypting Javascript or a 160K-byte PIF executable file or a 
125Kbyte encrypted ZIP file (or even just an 80K .COM file attachment) is 
probably someone that you didn't really want to receive that mail from anyhow.  
:-)

>> Ideally, such a whitelist is VERY finely-grained, and with a 
> "relatively safe" default behavior.

> Finely grained anything leads to scalability and complexity issues. 

"Scalability" is not a serious issue if the screening is (at least initially) 
conducted at the recipient end, where there are as a rule plenty of resources 
available.

Complexity issues depend entirely on how the software is implemented.  I claim 
that it's possible to implement the software such that it's easy to manage these 
sorts of finely-grained permissions list (for example, by offering the user the 
option to "allow this sender to send this type of E-mail message in the future" 
without them having to worry about what specific permissions are involved in 
achieving that).

> That is not to say such solutions cannot work, only that the trade off 
between complexity and effectiveness often yields diminishing returns.

Certainly there ARE diminishing returns in many anti-spam approaches, but I feel 
(strongly) that a good implementation of my approach does not end up there.

>> When you whitelist senders, you ought to be able to do so (again) in a
> fine-grained way.

> In your proposed implementation, who controls this white list? 

The whitelist would be controlled by the recipient.

> The "you" appears to indicate the mail service provider, however, managing 
a finely grained permission set for millions of users quickly outstrips 
available resources. 

No, my plan is for the recipient (the person directly affected by its operation, 
and most directly benfitting from its proper tuning) will be the person managing 
the filters.

> Another solution is to have service 
representatives default allow behavior when a customer complains, 
however this is expensive both in representative time and in the amount 
of churn it will cause due to dissatisfied customers. 

The problem with ISP-provided solutions is precisely that they are too crude and 
too un-nuanced to enable them to do a good job.

> The most scalable 
solution is to have the customer control their settings. 

Exactly.

> This most likely would fail as most customers will not understand the options 
of a finely grained control system.

That's an implementation issue, and a good software implementation will shield 
users from those specific details.

>> Once the HTML, scripting, and attachments are zapped, there are 
> relatively few
> tricks remaining that spammers can use to evade content filters... so 
> the
> fine-grained whitelist discrimination should be followed up by a good 
> content filter.

> Do content filters not weight based on the presence of HTML, scripting, 
and attachments? 

The problem is that HTML, scripting, attachments and the like are typically used 
by spammers to evade content filters... things like "text as image", encrypted 
attachments, scripting, misrepresented links, and so forth.  Content filters can 
do a FAR better job once the input data going into them is cleansed from such 
things.

Meanwhile, ESTABLISHED senders that you TRUST (if any) can be set to allow their 
use of such more dangerous/risky stuff.  But you could set your E-mail to ONLY 
allow THOSE (few, probably) specific users to send you that type of content...

> Does not automatically deleting matches to this blanket rule reduce, not 
increase, your amount of control over what mail you allow?

No, because you can set the defaults to whatever you want.  The important 
concept is that INITIAL contacts (to establish a relationship, whether personal 
or commercial) can be done in a general and 'safe' way that works for everyone. 
 Any more advanced content can be sent ONLY after being negotiated between the 
users.

>> I disagree... I think you CAN use it to reject spam, although one 
> might argue
> that it's even more useful to reject OTHER kinds of 
> unwanted/untrustworthy
> mail.... specifically, phishing scams with misrepresented hyperlinks,
> attachments which carry viruses or worms, and other unwanted junk.

> Once again, this is semantics. Default deny versus default reject.

No.  It's more nuanced than that.  Its default behavior depends on the TYPE of 
content in the mail (and its size).  

>> I agree that knowing who the "real" sender is can be useful, although 
> by itself
> it's NO guarantee of safety (or not being spam) either.  The fact that 
> the
> "from:" address represents the actual machine of (or an alternate 
> machine
> sometimes used by) that legitimate sender is NOT a guarantee that said 
> machine
> hasn't been commandeered by a spambot zombie which is sending out spam 
> or viruses using the victim's authorizations and certificates.

> This is a remote site management issue. 

It doesn't matter.

The point is that even (usually) well-managed systems CAN be infected, and other 
users at the same ISP can often forge addresses for other users at the same ISP 
domain name.  

The approach I propose has the very nice property that it would effortlessly 
discriminate between the mail from Aunt Gertrude that LOOKS like her mail, while 
still quarantining stuff CLAIMING to be from her that doesn't "look" right.

> It is up to the remote site to 
manage whether or not their senders can forge "From:" fields. 

It's fine to tell others what they "should" do but my proposal can be 
implemented on a single-ended basis and provide IMMEDIATE benefit to the person 
responsible for installing and tuning it, whether any sending-end changes are 
made or not.  Therefore it allows eliminating the ugliness of "someday" and 
allows besieged recipients to get relief RIGHT AWAY.

> The domain based identification allows us make policy decisions based on 
administrative boundaries, that is to say we can either block poor 
senders or return mail to the administrators for further investigation. 

Again, my approach offers MUCH greater nuance than that.  A trusted friend who 
happens to be using a somewhat-flakey or sloppy ISP doesn't have to be 
penalized.  Your suggestion results in painting folks with too wide a brush, 
which prevents effective filtering.

> You cannot guarantee that the mail was send legitimately, but you can 
guarantee that your policies are affecting the responsible parties.

It leaves innocent victims who are punished by association.

>> Many "certification" schemes are worse still in that they only try to 
> determine
> that the E-mail originated from a server authorized to send mail from 
> the
> indicated "sender domain", and for domains containing (literally) 
> millions of
> valid E-mail addresses, that means that ANY of those millions of 
> machines could
> counterfeit/spoof the E-mail address of any other user within the same 
> domain
> name and still successfully pass the "authorized mail server for that 
> domain"
> test.

> Once again, "spoofing" is a domain based sender policy issue. 

Fine.  Again, my approach allows a recipient to control what THEY receive, and 
without leaving them in the frustrating position of "I wish someone else would 
do <whatever>."

> If the remote site is only allowing "From:" fields to match the authenticated 
username this is a non issue. 

No, because the mail could be FULLY authenticated, if the victim's machine has 
been infected by a spambot zombie.  That spambot could send E-mail spams and 
worms using the REAL user's AUTHENTICATED permissions.

> If the site is not authenticated mail sourced within their domain, then my 
external policies are in place to restrict this behavior.

Even if they are, authenticated mail can still be fraudulent.

>> Ultimately, it's a FAR better solution to identify spam based on 
> whether it
> "looks right" (i.e. looks like mail is EXPECTED to look, coming from 
> that
> specific sender) and that it doesn't contain anything (executable 
> attachments,
> encrypted ZIP files, Javascript, ActiveX modules, etc etc) that make 
> its purpose or intentions suspect.

> I disagree. Keeping state on specific senders will not scale to the 
whole internet. 

It doesn't have to.  You only need to have individual recipients each have a 
finite number of TRUSTED recipients (and the GREAT majority of senders a given 
recipient receives mail from will NOT need to be whitelisted).

> If you're discussing keeping state on specific senders 
in an edge model, this might be appropriate as the number of legitimate 
senders to a single receiver is small, 

Exactly.

> ...however on an enterprise level I cannot fathom a manner to efficiently 
manage state for every sender destined to my customers. 

It's really a very simple problem, and not a particularly difficult or 
complicated one to resolve (probably easier, in fact, than IP routing and 
such...)

> It is less complex to create stateless policies based on message contents and 
keep state on high traffic authenticated sender domains.

Less complex approaches (and my approach is NOT particularly complicated, at 
that) are demonstrably inadequate, again because they paint people and mail with 
such a wide brush that inevitably a lot of unwanted mail slips through.

Gordon Peterson                  http://personal.terabites.com/
1977-2002  Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections!  http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: James Baldwin <[email-masked]>
Date: Thu, 28 Apr 2005 11:59:02 -0400

> users from those specific details.

I agree that an implementation which provide ease of use and 
accessibility for users to have this level of finely grained control is 
a good thing. I do not think it will solve our problem any more than 
current solutions in edge filtering already provide. The problem I am 
referring to is not, necessarily, that of the customer being delivered 
unsolicited mail but rather the support and operating costs it 
generates. I do not think the adoption rate of any application which 
requires consistent user interaction to derive effectiveness will be 
high enough to offset the costs necessary to reduce filtering in the 
core.

If you find anything which meets your requirements, I would love to see 
it if for nothing else than personal use.

> users at the same ISP can often forge addresses for other users at the 

Unless you are referring to the risk of an ISPs mail server itself 
becoming infected, you can mitigate this risk by ensuring that only 
authenticated users can send mail and that authenticated users can only 
send mail with "from:" addresses matching their specific allowed 
addresses.

> Therefore it allows eliminating the ugliness of "someday" and

Your definition of "right away" and mine differ. Your solution still 
requires development and cost incurred by promoting its adoption within 
our user base. It would be significantly sooner than waiting for 
adoption of all players in Internet mail to secure their mail servers 
and I look forward to seeing a mail client with this functionality.

> which prevents effective filtering.

I do not encourage sloppy or somewhat-flakey players in any space. 
Inconsistent mail service is a good reason to switch to a better 
network provider. I believe a more appropriate point would be that you 
do not wish to penalize your own subscribers by blocking legitimate 
mail received from friends on sloppy providers. Unfortunately, not 
penalizing a subset of customers who receive mail from legitimate 
sources on sloppy providers often times penalizes are larger subset of 
customers who receive unsolicited mail from this same provider. And 
thus begins the magic juggling dance of blocking unsolicited mail while 
reducing false positives.

> It leaves innocent victims who are punished by association.

Encourage innocent victims to make better choices.

> without leaving them in the frustrating position of "I wish someone 

This is akin to a thread I am currently participating in on NANOG. In 
cases where customers want to be in complete control of the traffic 
they receive, they should be willing and understand that this will be 
awarded at a higher price point and include maintaining their own mail 
infrastructure.

> worms using the REAL user's AUTHENTICATED permissions.

Yes. Regardless of the content recognition you place on the receiving 
end, in this case you will receive some spam or have a higher rate of 
false positives.

> Even if they are, authenticated mail can still be fraudulent.

Authenticated mail which does not match the finely grained controls, 
which could place undue burden on the sender to know those controls in 
advance, is going to be passed along in either case. When that happens, 
it is important to have authenticated sender schemes in place in order 
to respond to the appropriate party.

> recipient receives mail from will NOT need to be whitelisted).

Agreed. When you find a system like this which provides the finely 
grained control you want, pass it along. Mail clients have provided 
this for a long time in the form of accepting messages only from known 
senders. It doesn't have the finely grained control you desire, but it 
is an effective example. As in that case, I do not think it will prove 
effective in the long run as it requires too much interaction from the 
participant in spite of its high effectiveness rate.
---
James Baldwin
hkp://pgp.mit.edu/[email-masked]
"Syntatic sugar causes cancer of the semicolon."
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg