Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Date: Tue, 19 Jun 2007 13:35:19 -0400
List-Id: <ietf-smtp.imc.org>

Sabahattin Gucukoglu wrote:

[...]

> But putting that aside, aren't you giving these spammers quite a bit
> of credit for brains and familiarity with these swish programming
> ideas?

No; the average spammer is as dumb as a post.  However, there are talented
spamware authors out there who make a tidy profit selling their software,
some of which is quite high-quality.

[...]

>> There is no way to "make" optimized ratware clients written and used by
>> criminals to do anything either.

> No, there isn't, but the objective is to make those who are willing to 
> wait, wait (including spammers that take the bate).  Normally, clients 
> that give up before the RFC-defined timeout are a problem, which is why 
> greet_pause has limited usefulness.  So this is a greylisting replacement 
> and/or a glorified greet_pause option.

OK, but I don't see the advantage of this over a short (<2s) greet_pause
combined with reasonable (~10 minute) greylisting.

> The proposal suggests a very short tarpitting time; the objective is
> to hang on to the address that's calling so that the delivery
> doesn't have to be delayed any further than the initial tarpitting
> time per cycle, because otherwise the mail may arrive on another
> address and be delayed further.

The problem is this does not scale.  For example: We have a customer
who processes approximately 120 inbound messages per second.  If he
added five minutes to each SMTP transaction length, he'd be dead in
the water, at least with standard UNIX open-source MTAs.

So the idea is infeasible for large sites that process lots of mail.
And spammers won't particularly care/notice if small sites that don't
process lots of mail use the technique.

> But so long as your mailer were sufficiently robust, it probably
> wouldn't actually notice any given delivery being tarpitted briefly
> and the likelihood of its being simultaneously tarpitted by enough
> destination sites as to cause it ill-health is probably quite small
> (and, in any event, something the client should not be trying to do
> anyway).

It depends on who "you" are.  If you're an ISP delivering 100 outbound
messages/second, I would guess that a substantial portion of them are
to a few domains like hotmail.com, aol.com, gmail.com, etc.  If one
of those big sites started tarpitting, you'd *definitely* feel the pain.

Regards,

David.

From: "Sabahattin Gucukoglu" <[email-masked]>
To: "David F. Skoll" <[email-masked]>
Date: Tue, 19 Jun 2007 18:09:10 +0100
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
CC: [email-masked]
Content-description: Mail message body
Sender: [email-masked]

Hi David,

On 18 Jun 2007 at 20:33, David F. Skoll <[email-masked]> said:

> Sabahattin Gucukoglu wrote:
> 
> > They will have to assign more 
> > threads/processes to each machine to process the deliveries, which would
> > endanger the client host system (to presumably noticeable levels);
> 
> You'd be AMAZED at how many people don't notice when their Windoze machine
> is balky/misbehaving/unresponsive.  For many people, that's the normal
> state of affairs. :-)

Aye, you've got a point there. :-)

> [...]
> 
> > means we can put greater demand on each host while it is doing absolutely
> > nothing but reading continuation lines from, I don't know, 50 different
> > smarthosts it is planning to slam, then all the better.
> 
> You don't have a criminal mind. :-)  Spam-sending engines can be *highly*
> optimized.  They need not be multithreaded; they don't queue messages.  A
> simple highly-efficient event-driven sender can probably occupy 5,000 SMTP
> servers without putting noticeable load on the client.  Multiply that by
> tens of thousands of spambot machines, and you see the scale of the
> problem.

Yes, I'd actually intended on doing a lot of operations in my MTA using 
nothing but pure events (only disk I/O may block, so that's the only thing 
you need workers around for).  (I must be a spammer. :-) )  But putting 
that aside, aren't you giving these spammers quite a bit of credit for 
brains and familiarity with these swish programming ideas?  We have seen 
better spam-sending behaviour in company-developed hardware than in 
ratware (think Ironport).  Now, don't get me wrong - I'm not asking to be 
DoSed (please :-) ).  However, there are more factors to spam delivery 
that make a lot of the efficiencies they used to take advantage of in open 
relay days back then an impossibility now.  Personalised mails, for 
instance, to get around hash-based detection or pull off social 
engineering stunts.  Or word salad and unique headers or EURIs, to get 
around bayes filtering and net-based queries.  (If you can keep your food 
in your stomach, find the documentation for one of these pieces of 
slimeware.)  And the hardcoded fd limits in most personal OSs that no end-
user could ever tweak by himself and which probably isn't as easy as a 
single system call for a spambot to do.

Of course, if you're a spam king with some common sense, you'd want your 
mail sent from a Unix machine where the program you described would be 
running a hardcore job.  Then again, you'd just note that Postfix was very 
fast on such an OS at that, and use that instead.  And every now and then, 
I can actually confirm this happening (qmail's another popular one).

> > Please go back and read my original message again.  I think you missed
> > the point.  Pausing at the greeting is not an option because there is no
> > way to make the client hold the line at that stage for longer than it
> > wants to.
> 
> There is no way to "make" optimized ratware clients written and used by
> criminals to do anything either.

No, there isn't, but the objective is to make those who are willing to 
wait, wait (including spammers that take the bate).  Normally, clients 
that give up before the RFC-defined timeout are a problem, which is why 
greet_pause has limited usefulness.  So this is a greylisting replacement 
and/or a glorified greet_pause option.

> It works because greylisting doesn't waste server resources and (if
> properly implemented) doesn't unduly inconvenience legitimate clients. This
> proposal *will* inconvenience legitimate clients.  If AOL (for example)
> discovers that 1% of its outbound connections are being tarpitted, I'm darn
> sure they'll announce that they refuse to stand for it and simply won't
> deliver mail to would-be tarpitters.

(Putting aside the fact that AOL already has quite an attitude for doing 
things their way and no other ...)  The proposal suggests a very short 
tarpitting time; the objective is to hang on to the address that's calling 
so that the delivery doesn't have to be delayed any further than the 
initial tarpitting time per cycle, because otherwise the mail may arrive 
on another address and be delayed further.  Tarpitting is not the primary 
objective, we simply don't want the client to drop off before a given 
amount of time as it might otherwise be tempted to do.  Five minutes a 
month to any given organizations' smarthosts isn't going to bankrupt any 
reasonable site, no matter how disperse the mail.  Greylisting, while it 
may not take up resources, will suffer from the address problem (the 
resolutions either require administrative help, RBLs or diminished 
effectiveness, but I take your points about the implementation) and, for 
those MTAs which don't prefer to queue mail unless they have to (like Exim 
3, not sure about 4.5), *is* an inconvenience.  Perhaps it's a matter of 
perspective, or maybe I just wish more people would run their own MTA.  
But so long as your mailer were sufficiently robust, it probably wouldn't 
actually notice any given delivery being tarpitted briefly and the 
likelihood of its being simultaneously tarpitted by enough destination 
sites as to cause it ill-health is probably quite small (and, in any 
event, something the client should not be trying to do anyway).

Cheers,
Sabahattin

-- 
Sabahattin Gucukoglu <mail<at>sabahattin<dash>gucukoglu<dot>com>
Address harvesters, snag this: [email-masked]
Phone: +44 20 88008915
Mobile: +44 7986 053399

From: "Sabahattin Gucukoglu" <[email-masked]>
To: Hector Santos <[email-masked]>
Date: Tue, 19 Jun 2007 17:02:58 +0100
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
CC: [email-masked]
Content-description: Mail message body
Sender: [email-masked]

Hi Hector,

On 18 Jun 2007 at 21:22, Hector Santos <[email-masked]> said:

> The only problem is that, as Tony and myself eventually showed by 
> actually checking a selected group of open source software, there is 
> enough legacy software out there that don't follow or follow the parsing of
> responds code in different ways.
> 
> In other words, SMTP COMPATIBILITY WAS THE MAIN ISSUE, thus it wasn't

This is essentially what I've been looking for.  Thanks!  If we aren't 
sure we'll even get reasonably consistent parsing behaviour, then we might 
as well forget it.

I'm all for pleading conformance, which is all very well providing that no-
one objects when the big boys force everyone else to adopt their broken, 
pseudo-equivalent ideas of the same standard as the standard, both in 
working implementations and, later, in written standards published on 
behalf of the IETF.  They will, though, because interoperability is key 
(read: because they can't really do a great deal about it, even en-mass, 
without thoroughly discrediting themselves as being broken and unusable 
and unfair etc. in the process to all those people who have no knowledge 
of the significance of real standards, why they're there, and all that and 
by people who can put out enough spin to make it happen).  That's just 
business as usual. :-(

Here's another one: would I be in the wrong if I didn't allow for a space 
between ":" and "<" in MAIL and RCPT commands?  That one is the law, too.  
It, too, isn't seen in anything decent, only in ratware ... and popular 
email programs.  Even Microsoft's own SMTP service now doesn't do it.  And 
then there are the mobile devices which forget the angle brackets ... and 
yet we accept and put up with this.  If I do write my MTA with the 
assumption that everyone will keep to these little pieces of the standard, 
I'll probably get lynched for not observing what happens in the real world 
without any kind of reflection on why the standards aren't describing what 
happens in it.

I do agree that it will happen, sooner or later, that we'll end up 
formalising "Please hold the line" and I suspect we'll probably also 
gradually encourage clients to follow it for some reason, no matter how 
unthinkable it would have been a few years ago.  SPF is an example of 
this; people with big spam problems will be willing to punish those who 
don't get on their bandwaggon, even if that's not in the spirit of email 
exchange and interoperability.  Both would presumably help against 
returned-mail-DoSs, among other things.

Cheers,
Sabahattin

-- 
Sabahattin Gucukoglu <mail<at>sabahattin<dash>gucukoglu<dot>com>
Address harvesters, snag this: [email-masked]
Phone: +44 20 88008915
Mobile: +44 7986 053399

Date: Mon, 18 Jun 2007 21:22:34 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "David F. Skoll" <[email-masked]>
CC: [email-masked]
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Sender: [email-masked]

David F. Skoll wrote:

>> The five minute timeout is the minimum required, so it would be the 
>> foreign host in violation.  A zealot wouldn't care, but I would.  This 
>> idea doesn't use that timeout, though; it sends continuation lines 
>> periodically at short intervals that all hosts are practically guaranteed 
>> to tolerate.
> 
> If this list is archived anywhere, please read the exchange between Hector
> Santos and me.  He wanted to use this idea to get an SMTP server to "force"
> (or at least "encourage") a client to wait longer than its normal timeout.
> I believe this will annoy legitimate SMTP clients enough that they'll find it
> worthwhile to code an overall timeout rather than a per-read timeout (as is
> commonly done now.)

[if you are going to use my name in vain, please do so correctly]

Nahhhhhhh, as was already indicated, this was proven to work from a 100% 
SMTP compliant standpoint.

The only problem is that, as Tony and myself eventually showed by 
actually checking a selected group of open source software, there is 
enough legacy software out there that don't follow or follow the parsing 
of responds code in different ways.

In other words, SMTP COMPATIBILITY WAS THE MAIN ISSUE, thus it wasn't 
prudent to clarify the specification as I proposed it to be clarified. I 
knew exactly what I was doing when I proposed it and took my chances the 
opposite would prevail - as it did.  I knew exactly what I was doing and 
would not have had proposed it if it wasn't an issue that was discussed 
and put into motion in other forums.  It wasn't a devil on my shoulder, 
nor a raised hair in my back idea - but actual exploratory work studied 
with a working history based on precedence that does back 25 years.

It, nor its conclusion, had absolutely nothing to do what you are 
talking about, other than it was simply a matter of compatibility across 
the board.

But guess what, I still believe we are still leave the industry with the 
potential timeout design delimma which may not show up at all, but could 
depending on how certain new technologies on the horizon are adopted. I 
will guarantee you someone will come across he need: "Hmmmm, I need 
something that will tell the client - please hold."  once they attempt 
begin to implement dynamic server-side processing concepts with 
potential for delays - for one reason only - without, you can have 
premature drops as it happen under FTP scenarios where general FILE 
processing is more of a reality before EMAIL AVS processing came about.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Mon, 18 Jun 2007 20:33:28 -0400
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Sender: [email-masked]

Sabahattin Gucukoglu wrote:

>> On Jun 18, 2007, at 5:41 AM, David F. Skoll wrote:
>>> 1) Tarpitting occupies server resources, making it easier to DoS the
>>> server.

> It certainly does.  It depends on which position you take as to whether or 
> not this is a problem any more than it is already, though;

I guess my point is that in almost any imagninable scenario, the server
doing the tarpitting will be forced to yell "Uncle!" before the attackers
being tarpitted even notice.

>>> 2) Tarpitting is useless against an attacker with essentially infinite
>>> CPU and bandwidth resources --- and that's the kind of attacker a 
>>> serious spammer is.

> I'm aware this is no simple adversary.

[...]

> Unless spammers exponentially increase their supply of spambots,
> though, spammers will soon hit the impass at the current rate of
> spambot deployment where the number of bots available will simply
> not be able to get current throughput without skipping recipients.

Spammers can increase their supply of spambots quite easily.

> They will have to assign more 
> threads/processes to each machine to process the deliveries, which would 
> endanger the client host system (to presumably noticeable levels);

You'd be AMAZED at how many people don't notice when their Windoze machine
is balky/misbehaving/unresponsive.  For many people, that's the normal
state of affairs. :-)

[...]

> means we can put greater demand on each host while it is doing absolutely 
> nothing but reading continuation lines from, I don't know, 50 different 
> smarthosts it is planning to slam, then all the better.

You don't have a criminal mind. :-)  Spam-sending engines can be *highly*
optimized.  They need not be multithreaded; they don't queue messages.  A
simple highly-efficient event-driven sender can probably occupy 5,000 SMTP
servers without putting noticeable load on the client.  Multiply that by
tens of thousands of spambot machines, and you see the scale of the problem.

> The five minute timeout is the minimum required, so it would be the 
> foreign host in violation.  A zealot wouldn't care, but I would.  This 
> idea doesn't use that timeout, though; it sends continuation lines 
> periodically at short intervals that all hosts are practically guaranteed 
> to tolerate.

If this list is archived anywhere, please read the exchange between Hector
Santos and me.  He wanted to use this idea to get an SMTP server to "force"
(or at least "encourage") a client to wait longer than its normal timeout.
I believe this will annoy legitimate SMTP clients enough that they'll find it
worthwhile to code an overall timeout rather than a per-read timeout (as is
commonly done now.)

> My real worry is that a client might close the connection 
> before being sent the final line in a sequence; that's obviously a serious 
> problem for this idea.

Yep.

> Please go back and read my original message again.  I think you missed the 
> point.  Pausing at the greeting is not an option because there is no way 
> to make the client hold the line at that stage for longer than it wants 
> to.

There is no way to "make" optimized ratware clients written and used by
criminals to do anything either.

Greylisting works because it ensures that spammers have to stick to
the same IP address for a few minutes (or tens of minutes) to get
their messages through.  This gives RBLs a chance to catch up.
It works because greylisting doesn't waste server resources and (if
properly implemented) doesn't unduly inconvenience legitimate clients.
This proposal *will* inconvenience legitimate clients.  If AOL (for example)
discovers that 1% of its outbound connections are being tarpitted, I'm darn
sure they'll announce that they refuse to stand for it and simply won't
deliver mail to would-be tarpitters.

Regards,

David.

Date: Mon, 18 Jun 2007 20:08:09 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Sabahattin Gucukoglu <[email-masked]>
CC: Arnt Gulbrandsen <[email-masked]>, [email-masked]
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Sender: [email-masked]

Sabahattin Gucukoglu wrote:

> This is fine providing that the client can't pipeline.  If the client 
> tries to, it should be shot.  The danger is, as David mentioned, in 
> assuming that timeouts will be honoured by anyone.  That's why I propose 
> my alternative.

IMO, you can't design software with a negative idea that no one is going
to follow the standard.  You have to work the the idea that they WILL
honor the timeouts. If not, they will are simply not compliant.  Not
your fault.  I doubt any one can engineer a reliable and working system
based on chaos and random behavior.

> By the way, just in case someone needs a rationale for killing off clients 
> that talk prematurely: the idea is that a lot of broken spambots don't 
> actually know what a continuation line is.  

Sure, but in our experience, it is better to follow SMTP behavior than 
to prematurely knock off clients.  What can happen here is that you can 
actually create redundant retries.

> They will go on issuing 
> commands under the assumption that they have a complete, one-line response 
> when it becomes available.  

Actually, what I have seen is that most will just drop, unless they were 
just dumping - behaving like a pipeline client, which most won't care 
whether the server supports pipelining or not, they are just dumping.

If you follow what you need to do and stop worrying about them, it will 
all work out.

Most spammers are 100% relying on the purity of a legacy SMTP system, 
with the cat's meow being the system that behaves like an "open relay" 
system:

     a) doesn't do any kind of checking,
     b) doesn't do local or hosted recipient validation,
     c) pretty much accepts everything (250 at all levels).

Which means that an idea that requires client change most likely will 
not have a high payoff unless you have an enforcement concept.

>> I see no reason why your variation should have less success, and would be
>> very interested in seeing a comparison if you're able to carry one out.
> 
> As I said in my previous message, not just yet.  I'm just floating an idea 
> for the moment.  When I have some more time ...

Sabahattin, SMTP patterns have long be studied.  If there was any
meaningful AVS idea here, it would of been invented by now.

In my view, the best ideas are those that follow and enforce "SMTP 
compliance" and does recipient validation at the SMTP level.

To me, tarpitting and response delays doesn't help your system 
scalability. Other than you simply controlling your loads, putting 
clients on a connection queue and if they don't want to follow SMTP 
expected behavior, they can drop with no harm done to you.

Also, those that wish to not Listen to "response" are also handled 
gracefully with error counts.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

From: "Sabahattin Gucukoglu" <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 18 Jun 2007 23:58:59 +0100
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
CC: [email-masked]
Content-description: Mail message body
Sender: [email-masked]

Hi Arnt,

On 18 Jun 2007 at 9:59, Arnt Gulbrandsen <[email-masked]> said:

> This reminds me of a technique some people are using: Delay responses by x
> seconds, where x is a constant. x=46 and x=90 have been mentioned as good
> numbers; apparently quite a few spambots drop off at 45 seconds or less,
> while hardly any proper smtp clients do.

This is fine providing that the client can't pipeline.  If the client 
tries to, it should be shot.  The danger is, as David mentioned, in 
assuming that timeouts will be honoured by anyone.  That's why I propose 
my alternative.

By the way, just in case someone needs a rationale for killing off clients 
that talk prematurely: the idea is that a lot of broken spambots don't 
actually know what a continuation line is.  They will go on issuing 
commands under the assumption that they have a complete, one-line response 
when it becomes available.  It therefore makes sense to try to convince 
the client to give us a reason for terminating its connection by making it 
wait while it is being issued continuation lines.  My only fear is that 
some other, non-spam implementation will turn out to do the same. :-(

> I see no reason why your variation should have less success, and would be
> very interested in seeing a comparison if you're able to carry one out.

As I said in my previous message, not just yet.  I'm just floating an idea 
for the moment.  When I have some more time ...

Cheers,
Sabahattin

-- 
Sabahattin Gucukoglu <mail<at>sabahattin<dash>gucukoglu<dot>com>
Address harvesters, snag this: [email-masked]
Phone: +44 20 88008915
Mobile: +44 7986 053399

Date: Mon, 18 Jun 2007 18:57:18 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
CC: [email-masked]
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
To: [email-masked]
Sender: [email-masked]

Chris Callahan wrote:
> 
>> On Mon, 18 Jun 2007, David F. Skoll wrote:
>>> That's not my experience.  We use a greet_pause of only 5 seconds, and
>>> we still get quite a few:
>> We get about 1000 a day, which is negligible compared to the rest of our
>> traffic and compared to the 100,000 per day we got two years ago.
> 
> Perhaps people are still pointing old bots at us or something. We use
> the Sendmail greet_pause feature with a 5 second delay, and are still
> averaging about 120K rejects per day based on that hurdle.

+1, there is no overwhelming pattern.

For us, our philosophy is to design for query dissemination and minimal 
session residence time.  It all came down to enforce "obvious" SMTP 
compliance and providing connection limits, load limits, monitoring your 
connection queues, balancing, etc based on the system's scalability needs.

IOW, rather than pull your hair trying figure out the randomness of the 
bad guy, just set your controls and limits to handle the loading you 
need and it pretty much works itself out.

For example, one server may have:

Maximum Accept Load: X
Maximum Accept Connections:  X+5

This means that X sessions can be active at a given time, with 5 put 
into a wait queue, i.e. no greeting response until a new slot is available.

If there is an DoS, it is handled gracefully, and if the bad guys are 
impatience can't wait for the SMTP standard 5 minutes, they typically 
drop pretty quickly.  It tickles me pink to watch these randomly 
occurring attacks quickly peter off.

If and when the greeting is presented, we found the following observations:

- Multi-line greetings will knock out a good bit of the non-SMTP 
compliant bulk spammer not expecting it multiple line responses of any kind.

- Strict EHLO/HELO domain literal checking knocks out a significant 
amount of these type of bulk spammers.

-- 
Sincerely

Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

From: "Sabahattin Gucukoglu" <[email-masked]>
To: [email-masked]
Date: Mon, 18 Jun 2007 23:35:48 +0100
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Content-description: Mail message body
Sender: [email-masked]

Hi Lisa, David and others,

On 18 Jun 2007 at 10:47, Lisa Dusseault <[email-masked]> said:

> On Jun 18, 2007, at 5:41 AM, David F. Skoll wrote:
> >
> > 1) Tarpitting occupies server resources, making it easier to DoS the
> > server.

It certainly does.  It depends on which position you take as to whether or 
not this is a problem any more than it is already, though; someone 
connecting to a server and simply keeping the connection alive with NOOPs 
or equivalent commands can achieve the same thing if the servers limits 
are within easy reach of the client and/or the implementation lends itself 
to DoS (fork vs events, etc).  The administrator of the server is 
obviously responsible for making the compromise of tarpitting performance 
over immediate mail deliverability (if you use greylisting, you're already 
less concerned about the latter).

The only thing I can think of right now that would make DoS noticeably 
easier is that it now only takes a spammer to somehow entice other 
machines to send mail to a tarpitting host in order to increase the chance 
of DoS at the tarpit (either crashed host due to misconfiguration or 
delayed mail due to limits being exceeded).  The server should obviously 
avoid taking multiple connections from any given IP to cut off the most 
obvious line of attack if that's necessary.  Finally, the problem is only 
there if hosts are sending mail under an attacker's control.  Maybe 
bounces sent by mailers that accept mail before noting delivery problems 
(like qmail) will be an issue here.  Otherwise, the mass source of 
addresses that connect under spammer control is essentially the spambots, 
and those are the very machines you want to hold up for as long as they're 
willing to wait.  Also, every host that passes mustard gets whitelisted, 
so there's a maximum time that the mailer can spend tending to a 
connection.

> > 2) Tarpitting is useless against an attacker with essentially infinite
> > CPU and bandwidth resources --- and that's the kind of attacker a 
> > serious spammer is.

I'm aware this is no simple adversary.  (Deploy a dummy SMTP service that 
accepts all commands except RCPT which causes the connection to be closed, 
give it an MX, then spread addresses routable to it.  Then just watch what 
happens.  You'll just see each rejection followed by yet another host 
willing to try.  Fun, but be careful.)  If most spam gets sent to a host 
using this scheme, then most spambots are waiting to deliver mail.  Unless 
spammers exponentially increase their supply of spambots, though, spammers 
will soon hit the impass at the current rate of spambot deployment where 
the number of bots available will simply not be able to get current 
throughput without skipping recipients.  They will have to assign more 
threads/processes to each machine to process the deliveries, which would 
endanger the client host system (to presumably noticeable levels); they 
would slow down if they did things sequentially, and the timeouts unique 
to each host make it impossible for a spammer to plan ahead of time which 
hosts he will target for a full delivery (like greylisting, except that 
this can be defeated by simply running a blast twice, for instance).  That 
assumes spammers adapt, of course; if they don't (or while they don't) 
you'll presumably enjoy a niche privilege.  But the load has to be 
distributed among a finite number of spambots in the end, and if that 
means we can put greater demand on each host while it is doing absolutely 
nothing but reading continuation lines from, I don't know, 50 different 
smarthosts it is planning to slam, then all the better.

> > 3) Relying on "genuine" clients to adhere strictly to RFC-defined  
> > timeouts
> > is dangerous.

The five minute timeout is the minimum required, so it would be the 
foreign host in violation.  A zealot wouldn't care, but I would.  This 
idea doesn't use that timeout, though; it sends continuation lines 
periodically at short intervals that all hosts are practically guaranteed 
to tolerate.  My real worry is that a client might close the connection 
before being sent the final line in a sequence; that's obviously a serious 
problem for this idea.

> > 4) It is perfectly possible to delay the client before EHLO.   
> > That's what
> > Sendmail's greet_pause feature does.  However, it's *not* designed  to be

Please go back and read my original message again.  I think you missed the 
point.  Pausing at the greeting is not an option because there is no way 
to make the client hold the line at that stage for longer than it wants 
to.  Force-feeding lines to a client which it must read the whole time 
you're doing it without daring to say anything, though, is enough either 
to waste a spammer's time or make it lose patience and incriminate itself 
(either by saying something after what it presumably thinks is a complete 
response or by dropping the connection).  In the former case, multiple 
implementations of the tarpit endanger more bots, and in the latter you 
get less spam!  That's the goal.

> > a tarpitting mechanism.  Rather, it's designed to detect SMTP clients
> > that send everything in one burst without waiting for the initial 
> > greeting (and also to detect clients that use broken proxy servers.)

That's true, although a lot of people reported setting quite insane values 
for greet_pause like a full two minutes, since a lot of spambots will 
indeed have lost interest by then.  Unfortunately, as you remarked, a few 
clients won't use the timeouts given in RFC 2821 and will also have quit 
before then, too.  I'm not sure whether HTTP proxies are in fashion 
anymore; I never see them more often than about twice a month on my 
personal machine compared to the 3000+ aborted attempts.  (But once again: 
this is not the way this proposal functions.)  

> adding:
> 5) If this were standardized or even widespread, spammers would adapt 
> easily (much easier than the good actors can change).  It can only be 
> useful as a trick that a few systems use.

I agree with all except the last sentence in the long-term, all otherwise. 
 Having deployment is part of the master plan (You can tell I've got a 
real gripe against spammers, can't you? :-) ), but not having more than a 
few is as likely to bring benefits like those of greylisting to those who 
want it while helping to slow down the existing spam-sending client 
population if they choose to wait.  If they don't, you've essentially got 
a safer greet_pause function.

I'm going to put this on the back burner for a while because I'll 
obviously have to test it for myself, and that'll require some coding for 
which I don't yet have the time (probably start with a proxy).  I'd be 
interested in any further comments, though.  The *main* point here is not 
so much the exact specification of an extension (the extension is only 
there so that we can send multiple, delayed lines in response to 
EHLO/HELO), but that client behaviour is, to the best knowledge of people 
here, sufficiently good in all truely robust client implementations to 
support this slight abuse of the characteristics of a linear protocol like 
SMTP with continuation lines.

Cheers,
Sabahattin

-- 
Sabahattin Gucukoglu <mail<at>sabahattin<dash>gucukoglu<dot>com>
Address harvesters, snag this: [email-masked]
Phone: +44 20 88008915
Mobile: +44 7986 053399

Date: Mon, 18 Jun 2007 15:27:50 -0700
To: Sabahattin Gucukoglu <[email-masked]>
From: SM <[email-masked]>
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Cc: [email-masked]
Sender: [email-masked]

Hi Sabahattin,
At 13:08 18-06-2007, Sabahattin Gucukoglu wrote:
>It would indeed, if it is true that (sensibly-minded) clients choose to
>abort a long-running transaction.  That might be a security advantage to
>avoid a DoS; I'd certainly recommend people set an upper limit on
>lingering connections that aren't actually taking mail, just in case some

That limit can be quite short nowadays to avoid tying server 
resources.  Playing SMTP tricks increases the probability of losing 
valid mail.  Your proposal is interesting.  However, it makes 
debugging more difficult.  That in itself is a significant 
disadvantage in my view.

Regards,
-sm 

Date: Mon, 18 Jun 2007 23:21:31 +0200
From: "Peter J. Holzer" <[email-masked]>
To: [email-masked]
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Mail-Followup-To: [email-masked]
Sender: [email-masked]

--VS++wcV0S1rZb1Fb
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-06-18 02:27:48 +0100, Sabahattin Gucukoglu wrote:
> However, in environments where clustering, proxying, load balancing or=20
> gatewaying are used to share the load of distributing mail from an=20
> identical source, greylisting will delay mail for longer than is necessar=
y=20
> if multiple attempts happen to be made by different hosts in a cluster.  =
=20
> If there are enough hosts sending mail, and depending on the greylisting=
=20
> timeouts and the queueing timeouts set at recipient and sender of the mai=
l=20
> respectively, and supposing every host that tries is uniquely chosen, it=
=20
> might even be possible that mail will fail to be delivered in the time th=
e=20
> sender decides mail can wait for delivery in a queue.   The mail will the=
n=20
> be returned to the sender as undeliverable for a transient reason, which=
=20
> most greylisting implementations obfuscate as general system faults. =20
> (Although it seems that this problem is already known well enough, no-one=
=20
> seems to have noticed any such returned mail and plenty of people are=20
> using greylisting now.)

When we started to use greylisting in 2003 we noticed this problem
almost immediately, because one of the largest ISPs in Austria was using
such a load-balancing cluster at the time.

Since the use of such clusters is relatively rare (right now gmail is
only one I can think of off the top of my head), it is often possible to
simply whitelist those ip ranges where you notice the problem.

> There's no way for me to experiment with this idea easily without writing=
=20
> my own MTA (something I was thinking of doing anyway, also mostly in Tcl,=
=20
> as it happens), as Sendmail, my current choice, can't possibly be extende=
d=20
> without pretty heavy patching.

If you know Perl you might want to take a look at qpsmtpd, which lets
you write small plugins in Perl which can modify just about anything in
an SMTP transaction.

	hp

--=20
   _  | Peter J. Holzer    | I know I'd be respectful of a pirate=20
|_|_) | Sysadmin WSR       | with an emu on his shoulder.
| |   | [email-masked]         |
__/   | http://www.hjp.at/ |	-- Sam in "Freefall"

--VS++wcV0S1rZb1Fb
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--VS++wcV0S1rZb1Fb--

From: "Sabahattin Gucukoglu" <[email-masked]>
To: SM <[email-masked]>
Date: Mon, 18 Jun 2007 21:08:58 +0100
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
CC: [email-masked]
Content-description: Mail message body
Sender: [email-masked]

Hi SM,

On 18 Jun 2007 at 8:50, SM <[email-masked]> said:

> At 18:27 17-06-2007, Sabahattin Gucukoglu wrote:
> >My idea is to replace greylisting with a connection-delaying technique
> >that will make the SMTP client wait until we're certain it is genuine.  We
> >differ from Greylisting only in how we determine that an SMTP client is
> >genuine; greylisting uses the fact that the client will come back, and
> >we'll use the fact that the client knows what continuation lines are and
> >that it must stay silent while we send periodic lines in the HELO/EHLO
> >response for a given time.  This isn't unlike teergrubing, except that we
> >don't need dedicated hosts for this technique and we slow the connection
> >down as soon as it's possible rather than while mail is being delivered.
> >We must not allow the client to pipeline any commands before EHLO or HELO,
> >and we must not allow the client to initiate a MAIL transaction until
> >issuing EHLO or HELO and completing the challenge either (unless, at the
> >server's discretion, the client has already "Proven" itself).  A client
> >that waits ("Proves itself") for a specified duration in minutes (five,
> >say), during which it is receiving five-secondly (for instance) "Stay on
> >the line" notices, shall eventually be allowed to see service extension
> 
> Some clients won't wait that long.  They will close the connection 
> and may retry again.  There is no guarantee that the delivery retry 
> will be from the same host.  The connection oriented approach suffers from
> the same problems as what you listed for the transaction-oriented approach
> to greylisting.

It would indeed, if it is true that (sensibly-minded) clients choose to 
abort a long-running transaction.  That might be a security advantage to 
avoid a DoS; I'd certainly recommend people set an upper limit on 
lingering connections that aren't actually taking mail, just in case some 
hostile person (or an evil admin) deploys a tarpit that never gives up.  
However, once again, this isn't what I've actually observed in a 
standalone tarpit taking mail from both Postfix and Sendmail forwarders as 
well as the big bad Internet; they have seemingly gone on for hours, 
possibly even days, waiting for the continuation lines which have 
presumably just been logged and then discarded.  But yes, if the 
connection is closed then the whole point of this idea is wasted.  If it 
happens often enough in non-robust mailers (some of which don't quite 
manage to adhere to the itty-bitty aspects of the standard) then this 
trick is only for the zealots.

Cheers,
Sabahattin

-- 
Sabahattin Gucukoglu <mail<at>sabahattin<dash>gucukoglu<dot>com>
Address harvesters, snag this: [email-masked]
Phone: +44 20 88008915
Mobile: +44 7986 053399

Date: Mon, 18 Jun 2007 11:36:52 -0700 (PDT)
From: Chris Callahan <[email-masked]>
To: [email-masked]
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Sender: [email-masked]

> On Mon, 18 Jun 2007, David F. Skoll wrote:
> >
> > That's not my experience.  We use a greet_pause of only 5 seconds, and
> > we still get quite a few:
>
> We get about 1000 a day, which is negligible compared to the rest of our
> traffic and compared to the 100,000 per day we got two years ago.

Perhaps people are still pointing old bots at us or something. We use
the Sendmail greet_pause feature with a 5 second delay, and are still
averaging about 120K rejects per day based on that hurdle.

-C
--
#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!
Chris Callahan - UCD Information Technology - [email-masked]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"[T]he most important point being debated here today is whether
governments have the right to intrude on the privacy of their
citizens in such a way that citizens ultimately cannot have a
private conversation. I should be able to whisper in your ear,
even if your ear is a thousand kilometers away."  --Philip Zimmerman

Date: Mon, 18 Jun 2007 18:20:59 +0100
From: Tony Finch <[email-masked]>
To: "David F. Skoll" <[email-masked]>
cc: [email-masked]
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Sender: [email-masked]

On Mon, 18 Jun 2007, David F. Skoll wrote:
>
> That's not my experience.  We use a greet_pause of only 5 seconds, and
> we still get quite a few:

We get about 1000 a day, which is negligible compared to the rest of our
traffic and compared to the 100,000 per day we got two years ago.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
DOVER WIGHT: VARIABLE 3, BECOMING SOUTHEAST 4 OR 5. SLIGHT OR MODERATE.
SHOWERS, RAIN LATER. MODERATE OR GOOD.

Date: Mon, 18 Jun 2007 13:04:30 -0400
From: "David F. Skoll" <[email-masked]>
To: Tony Finch <[email-masked]>, [email-masked]
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Sender: [email-masked]

Tony Finch wrote:

> Actually, greet_pause is nowadays for detecting bad clients with
> excessively short timeouts. Bad clients stopped sending all smtp
> commands in one go about two years ago.

That's not my experience.  We use a greet_pause of only 5 seconds, and
we still get quite a few:

"rejecting commands from xxxx [a.b.c.d] due to pre-greeting traffic after 0 seconds"

in our logs.

Regards,

David.

Cc: [email-masked]
From: Lisa Dusseault <[email-masked]>
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Date: Mon, 18 Jun 2007 10:47:20 -0700
To: "David F. Skoll" <[email-masked]>
Sender: [email-masked]

On Jun 18, 2007, at 5:41 AM, David F. Skoll wrote:

>
> This is a very bad idea on a number of levels:
>
> 1) Tarpitting occupies server resources, making it easier to DoS the
> server.
>
> 2) Tarpitting is useless against an attacker with essentially infinite
> CPU and bandwidth resources --- and that's the kind of attacker a  
> serious
> spammer is.

Agreed -- particularly if the spammer has access to botnets.

>
> 3) Relying on "genuine" clients to adhere strictly to RFC-defined  
> timeouts
> is dangerous.
>
> 4) It is perfectly possible to delay the client before EHLO.   
> That's what
> Sendmail's greet_pause feature does.  However, it's *not* designed  
> to be
> a tarpitting mechanism.  Rather, it's designed to detect SMTP clients
> that send everything in one burst without waiting for the initial  
> greeting
> (and also to detect clients that use broken proxy servers.)

adding:
5) If this were standardized or even widespread, spammers would adapt  
easily (much easier than the good actors can change).  It can only be  
useful as a trick that a few systems use.

Lisa

Date: Mon, 18 Jun 2007 08:50:57 -0700
To: [email-masked]
From: SM <[email-masked]>
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Sender: [email-masked]

Hi Sabahattin,
At 18:27 17-06-2007, Sabahattin Gucukoglu wrote:
>My idea is to replace greylisting with a connection-delaying technique
>that will make the SMTP client wait until we're certain it is genuine.  We
>differ from Greylisting only in how we determine that an SMTP client is
>genuine; greylisting uses the fact that the client will come back, and
>we'll use the fact that the client knows what continuation lines are and
>that it must stay silent while we send periodic lines in the HELO/EHLO
>response for a given time.  This isn't unlike teergrubing, except that we
>don't need dedicated hosts for this technique and we slow the connection
>down as soon as it's possible rather than while mail is being delivered.
>We must not allow the client to pipeline any commands before EHLO or HELO,
>and we must not allow the client to initiate a MAIL transaction until
>issuing EHLO or HELO and completing the challenge either (unless, at the
>server's discretion, the client has already "Proven" itself).  A client
>that waits ("Proves itself") for a specified duration in minutes (five,
>say), during which it is receiving five-secondly (for instance) "Stay on
>the line" notices, shall eventually be allowed to see service extension

Some clients won't wait that long.  They will close the connection 
and may retry again.  There is no guarantee that the delivery retry 
will be from the same host.  The connection oriented approach suffers 
from the same problems as what you listed for the 
transaction-oriented approach to greylisting.

Regards,
-sm 

Date: Mon, 18 Jun 2007 15:23:49 +0100
From: Tony Finch <[email-masked]>
To: "David F. Skoll" <[email-masked]>
cc: [email-masked]
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Sender: [email-masked]

On Mon, 18 Jun 2007, David F. Skoll wrote:
>
> 4) It is perfectly possible to delay the client before EHLO.  That's what
> Sendmail's greet_pause feature does.  However, it's *not* designed to be
> a tarpitting mechanism.  Rather, it's designed to detect SMTP clients
> that send everything in one burst without waiting for the initial greeting
> (and also to detect clients that use broken proxy servers.)

Actually, greet_pause is nowadays for detecting bad clients with
excessively short timeouts. Bad clients stopped sending all smtp
commands in one go about two years ago.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOUTHEAST ICELAND: NORTHEASTERLY 4 OR 5, OCCASIONALLY 6 IN SOUTHEAST. SLIGHT
OR MODERATE, OCCASIONALLY ROUGH. OCCASIONAL RAIN. MODERATE OR GOOD.

Date: Mon, 18 Jun 2007 08:41:10 -0400
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Sender: [email-masked]

Sabahattin Gucukoglu wrote:

> Greylisting ( http://www.greylisting.org/ ) would be a neat trick but for 
> one essential problem that is overlooked too often: it makes the 
> assumption that all mail transactions, having been issued at least once, 
> are always issued again and again until the delivery is successful from 
> the same host that made the initial attempt.

Not necessarily.  Our greylisting implementation only considers the
first three octets of the IP address.  So while it's true that many
organizations have a pool of sending machines, it's also true that they're
almost all within a class-C network (or a handful of class-C's, which means
maybe a couple more attempts before they break through greylisting.)

> My idea is to replace greylisting with a connection-delaying technique 
> that will make the SMTP client wait until we're certain it is genuine.

This is a very bad idea on a number of levels:

1) Tarpitting occupies server resources, making it easier to DoS the
server.

2) Tarpitting is useless against an attacker with essentially infinite
CPU and bandwidth resources --- and that's the kind of attacker a serious
spammer is.

3) Relying on "genuine" clients to adhere strictly to RFC-defined timeouts
is dangerous.

4) It is perfectly possible to delay the client before EHLO.  That's what
Sendmail's greet_pause feature does.  However, it's *not* designed to be
a tarpitting mechanism.  Rather, it's designed to detect SMTP clients
that send everything in one burst without waiting for the initial greeting
(and also to detect clients that use broken proxy servers.)

Regards,

David.

Date: Mon, 18 Jun 2007 09:59:27 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Cc: Sabahattin Gucukoglu <[email-masked]>
Sender: [email-masked]

This reminds me of a technique some people are using: Delay responses by 
x seconds, where x is a constant. x=46 and x=90 have been mentioned as 
good numbers; apparently quite a few spambots drop off at 45 seconds or 
less, while hardly any proper smtp clients do.

I see no reason why your variation should have less success, and would 
be very interested in seeing a comparison if you're able to carry one 
out.

Arnt

Date: Mon, 18 Jun 2007 08:08:26 +0100
From: Tony Finch <[email-masked]>
To: Sabahattin Gucukoglu <[email-masked]>
cc: [email-masked]
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Sender: [email-masked]

You can get pretty much the effect you want in practice by just delaying
the server greeting and/or the EHLO or HELO reply for long enough. (Much
less than the standard timeout.) Subdividing the reply and putting a delay
between the packets might help - you'd be detecting something similar to
a common bug in crappy web servers that require requests to arrive in a
single packet. However you don't need an extension to do it because it
doesn't affect correct clients and there's enough leeway in the existing
spec. Also, specifying an extension is a waste of time since it'll be
obsolete when the bad guys fix their code.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FAEROES SOUTHEAST ICELAND: NORTHEAST 5 OR 6, OCCASIONALLY 7 IN FAEROES. SLIGHT
TO MODERATE, OCCASIONALLY ROUGH. OCCASIONAL RAIN. MODERATE OR GOOD.

Date: Sun, 17 Jun 2007 22:01:47 -0600
From: Philip Guenther <[email-masked]>
To: "Robert A. Rosenberg" <[email-masked]>
cc: [email-masked]
Subject: Re: draft-klensin-rfc2821bis-04: VRFY and EXPN syntax
Sender: [email-masked]

On Sun, 17 Jun 2007, Robert A. Rosenberg wrote:
...
> When a SMTP Server hosts multiple domains (at least when Virtual 
> Hosting), it knows (or should know via the FQDN passed when the remote 
> client connects) which domain it is acting as.

Umm, what?  When does the client send the server's FQDN?  If you're 
thinking of HELO or EHLO, they include the client's name, not the 
server's.  Even if they did include the server's FQDN, that's not enough 
to tell you which domain the client was interested in talking about, as 
there may be MXes for multiple domains pointing to the same FQDN.

Philip Guenther

Date: Mon, 18 Jun 2007 14:27:03 +1200
To: [email-masked]
From: Glenn Anderson <[email-masked]>
Subject: Re: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Sender: [email-masked]

At 2:27 am +0100 18/6/2007, Sabahattin Gucukoglu wrote:
>However, in environments where clustering, proxying, load balancing or
>gatewaying are used to share the load of distributing mail from an
>identical source, greylisting will delay mail for longer than is necessary
>if multiple attempts happen to be made by different hosts in a cluster.

If that is a problem for operators of such a system, they can always 
change it so that attempts for a particular transaction are always 
retried from the same IP.

I don't think making such a system hang around on a connection for 5 
minutes is gaining them anything compared to having the same node in 
a cluster retry again 5 minutes.

>I think that's everything.  Please let me know what you think.  Is it
>feasible?  What would the implementers say?  Are the assumptions I've made
>bad or violating anything?

In the MTA I write, graylisting isn't just used as an anti-spam 
mechanism, it is also used in combination with connection 
prioritization and some other features to try and minimize the impact 
of DDoS attacks. Holding connections open for significant periods of 
time rather than quickly returning a 4XX response would be 
incompatible with that.

Glenn.

Date: Sun, 17 Jun 2007 21:54:59 -0400
From: John C Klensin <[email-masked]>
To: "David F. Skoll" <[email-masked]>, [email-masked]
Subject: Re: 2821bis: received: ... for clause
Sender: [email-masked]

--On Sunday, 17 June, 2007 20:28 -0400 "David F. Skoll"
<[email-masked]> wrote:

> 
> Robert A. Rosenberg wrote:
> 
>> Given the single-mailbox constraint, the exposure of the
>> address in a for clause, should be a non-issue no matter how
>> the address ended up in the RCPT-TO envelope list (Address in
>> To/Cc/Bcc or via a mailing list subscribe) since the message
>> is being delivered to that single mailbox not more than one
>> mailbox at that domain.
> 
> I gave an admittedly-contrived example of how this could be a
> major issue:
> 
> MAIL FROM:<[email-masked]>
> RCPT TO:<[email-masked]>
> DATA
> From: [email-masked]
> To: [email-masked]
> Subject: Memo about stock-option vesting
> 
> ...
> 
> 
> In other words, you may not want members of an alias to know
> which alias they're on.

But doesn't the current (as of 2821) warning under Security
Considerations adequately cover this case?  If not, what
language would you suggest?

    john

From: "Sabahattin Gucukoglu" <[email-masked]>
To: [email-masked]
Date: Mon, 18 Jun 2007 02:27:48 +0100
Subject: Proposal: Using Conservative EHLO Response Parser Behaviour For Tarpitting
Content-description: Mail message body
Sender: [email-masked]

Hi all,

First, sorry for the (possibly - probably - undue) length of this note!

This thought has been bugging me for a while now, and I think it might be 
the time to let you ponder over it too, what with my needing sleep sooner 
or later and so on.  I especially want to know if the idea is a bad one 
simply because it might be a little bit dishonest; it relies on SMTP 
client behaviour being almost always predictably simple.  The extension 
needed to implement the idea will be a little different than most, even if 
most clients won't notice a thing while they are being treated to its 
compulsory effects.  So here goes:

Greylisting ( http://www.greylisting.org/ ) would be a neat trick but for 
one essential problem that is overlooked too often: it makes the 
assumption that all mail transactions, having been issued at least once, 
are always issued again and again until the delivery is successful from 
the same host that made the initial attempt.  That is, the IP address of 
the connecting client is a part of the identification of that transaction, 
and changing it is impossible without triggering a new greylisting.  
Without the IP address as part of the greylisting process, of course, you 
lose most of the defense offered by it.  There are even Greylisting 
implementations which rely exclusively on the IP address, with quite a 
good success rating.

However, in environments where clustering, proxying, load balancing or 
gatewaying are used to share the load of distributing mail from an 
identical source, greylisting will delay mail for longer than is necessary 
if multiple attempts happen to be made by different hosts in a cluster.   
If there are enough hosts sending mail, and depending on the greylisting 
timeouts and the queueing timeouts set at recipient and sender of the mail 
respectively, and supposing every host that tries is uniquely chosen, it 
might even be possible that mail will fail to be delivered in the time the 
sender decides mail can wait for delivery in a queue.   The mail will then 
be returned to the sender as undeliverable for a transient reason, which 
most greylisting implementations obfuscate as general system faults.  
(Although it seems that this problem is already known well enough, no-one 
seems to have noticed any such returned mail and plenty of people are 
using greylisting now.)  There is also the fact that the delay is applied 
to a *transaction*, not a *connection*; spammers are not punished for not 
trying again, they simply don't get their mail delivered in some cases 
when they fail to come back and the few times when the mail does get 
delivered they have simply been lucky enough to make the attempt with all 
the right circumstances at a later time when greylisting wasn't being 
imposed rather than because they were conscientiously trying to deliver 
again.  Ideally, there would be a way to make the spammer suffer until 
greylisting were no longer needed for a time.

My idea is to replace greylisting with a connection-delaying technique 
that will make the SMTP client wait until we're certain it is genuine.  We 
differ from Greylisting only in how we determine that an SMTP client is 
genuine; greylisting uses the fact that the client will come back, and 
we'll use the fact that the client knows what continuation lines are and 
that it must stay silent while we send periodic lines in the HELO/EHLO 
response for a given time.  This isn't unlike teergrubing, except that we 
don't need dedicated hosts for this technique and we slow the connection 
down as soon as it's possible rather than while mail is being delivered.  
We must not allow the client to pipeline any commands before EHLO or HELO, 
and we must not allow the client to initiate a MAIL transaction until 
issuing EHLO or HELO and completing the challenge either (unless, at the 
server's discretion, the client has already "Proven" itself).  A client 
that waits ("Proves itself") for a specified duration in minutes (five, 
say), during which it is receiving five-secondly (for instance) "Stay on 
the line" notices, shall eventually be allowed to see service extension 
lines and continue the mail transaction, while its IP address is stored in 
a whitelist.  The client must not say anything (or have pipelined anything 
in advance) beyond the HELO/EHLO line, for otherwise it shall be banished 
in an administratively-defined fashion (probably just a 421 response 
followed by a connection closure, but there's nothing to stop you 
blacklisting, firewalling, or what have you; people sensitive about read 
buffers can just empty the buffer with nonblocking reads, issue a final 
"250 XTARPIT Sorry, matey, you've blown it!" and then reject every 
following line except QUIT with 503).  Whitelisted entries run down to 
zero in seconds, starting with some administratively-defined time (a 
month, say).

How I came up with the idea: For a time, I used to rely on the fact that 
most genuine clients could wait up to the RFC 2821-defined five minute 
read timeout.  Unfortunately (of course, there has to be someone, doesn't 
there?) not all hosts would wait even a minute.  Sendmail's greet_pause 
rule gave me the idea of trying long timeouts before the 220 greeting in 
the hope that spammers would drop connections in disgust, not for just 
avoiding HTTP proxies identified with pre-greeting traffic.  They did.  I 
was on Windows at the time with Mercury/32, so I wrote a proxy (in Tcl, 
naturally) to do the delay work, drop connections with pre-greeting 
traffic and log local or remote closures and then experimented with 
various timeouts.  I soon noticed that spammers would be gradually 
(surprise, surprise) willing to wait longer and longer, until I had one 
waiting the full 150 seconds (2.5 minutes) without closing.  From the 
addresses I saw connect, I'd assume they were zombie boxes.  Soon 
thereafter, it was what I saw most of the time.  By this time, I was 
losing connections from such prominently uninteresting services as 
Gmail.com, and so gave up the hope.  With this scheme, I hope to take away 
any final excuse any self-respecting SMTP client might have for giving up 
too early, by essentially not letting them time out.  I think it can work, 
and that we can set our timeouts as high as we like and still manage to 
exhaust spammer patience without severely impacting our mail deliveries 
or, should we fail to impact the spammers when many people are 
implementing this and spammers are willing to wait, to at least increase 
the risk that their mass-mailing tools will break under them on however 
many machines they are deployed upon.  Each waiting session to a mailer is 
a thread or process wasted, but only the spammers lose by not being 
patient - the rest are tied up only for one occasion in a set period.  
Spammers are, as we all know, lazy, and look for excuses to move on.  The 
one thing they really do have, though, is potentially limitless botpower; 
we have to put ourselves in a position where we can really challenge it.

Now, to the extension.  This is where we feel a bit dirty.  I hope you can 
see so far that there are no opportunities for delaying the client before 
he greets us (no, we can't tamper with the TCP send window, nor can we add 
continuation lines to the 220 greeting line without definitely stepping on 
very shaky ground [RFCs 821 and 2821 don't clearly provision for a 
multiline response that isn't sent in response to a command]), and there's 
no point in delaying the client once he's greeted us since our PIPELINING 
extension is there for a reason and we want every good mailer to use it.  
(We might do this tarpit at the MAIL command if we didn't send PIPELINING 
to a new client and did to a known-good one, but then the first 
transaction of a new client would always have to be non-pipelined and if 
it happened to be, say, the output of a mailing list job for a given host, 
we'd have a lot of waste.  However, if the client is using HELO he can 
never use pipelining anyway, so we can delay it until MAIL if anyone here 
thinks it would help at all.)  The extension must be repeated multiple 
times in the EHLO response.  This is the important bit (and the point of 
this long message).  The below conditions are found to be true with 
Sendmail:

This is all for nothing if SMTP clients will not ignore unknown extensions 
by simply forgetting about them and expect all of the lines sent to be in 
one write!

Personally, I'm pretty sure this is true for *most* truely robust clients, 
but it's also true that this extension makes novel use of the response by 
outputting lines on a timer.  OpenBSD's spamd stutters characters on a 
line; however, I am aware of at least one modern implementation which will 
actually choke if the entire line is not sent in one write with CRLF, 
never mind the group of lines.

It may be that the extension used can convey additional data to the 
client, such as how long it'll have to wait, but it could just as easily 
(and this is probably better for security reasons) convey a human-readable 
string (see below for an example).

There's no way for me to experiment with this idea easily without writing 
my own MTA (something I was thinking of doing anyway, also mostly in Tcl, 
as it happens), as Sendmail, my current choice, can't possibly be extended 
without pretty heavy patching.  I might try to whip up a proxy, but it'll 
be tricky if I still intend to keep all the other service extensions and 
if I can get other people's ideas first that would go a long way.

Using the time-honoured convention of "C: " marking lines sent by an SMTP 
client and "S: " marking those sent by the SMTP service, here's an example 
of the whole thing in action, as I imagine it.  The client is connecting 
from the same address on two occasions.  Not all the possibilities are 
shown here, but that can wait.  Remember that the HELO command works in 
the same way as EHLO except that the extensions aren't listed (or, as 
stated above, it could work in the usual way and the tarpit could be 
delayed until MAIL if it helps since there can be no pipelining anyway).  
Also, a server might (but it isn't shown here) be enterprising enough to 
note the hostname in EHLO/HELO in the whitelisting along with the IP 
address, so that it can react differently when the same host issues 
another name on a reconnection.  I don't know if this is necessarily a 
good idea, although it shouldn't affect most clients.  Here's the example:

S: [listening on SMTP port]
C: [Connects to server's listening port]
S: 220 bloodstone.yamta.org ESMTP YAMTA (MEOW!)
C: ehlo goodmail.example.net
S: 250-bloodstone.yamta.org Okay, I'll run off and hide.  You wait.
S: 250-XTARPITTING Be quiet!
S: [Hesitates 5 sec]
S: 250-XTARPITTING Be quiet!
S: [Hesitates 5 sec]
S: 250-XTARPITTING Be quiet!
S: [... and so on until five minutes have passed]
S: 250-XTARPITTING No, you're nice! (PURR)  Feed me!  I support:
S: 250-PIPELINING
S: 250-8BITMIME
S: 250-BINARYMIME
S: 250-CHUNKING
S: 250-CHECKPOINT
S: 250-SIZE
S: 250-DSN
S: 250-AUTH CRAM-MD5 DIGEST-MD5 PLAIN
S: 250 HELP
C: [Goes on to complete any number of MAIL/RCPT/DATA|BDAT transactions]
C: quit
S: 221 Nice to meet you! (RUB)
S: [Closure of channel, back to listening]
C: [After arbitrary amount of time prior to whitelist record expiry, 
connects again]
S: 220 bloodstone.yamta.org ESMTP YAMTA (MEOW!)
C: ehlo goodmail.example.net
S: 250-bloodstone.yamta.org You again? (PURR)  Feed me!  I support:
S: [ ... service extensions as before]
C: [As before, transactions]
C: quit
S: 221 Nice to meet you! (RUB)
S: [Closure of channel, back to listening]
C: [closure of channel]

I think that's everything.  Please let me know what you think.  Is it 
feasible?  What would the implementers say?  Are the assumptions I've made 
bad or violating anything?  Even if it worked in practice (I'm quite sure 
it will), should it be discarded for being possibly a little too dependent 
on common implementation strategy (I.E. the robustness principle)?

Cheers,
Sabahattin

-- 
Sabahattin Gucukoglu <mail<at>sabahattin<dash>gucukoglu<dot>com>
Address harvesters, snag this: [email-masked]
Phone: +44 20 88008915
Mobile: +44 7986 053399

Date: Sun, 17 Jun 2007 20:28:51 -0400
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Re: 2821bis: received: ... for clause
Sender: [email-masked]

Robert A. Rosenberg wrote:

> Given the single-mailbox constraint, the exposure of the address in a
> for clause, should be a non-issue no matter how the address ended up in
> the RCPT-TO envelope list (Address in To/Cc/Bcc or via a mailing list
> subscribe) since the message is being delivered to that single mailbox
> not more than one mailbox at that domain.

I gave an admittedly-contrived example of how this could be a major issue:

MAIL FROM:<[email-masked]>
RCPT TO:<[email-masked]>
DATA
From: [email-masked]
To: [email-masked]
Subject: Memo about stock-option vesting

...

In other words, you may not want members of an alias to know which
alias they're on.

Regards,

David.

Date: Sun, 17 Jun 2007 18:32:51 -0400
To: [email-masked]
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: 2821bis: received: ... for clause
Sender: [email-masked]

At 09:35 +0200 on 06/14/2007, Arnt Gulbrandsen wrote about Re: 
2821bis: received: ... for clause:

>[email-masked] writes:
>>  The implementations I've seen use the "for" only if there is one
>>  address.  Isn't it already constrained to one mailbox?
>
>Sort of. Sane implementations constrain it to one mailbox. Must not
>reveal bcc and so on and so forth.
>
>I think my opinion is that since other concerns constrain it to one
>mailbox and the syntax is broken except in the single-mailbox case, the
>syntax should constrain it, too.
>
>Arnt

Given the single-mailbox constraint, the exposure of the address in a 
for clause, should be a non-issue no matter how the address ended up 
in the RCPT-TO envelope list (Address in To/Cc/Bcc or via a mailing 
list subscribe) since the message is being delivered to that single 
mailbox not more than one mailbox at that domain.

Date: Sun, 17 Jun 2007 18:35:09 -0400
To: [email-masked]
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: draft-klensin-rfc2821bis-04: VRFY and EXPN syntax
Sender: [email-masked]

At 10:05 +0300 on 06/16/2007, Kari Hurtta wrote about 
draft-klensin-rfc2821bis-04: VRFY and EXPN syntax:

>--- start quote ------
>   "User name" is a fuzzy term and has been used deliberately.  An
>    implementation of the VRFY or EXPN commands MUST include at least
>    recognition of local mailboxes as "user names".  However, since
>    current Internet practice often results in a single host handling
>    mail for multiple domains, hosts, especially hosts that provide this
>    functionality, SHOULD accept the "local-part@domain" form as a "user
>    name"; hosts MAY also choose to recognize other strings as "user
>    names".
>--- end quote ------
>
>However later actual syntax seems not allow "@domain" which is "SHOULD"
>on here.

When a SMTP Server hosts multiple domains (at least when Virtual 
Hosting), it knows (or should know via the FQDN passed when the 
remote client connects) which domain it is acting as. Thus a 
Local-Part only should be adequate. A note in the quote above of the 
requirement that it knows who it is acting as might be helpful to 
clarify the situation.

To: [email-masked]
From: Kari Hurtta <[email-masked]>
Subject:  draft-klensin-rfc2821bis-04: VRFY and EXPN syntax
Date:  16 Jun 2007 10:05:20 +0300
Lines: 84
Cc: John C Klensin <[email-masked]>, Kari Hurtta <[email-masked]>
Sender: [email-masked]

Chapter "3.5.1.  Overview"   says:

--- start quote ------
  "User name" is a fuzzy term and has been used deliberately.  An
   implementation of the VRFY or EXPN commands MUST include at least
   recognition of local mailboxes as "user names".  However, since
   current Internet practice often results in a single host handling
   mail for multiple domains, hosts, especially hosts that provide this
   functionality, SHOULD accept the "local-part@domain" form as a "user
   name"; hosts MAY also choose to recognize other strings as "user
   names".
--- end quote ------

However later actual syntax seems not allow "@domain" which is "SHOULD"
on here.

--- start quote ------
4.1.1.6.  VERIFY (VRFY)

   This command asks the receiver to confirm that the argument
   identifies a user or mailbox.  If it is a user name, information is
   returned as specified in Section 3.5.

   This command has no effect on the reverse-path buffer, the forward-
   path buffer, or the mail data buffer.

   Syntax:

      "VRFY" SP String CRLF

4.1.1.7.  EXPAND (EXPN)

   This command asks the receiver to confirm that the argument
   identifies a mailing list, and if so, to return the membership of
   that list.  If the command is successful, a reply is returned
   containing information as described in Section 3.5.  This reply will
   have multiple lines except in the trivial case of a one-member list.

   This command has no effect on the reverse-path buffer, the forward-
   path buffer, or the mail data buffer and may be issued at any time.

   Syntax:

      "EXPN" SP String CRLF
--- end quote ------

And String was defined as:

--- start quote ------

   Local-part     = Dot-string / Quoted-string

                     ; MAY be case-sensitive

   Dot-string     = Atom *("."  Atom)

   Atom           = 1*atext

   Quoted-string  = DQUOTE *qcontent DQUOTE

   String         = Atom / Quoted-string
--- end quote ------

Should  syntax to be

      "VRFY" SP ( Local-part / Mailbox ) CRLF

and

      "EXPN" SP ( Local-part / Mailbox ) CRLF

/ Kari Hurtta

Date: Thu, 14 Jun 2007 12:52:50 -0400
From: John C Klensin <[email-masked]>
To: Matti Aarnio <[email-masked]>
cc: [email-masked]
Subject: 2821bis-04 Issues list (wasL Re: 2821bis: received: ... for clause)
Sender: [email-masked]

--==========B7563714DBA24CA52293==========
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

--On Thursday, June 14, 2007 19:13 +0300 Matti Aarnio 
<[email-masked]> wrote:

>...
>> Would it help if I redistributed the list?
>
> Yes please.  And/or give an URL where it is online viewable.

Attached.
     john

--==========B7563714DBA24CA52293==========
Content-Type: application/pdf; name="RFC2821bis-status.pdf"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="RFC2821bis-status.pdf";
 size=14193

--==========B7563714DBA24CA52293==========--

Date: 	Thu, 14 Jun 2007 19:13:50 +0300
From: Matti Aarnio <[email-masked]>
To: John C Klensin <[email-masked]>
Cc: Arnt Gulbrandsen <[email-masked]>, [email-masked], [email-masked]
Subject: Re: 2821bis: received: ... for clause
Sender: [email-masked]

On Thu, Jun 14, 2007 at 11:21:51AM -0400, John C Klensin wrote:
...
> Tony needs to make a ruling, but I thought we had already agreed 
> to a one-address restriction and a syntax for Opt-info that was, 
> more or less,
>     [Via] [With] [ID] [For] [Additional-registered-clauses]
> 
> It is on the list as Issue 37 in any event.
> 
> Would it help if I redistributed the list?

Yes please.  And/or give an URL where it is online viewable.

>      john

-- 
/Matti Aarnio	<[email-masked]>

Date: Thu, 14 Jun 2007 11:21:51 -0400
From: John C Klensin <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>, [email-masked]
cc: [email-masked]
Subject: Re: 2821bis: received: ... for clause
Sender: [email-masked]

--On Thursday, June 14, 2007 09:35 +0200 Arnt Gulbrandsen 
<[email-masked]> wrote:

>
> [email-masked] writes:
>> The implementations I've seen use the "for" only if there is
>> one  address.  Isn't it already constrained to one mailbox?
>
> Sort of. Sane implementations constrain it to one mailbox.
> Must not reveal bcc and so on and so forth.
>
> I think my opinion is that since other concerns constrain it
> to one mailbox and the syntax is broken except in the
> single-mailbox case, the syntax should constrain it, too.

Tony needs to make a ruling, but I thought we had already agreed 
to a one-address restriction and a syntax for Opt-info that was, 
more or less,
     [Via] [With] [ID] [For] [Additional-registered-clauses]

It is on the list as Issue 37 in any event.

Would it help if I redistributed the list?

      john

Date: Thu, 14 Jun 2007 09:35:35 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: 2821bis: received: ... for clause
Cc: [email-masked]
Sender: [email-masked]

[email-masked] writes:
> The implementations I've seen use the "for" only if there is one 
> address.  Isn't it already constrained to one mailbox?

Sort of. Sane implementations constrain it to one mailbox. Must not 
reveal bcc and so on and so forth.

I think my opinion is that since other concerns constrain it to one 
mailbox and the syntax is broken except in the single-mailbox case, the 
syntax should constrain it, too.

Arnt

Date: Wed, 13 Jun 2007 22:17:07 -0700
To: Matti Aarnio <[email-masked]>, [email-masked]
From: SM <[email-masked]>
Subject: Re: 2821bis: three-digit-and-crlf replies
Sender: [email-masked]

At 04:55 13-06-2007, Matti Aarnio wrote:
>My smtp-client reply mistreatment pitfall list has:
>
>
>   - SMTP server replies are not guaranteed to arrive to client
>     in single network packet, single line may arrive in multiple
>     network packets.
>   - SMTP server may use multi-line reply (see X.X.X)
>   - Multiline replies may arrive in multiple network packets
>     or in a single one
>   - SMTP server may reply with "unexpected" reply codes, and
>     thus knowing only how to react on expected set of precise
>     codes is usually worst possible implementation strategy.

The last one above is a common pitfall.

>     In X.X.X of "Theory of reply codes" (sp?) do pay attention
>     on the meaning of the first digit of the reply code!

That's the theory.  The command-reply sequences depicts what codes to 
expect.  The 1xx reply code is not used for SMTP.

>All of those were encountered some 10 years ago in one big
>and widely used workgroup communication and document management
>system that has ad-hoc internet email interface .. possibly
>the bugs still exist in it.

Thanks for sharing the above list.

Regards,
-sm

Date: Wed, 13 Jun 2007 21:38:17 -0700
To: SMTP Interest Group <[email-masked]>
From: SM <[email-masked]>
Subject: Re: 2821bis: received: ... for clause
Sender: [email-masked]

At 09:38 12-06-2007, Arnt Gulbrandsen wrote:
>The 'for' production is incorrect as it stands. "Received: .... for 
>[email-masked]@att.com; ..." is legal according to 2821. Oops. 
>There are 2-3 ways out of this:
>  - constrain 'for' to one address
>  - introduce a separator:
>    - either FWS
>    - or "," FWS.
>
>My preference is to constrain it to one address. I cannot remember 
>seeing production softwaret that inserts multiple addresses on 
>purpose, so IMO this can be classified as an unused feature and 
>dropped. (I checked 280k received fields just now and found none.)

The implementations I've seen use the "for" only if there is one 
address.  Isn't it already constrained to one mailbox?

Regards,
-sm 

Date: Wed, 13 Jun 2007 23:25:12 -0400
To: SMTP Interest Group <[email-masked]>
From: "Robert A. Rosenberg" <[email-masked]>
Subject: Re: 2821bis: received: ... for clause
Sender: [email-masked]

At 18:38 +0200 on 06/12/2007, Arnt Gulbrandsen wrote about 2821bis: 
received: ... for clause:

>Hi,
>
>The 'for' production is incorrect as it stands. "Received: .... for 
>[email-masked]@att.com; ..." is legal according to 2821. Oops. 
>There are 2-3 ways out of this:
>  - constrain 'for' to one address
>  - introduce a separator:
>    - either FWS
>    - or "," FWS.
>
>My preference is to constrain it to one address. I cannot remember 
>seeing production softwaret that inserts multiple addresses on 
>purpose, so IMO this can be classified as an unused feature and 
>dropped. (I checked 280k received fields just now and found none.)
>
>Arnt

The rules as implemented (so far as all mail that I've received that 
has a "for" clause) SEEMS to be that the "for address" is optional in 
the case where there is ONE RCPT-TO envelope command and suppressed 
when more than one RCPT-TO is supplied (which serves as a privacy 
protection in that Bcc and mailing-list echoed messages that are 
handed off to an SMTP Server with more than one addressee being 
handled by that server will not expose the other addressees in the 
Received headers of the POP/IMAP cloned copies). IOW: The a "for" 
clause with more than one address is not valid in implementations.

OTOH - I have no problem with an implementation that once it 
recognizes that it is supporting the ultimate destination domain, 
either clones a multi-RCPT-TO message and inserts the correct 
single-address "for" clause into each cloned copy OR does the 
insertion when it does final delivery into a POP/IMAP mailbox 
(assuming a Mail Box implementation where each mailbox is a private 
store and not one that is linked to a common store which has a common 
copy that gets deleted when the share count on the message goes to 0).

Date: Wed, 13 Jun 2007 21:01:06 -0400
From: "David F. Skoll" <[email-masked]>
To: [email-masked]
Subject: Re: Question on retry strategy in 2821
Sender: [email-masked]

Rolf E. Sonneveld wrote:

> The problem here is that the greylisting mechanism used by Getronics
> remembers the connection attempt for only one hour.

That greylisting mechanism is broken, then.  Getronics should change
it.

Greylisting is unfriendly, so greylist implementors should do
everything in their power to mitigate collateral damage.  Our
greylisting implementation remembers greylisting details for 48 hours.
In addition, once an SMTP host has passed the greylisting hurdle, we
disable greylisting completely for that host for 40 days.  (A host
that retried once is very likely to retry every time, so further
greylisting is pointless.)

Even so, we do not enable greylisting by default; our customers have
to turn it on if they want to use it.

Regards,

David.

Date: Wed, 13 Jun 2007 17:20:30 -0700
To: "Rolf E. Sonneveld" <[email-masked]>, [email-masked]
From: SM <[email-masked]>
Subject: Re: Question on retry strategy in 2821
Sender: [email-masked]

Hi Rolf,
At 14:36 13-06-2007, Rolf E. Sonneveld wrote:
>today I got the following mail from Postmaster at getronics.com 
>(Getronics being an ICT company with many thousands of employees, 
>spread over several countries):
>
><quote>
>The sending system (IP address <my MTA's IP address here>) 
>apparently retries delivery only every two hours. The intial retry 
>after a 421 status should be less than 30 minutes. Please refer to 
>RFC 2821, section 4.5.4.1 "Sending Strategy", where the recommended 
>first retry should be 30 minutes, and there should be two retries 
>within the first hour. The initial retry after a 421 status should 
>probably be somewhat shorter (say 15 - 20 minutes).
>
>The problem here is that the greylisting mechanism used by Getronics 
>remembers the connection attempt for only one hour. After that the 
>next connection is seen as if was a new, and not a retried, 
>connection and it's greylisted again.

A 450 status code would have been more appropriate for greylisting.

>The owner of the sending server should adjust the first two retry 
>intervals (and certainly the first one) to fall within the 
>recommendation made in RFC2821 to avoid problems when interacting 
>with other email systems.
></quote>
>
>Am I misinterpreting RFC2821 in that (IMO) there is no such 
>requirement as stated here?

There is a recommendation in Section 4.5.4.1 for the first retry to 
be at least 30 minutes. There is no recommendation for two retries 
within an hour.

>It is not my intention to start a (flame) war on greylisting, but 
>keeping greylist information for only one hour needs a reality 
>check, isn't it? I know at least two mature and well-known MTA 
>implementations that have a default retry interval which doens't 
>match the requirements of this domain.

One hour can be quite short for such a purpose.  Some SMTP clients 
may retry after a few hours.  Even if the retry is after 30 minutes, 
there is no way to ensure that the connection will be successful.

Regards,
-sm 

Date: Thu, 14 Jun 2007 11:59:06 +1200
To: [email-masked]
From: Glenn Anderson <[email-masked]>
Subject: Re: Question on retry strategy in 2821
Sender: [email-masked]

At 11:36 pm +0200 13/6/2007, Rolf E. Sonneveld wrote:
>Hi,
>
>today I got the following mail from Postmaster at getronics.com 
>(Getronics being an ICT company with many thousands of employees, 
>spread over several countries):
>
><quote>
>The sending system (IP address <my MTA's IP address here>) 
>apparently retries delivery only every two hours. The intial retry 
>after a 421 status should be less than 30 minutes. Please refer to 
>RFC 2821, section 4.5.4.1 "Sending Strategy", where the recommended 
>first retry should be 30 minutes, and there should be two retries 
>within the first hour. The initial retry after a 421 status should 
>probably be somewhat shorter (say 15 - 20 minutes).

<snip>

Section 4.5.4.1 of RFC 2821 says "The sender MUST delay retrying a 
particular destination after one attempt has failed.  In general, the 
retry interval SHOULD be at least 30 minutes;"

"at least 30 minutes" to me means the retry interval should be 30 
minutes or more, not 30 minutes or less.

>Am I misinterpreting RFC2821 in that (IMO) there is no such 
>requirement as stated here?

There certainly doesn't seem to be any requirement for a shorter 
retry after a 421 response.

>It is not my intention to start a (flame) war on greylisting, but 
>keeping greylist information for only one hour needs a reality 
>check, isn't it?

I don't think their implementation is very robust. The default 
setting for my MTA is to keep graylisting information for 30 days.

>I know at least two mature and well-known MTA implementations that 
>have a default retry interval which doens't match the requirements 
>of this domain.

My MTA defaults to an hour delay to retry, typically that will result 
in slightly more than an hour between retries, which would not appear 
to meet the requirements of that domain.

Glenn.

Date: Wed, 13 Jun 2007 18:24:13 -0400
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "Rolf E. Sonneveld" <[email-masked]>
CC: [email-masked]
Subject: Re: Question on retry strategy in 2821
Sender: [email-masked]

Rolf E. Sonneveld wrote:
> 
> Hi,
> 
> today I got the following mail from Postmaster at getronics.com 
> (Getronics being an ICT company with many thousands of employees, spread 
> over several countries):
> 
> <quote>
> The sending system (IP address <my MTA's IP address here>) apparently 
> retries delivery only every two hours. The intial retry after a 421 
> status should be less than 30 minutes. Please refer to RFC 2821, section 
> 4.5.4.1 "Sending Strategy", where the recommended first retry should be 
> 30 minutes, and there should be two retries within the first hour. The 
> initial retry after a 421 status should probably be somewhat shorter 
> (say 15 - 20 minutes).
> 
> The problem here is that the greylisting mechanism used by Getronics 
> remembers the connection attempt for only one hour. After that the next 
> connection is seen as if was a new, and not a retried, connection and 
> it's greylisted again.
> 
> The owner of the sending server should adjust the first two retry 
> intervals (and certainly the first one) to fall within the 
> recommendation made in RFC2821 to avoid problems when interacting with 
> other email systems.
> </quote>
> 
> Am I misinterpreting RFC2821 in that (IMO) there is no such requirement 
> as stated here?
> 
> It is not my intention to start a (flame) war on greylisting, but 
> keeping greylist information for only one hour needs a reality check, 
> isn't it? I know at least two mature and well-known MTA implementations 
> that have a default retry interval which doens't match the requirements 
> of this domain.
> 
> /rolf

Right. These are no requirements but suggestions. But RFC2821 does 
suggest at least 30 minutes.

4.5.4.1 Sending Strategy

    .... In general, the retry interval SHOULD be at
    least 30 minutes; however, more sophisticated and variable strategies
    will be beneficial when the SMTP client can determine the reason for
    non-delivery.

IMO, if you have incorporated a greylist system or wish to consider 
greylist encounters with outbound mail, then you really do need an 
variable frequency, with a much shorter retry for at least the 2nd 
attempt.   Otherwise, greylist will give you a blackeye.  Greylist is 
real. People use it, and statistically it was found that at least 1 hour 
(or < 60 minutes like 55 mins) is used as the hold/block time.

We use to have a DEFAULT 1 hour retry, 72 attempts or 3 days.  Sysops 
changed it as they wanted it.  But the frequency was fixed per attempt.

When we added greylisted, it is immediate and obvious that a more 
variable frequency table was prudent.

Today we give sysops a frequency table and the default table is:

[Attempts]
Default=60
Attempt1=5
Attempt2=5
Attempt3=15
Attempt5=30
Attempt10=120
Attempt21=5
Attempt22=5
Attempt23=15
Attempt25=30
Attempt30=120
Attempt40=60
Attempt72=60

-- 
Hector Santos, CTO
http://www.santronics.com
http://santronics.blogspot.com

Date: Wed, 13 Jun 2007 23:36:28 +0200
From: "Rolf E. Sonneveld" <[email-masked]>
Subject: Question on retry strategy in 2821
To: [email-masked]
Sender: [email-masked]

Hi,

today I got the following mail from Postmaster at getronics.com (Getronics being an ICT company with many thousands of employees, spread over several countries):

<quote>
The sending system (IP address <my MTA's IP address here>) apparently retries delivery 
only every two hours. The intial retry after a 421 status should be less 
than 30 minutes. Please refer to RFC 2821, section 4.5.4.1 "Sending 
Strategy", where the recommended first retry should be 30 minutes, and there 
should be two retries within the first hour. The initial retry after a 421 
status should probably be somewhat shorter (say 15 - 20 minutes).

The problem here is that the greylisting mechanism used by Getronics 
remembers the connection attempt for only one hour. After that the next 
connection is seen as if was a new, and not a retried, connection and it's 
greylisted again.

The owner of the sending server should adjust the first two retry intervals 
(and certainly the first one) to fall within the recommendation made in 
RFC2821 to avoid problems when interacting with other email systems.
</quote>

Am I misinterpreting RFC2821 in that (IMO) there is no such requirement as stated here?

It is not my intention to start a (flame) war on greylisting, but keeping greylist information for only one hour needs a reality check, isn't it? I know at least two mature and well-known MTA implementations that have a default retry interval which doens't match the requirements of this domain.

/rolf

Date: 	Wed, 13 Jun 2007 14:55:40 +0300
From: Matti Aarnio <[email-masked]>
To: [email-masked]
Subject: Re: 2821bis: three-digit-and-crlf replies
Sender: [email-masked]

On Tue, Jun 12, 2007 at 05:20:18PM -0400, John C Klensin wrote:
....
> So...
> 
> Peter is correct: "250"CRLF (and "250 "CRLF) are in violation of 
> the spec, but clients are expected to handle it as if they had 
> received
>   "250 random-string"CRLF
> 
> If one wants the ABNF to be normative and to accurately cover 
> all of the corner cases, the 2821 syntax and the 2821bis fix are 
> a little buggy.
> 
>    john

I am following semi-strict SMTP protocol syntax analysis at the
smtp-server, but very loose one in smtp-client code that parses
replies given to it.

My server will in general conform to that "must have random-string"
type thinking.  (Replies to "HELP" may have blank lines, perhaps
even at last reply line.)

I would prefer if the syntax rules are precise for the input
that servers are expecting, but the reply processing should IMO
be as relaxed as possible.

My smtp-client reply mistreatment pitfall list has:

  - SMTP server replies are not guaranteed to arrive to client
    in single network packet, single line may arrive in multiple
    network packets.
  - SMTP server may use multi-line reply (see X.X.X)
  - Multiline replies may arrive in multiple network packets
    or in a single one
  - SMTP server may reply with "unexpected" reply codes, and
    thus knowing only how to react on expected set of precise
    codes is usually worst possible implementation strategy.
    In X.X.X of "Theory of reply codes" (sp?) do pay attention
    on the meaning of the first digit of the reply code!
  - In case implementer wants to have precise treatment of
    something (like "551 User not local; please try <forward-path>"
    or "251 User not local; will forward to <forward-path>")
    they can do so, but at the same time must also handle
    first-digit based rough classification for everything
    that does not need special cases - or are not known
    at the implementation time.)

All of those were encountered some 10 years ago in one big
and widely used workgroup communication and document management
system that has ad-hoc internet email interface .. possibly
the bugs still exist in it.

Should that kind of "SMTP pitfalls avoidance guide" be written
as separate RFC ?  Or as some sort of Wiki ?

-- 
/Matti Aarnio	<[email-masked]>

Date: Wed, 13 Jun 2007 02:35:39 -0700
From: "Chakkaravarthy PB" <[email-masked]>
To: [email-masked]
Subject: Question on STARTTLS (rfc 3207)
Sender: [email-masked]

I have a question on when the client can issue STARTTLS after the
server advertises the same in response to ehlo. The usage example in
section 5, illustrates the way STARTTLS would typically be used -  the
client issues STARTTLS immediately after it knows that the server
supports it. The client issues any other command only after
successfully completing the TLS handshake. I understand that this is
how most of the clients configured to use STARTTLS would behave.
However, a client can issue a STARTTLS after sending a few messages to
the server in the clear.For instance, sendmail allows the client to
send messages in the clear. However, if the client decides to change
it's mind and issues a STARTTLS, it allows the client to complete the
TLS handshake(if configured properly) and thereupon, the communication
b/w the client & sendmail server would be over the TLS channel. The
question is: Should SMTP servers necessarily allow clients to upgrade
to TLS in the middle of the same SMTP session? Instead, can a SMTP
server support STARTTLS only if the client issues the same before
sending any messages to the server and still claim to be  conforming
to rfc 3207?

Thanks,
Chakkaravarthy.

Date: Tue, 12 Jun 2007 17:20:18 -0400
From: John C Klensin <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>, [email-masked]
cc: "Peter J. Holzer" <[email-masked]>
Subject: Re: 2821bis: three-digit-and-crlf replies
Sender: [email-masked]

--On Tuesday, June 12, 2007 22:15 +0200 Arnt Gulbrandsen 
<[email-masked]> wrote:

>
> Peter J. Holzer writes:
>> I don't think so. Section 4.2 says:
>>
>> |                    Formally, a reply is defined to be the
>> sequence: a |  three-digit code, <SP>, one line of text, and
>> <CRLF>, or a multiline |  reply (as defined in section
>> 4.2.1).  Since, in violation of this |  specification, the
>> text is sometimes not sent, clients which do not |  receive
>> it SHOULD be prepared to process the code alone (with or |
>> without a trailing space character).
>>
>> To me that means that "250"CRLF is not permitted, but a
>> frequent bug  which clients should handle gracefully.
>
> 2821 has this grammar:
>
>        Reply-line = Reply-code [ SP text ] CRLF
>
> draft-klensin has this:
>
>     Reply-line     = *( Reply-code "-" [ text ] CRLF )
>                    Reply-code [ SP text ] CRLF
>
> The text you quite indicates "Reply-code SP [ text ] CRLF", at
> least when I read it, but the grammar puts the SP inside the
> optional part in both cases.

I hope that the ABNF revision effort with resolve any 
differences.  However, in 2821 as in 821, the grammar is 
illustrative rather than definitive.  The text is definitive.

So...

Peter is correct: "250"CRLF (and "250 "CRLF) are in violation of 
the spec, but clients are expected to handle it as if they had 
received
   "250 random-string"CRLF

If one wants the ABNF to be normative and to accurately cover 
all of the corner cases, the 2821 syntax and the 2821bis fix are 
a little buggy.

    john

Date: Tue, 12 Jun 2007 22:40:44 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: 2821bis: three-digit-and-crlf replies
Cc: "Peter J. Holzer" <[email-masked]>
Sender: [email-masked]

Forget it. I'm happy with the text.

(I was looking too much at the grammar in isolation.)

Arnt

Date: Tue, 12 Jun 2007 22:15:54 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: 2821bis: three-digit-and-crlf replies
Cc: "Peter J. Holzer" <[email-masked]>
Sender: [email-masked]

Peter J. Holzer writes:
> I don't think so. Section 4.2 says:
>
> |                    Formally, a reply is defined to be the sequence: a
> |  three-digit code, <SP>, one line of text, and <CRLF>, or a multiline
> |  reply (as defined in section 4.2.1).  Since, in violation of this
> |  specification, the text is sometimes not sent, clients which do not
> |  receive it SHOULD be prepared to process the code alone (with or
> |  without a trailing space character).
>
> To me that means that "250"CRLF is not permitted, but a frequent bug 
> which clients should handle gracefully.

2821 has this grammar:

       Reply-line = Reply-code [ SP text ] CRLF

draft-klensin has this:

    Reply-line     = *( Reply-code "-" [ text ] CRLF )
                   Reply-code [ SP text ] CRLF

The text you quite indicates "Reply-code SP [ text ] CRLF", at least 
when I read it, but the grammar puts the SP inside the optional part in 
both cases.

Arnt

Date: Tue, 12 Jun 2007 21:46:13 +0200
From: "Peter J. Holzer" <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: Re: 2821bis: three-digit-and-crlf replies
Mail-Followup-To: SMTP Interest Group <[email-masked]>
Sender: [email-masked]

--7cm2iqirTL37Ot+N
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2007-06-12 18:38:44 +0200, Arnt Gulbrandsen wrote:
> "an SMTP response consists of zero or more lines which start with a=20
> three-digit number and a hyphen, and then exactly one line which starts=
=20
> with a three-digit number and a space". How many people think that's=20
> correct?
>=20
> 2821 says the space on the final line is optional. "250" CRLF is permitte=
d.

I don't think so. Section 4.2 says:

|                    Formally, a reply is defined to be the sequence: a
|  three-digit code, <SP>, one line of text, and <CRLF>, or a multiline
|  reply (as defined in section 4.2.1).  Since, in violation of this
|  specification, the text is sometimes not sent, clients which do not
|  receive it SHOULD be prepared to process the code alone (with or
|  without a trailing space character).=20

To me that means that "250"CRLF is not permitted, but a frequent bug
which clients should handle gracefully.

draft-klensin-rfc2821bis-04 contains the same two sentences.

> Since every SMTP server I've ever seen sends the space and some text, it=
=20
> seems reasonable for 2821bis to say something like "servers SHOULD send=
=20
> accompanying text, not just the three-digit response code".

The second sentence might be removed if this "violation of this
specification" is now so infrequent as to be negligible (I also cannot
think any server which returns only bare return codes, but that's hardly
evidence). OTOH, urging implementors to caution never seems wrong to me.

	hp

--=20
   _  | Peter J. Holzer    | I know I'd be respectful of a pirate=20
|_|_) | Sysadmin WSR       | with an emu on his shoulder.
| |   | [email-masked]         |
__/   | http://www.hjp.at/ |	-- Sam in "Freefall"

--7cm2iqirTL37Ot+N
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

--7cm2iqirTL37Ot+N--

Date: Tue, 12 Jun 2007 18:38:47 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: 2821bis: received: ... for clause
Sender: [email-masked]

Hi,

The 'for' production is incorrect as it stands. "Received: .... for 
[email-masked]@att.com; ..." is legal according to 2821. Oops. There 
are 2-3 ways out of this:
  - constrain 'for' to one address
  - introduce a separator:
    - either FWS
    - or "," FWS.

My preference is to constrain it to one address. I cannot remember 
seeing production softwaret that inserts multiple addresses on purpose, 
so IMO this can be classified as an unused feature and dropped. (I 
checked 280k received fields just now and found none.)

Arnt

Date: Tue, 12 Jun 2007 18:38:44 +0200
From: Arnt Gulbrandsen <[email-masked]>
To: SMTP Interest Group <[email-masked]>
Subject: 2821bis: three-digit-and-crlf replies
Sender: [email-masked]

Hi,

"an SMTP response consists of zero or more lines which start with a 
three-digit number and a hyphen, and then exactly one line which starts 
with a three-digit number and a space". How many people think that's 
correct?

2821 says the space on the final line is optional. "250" CRLF is permitted.

Since every SMTP server I've ever seen sends the space and some text, it 
seems reasonable for 2821bis to say something like "servers SHOULD send 
accompanying text, not just the three-digit response code". I do not 
have a strong opinion on the subject, however. Does anyone else?

Arnt

Date: Wed, 06 Jun 2007 16:01:59 +0100
From: Dave Crocker <[email-masked]>
Reply-To: [email-masked]
Organization: Brandenburg InternetWorking
To: [email-masked]
Subject: [Fwd: Last Call: draft-hutzler-spamops (Email Submission: Access and  Accountability) to BCP]
Sender: [email-masked]

(Apologies if you receive this more than once.  I am sending it to each list
that is relevant to the topic, in order to make sure the community is aware of
the opportunity and need for comment.  /d)

Folks,

The enclosed announcement is solicits comments on "Email Submission:
Access and Accountability" a modest BCP for basic guidance about an aspect of
email service.  (Pretty versions of the document are available at
<http://bbiw.net/recent.html#spamops>.)

The IESG needs to hear comments about the utility of this document, with
respect to the operational guidance it provides.

Please consider posting comments, as suggested in the enclosed note, noting
concrete benefits and/or concerns you see for this document.  (By being
concrete, you show that you have read the document and understand it; that
way, the IESG can know that you are offering an informed opinion...)

Thanks.

d/

-------- Original Message --------
Subject: Last Call: draft-hutzler-spamops (Email Submission: Access and
Accountability) to BCP
Date: Wed, 06 Jun 2007 10:32:13 -0400
From: The IESG <[email-masked]>

The IESG has received a request from an individual submitter to consider
the following document:

- 'Email Submission: Access and Accountability '
    <draft-hutzler-spamops-07.txt> as a BCP

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action.  Please send substantive comments to the
[email-masked] mailing lists by 2007-06-20. Exceptionally,
comments may be sent to [email-masked] instead. In either case, please
retain the beginning of the Subject line to allow automated sorting.

Note that this is a second run of the Last Call. The first Last Call
happened in 2005, and the current version tried to address all the
comments received then. Taking into account the amount of time since the
first Last Call the Area Director and the editors decided to run again a
two weeks Last Call.

The file can be obtained via
http://www.ietf.org/internet-drafts/draft-hutzler-spamops-07.txt

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

From: [email-masked]
Cc: John C Klensin <[email-masked]>, [email-masked]
Date: Fri, 01 Jun 2007 06:16:36 -0700 (PDT)
Subject: Re: rfc2821bis-04, Issue 38: MX queuing behavior with options
To: Tony Finch <[email-masked]>
Sender: [email-masked]

> On Thu, 31 May 2007, John C Klensin wrote:
> >
> > I believe that, with the way the text in 2821 is written, a
> > client that is successful in opening an SMTP session (getting
> > the 220 greeting, sending an EHLO command, and getting a
> > response) to at least one of smtp1 or smtp2 is not permitted to
> > fall back to backup.foo.example under any circumstances.

> No: it says "the SMTP client MUST be able to try each of the relevant
> addresses in this list in order, until a delivery attempt succeeds", so
> after a successfully established connection but a failed (4yz) delivery
> the client will try another server.

Not necessarily, if for no other reason than that some believe there's
a difference between a successful *delivery* and a successful delivery
*attempt*.

As a practical matter the further in to the transaction the faiure occurs the
more time and resources the client has devoted to this one delivery (no doubt
out of many pending) and the less likely that other MXes will succeed where
this one has failed. The question is where to draw the line: Some
implementations do as you say and will retry all MXes in the event of a 4yz
error at any  point, others may give up on receipt of a 4xy as banner greeting
and still others put the cut point somewhere between the beginning and the end.
And there are also some clients that take the type of 4yz error into account.
But since clients don't have a crystal ball to consult this becomes a question
of what are the best heuristics to use.

My personal position is that it definitely makes sense to retry if the failure
occurs before a transaction has started, and retries after message data has
been transferred are more trouble than they are worth. Beyond that my opinion
varies depending on my mood and behavior I've observed recently.

				Ned

Date: Fri, 1 Jun 2007 11:43:30 +0100
From: Tony Finch <[email-masked]>
To: John C Klensin <[email-masked]>
cc: [email-masked]
Subject: Re: rfc2821bis-04, Issue 38: MX queuing behavior with options
Sender: [email-masked]

On Thu, 31 May 2007, John C Klensin wrote:
>
> I believe that, with the way the text in 2821 is written, a
> client that is successful in opening an SMTP session (getting
> the 220 greeting, sending an EHLO command, and getting a
> response) to at least one of smtp1 or smtp2 is not permitted to
> fall back to backup.foo.example under any circumstances.

No: it says "the SMTP client MUST be able to try each of the relevant
addresses in this list in order, until a delivery attempt succeeds", so
after a successfully established connection but a failed (4yz) delivery
the client will try another server.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
DOGGER: EAST OR NORTHEAST 3 OR 4. SLIGHT OR MODERATE. SHOWERS, FOG PATCHES.
MODERATE OR GOOD, OCCASIONALLY VERY POOR.