Subject: Re: UUEncoding File Attachment
From: "Tim Kehres" <[email-masked]>
To: "Russ Allbery" <[email-masked]>, <[email-masked]>
Date: Fri, 29 Jan 2010 12:36:17 +0800
List-Id: <ietf-smtp.imc.org>

> Hector Santos <[email-masked]> writes:
> Getting rid of uuencode for MIME is not change for the sake of change.
> uuencode was a badly broken way of handling attachments in multiple ways.
> I say this as someone who has written his own uuencode and uudecode
> processors and used the protocol for various applications, mail and
> non-mail, over the years.  I'd be happy to never see it again.
>
> It's like many pre-MIME ways of doing things that MIME now has better ways
> of doing, such as some of the Usenet article signature algorithms.  The
> old ways mostly work until you look at them funny or hit some corner case,
> and then they turn into a mess.  It's that lack of attention to detail and
> edge cases that I mean when I say poor engineering.  Most of those systems
> were designed in a much simpler time and were, at the time, following the
> "simplest thing that could possibly work" approach.  MIME is a refactoring
> to deal with all of the interoperability problems that came up in practice
> after using them for many years.

At the time the MIME specifications were being written, I was one of the few 
that was advocating at least a temporary way to handle UUENCODE within the 
MIME framework.   The reason was to allow some greater form of 
interoperability, at least for the short term.

The arguments at the time were precisely what Hector mentions above. 
Co-existence of UUENCODE and Base64 I believe was one of the issues, but I 
am going by memory on this.

Being able to look back on this I have to admit that the correct decision 
was made NOT to include any support at all within the MIME framework from 
UUENCODED attachment types.  Many of the messaging vendors (which my company 
at the time was one of), got around this problem in a very simple way.  When 
a message came in, it was trivial to determine if it was MIME or non-MIME. 
If and only if it was non-MIME would we scan for any apparent UUENCODE 
blocks, thereby getting around potential MIME issues.  In other words, 
having UUENCODE completely outside the MIME framework seems to have worked 
out best.   And from the messages in this thread, it seems that all existing 
remnants of its use continue to be in the  non-MIME context, so all is good. 
:-)

Best Regards,

Tim 

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: UUEncoding File Attachment
Organization: The Eyrie
Date: Thu, 28 Jan 2010 14:48:57 -0800
Sender: [email-masked]

Hector Santos <[email-masked]> writes:

> Don't disagree with you Russ.

> But I guess I still have a compassion for legacy operations and
> support. :)  Over the holidays, as I normally do every year, I go over
> the source code to review with the ideas of "cleaning things up," maybe
> updating a piece of code or module with a new method that is used
> somewhere else perhaps, including deprecating/removing real old
> "things."  Ironically, uuencoding was one of the items reviewed. But as
> it is almost always the case, sudden something reminds you that it is
> probably best to leave things alone.  I really don't care to see reports
> from long time loyal customers who pays their monthly, quarterly or
> annual product maintenance and support fees, but hardly heard from in a
> long time say after an update, "Hey, what the hell happen here?"  No, I
> rather not have that.

Oh, yeah, I completely agree that as a mail service provider, you have to
keep supporting it as long as people keep sending it.  I hadn't realized
that people were still sending it.  My objections are to the people who
are still sending it, not to the people who are dealing with it on the
receiving end.  Be conservative in what you send and liberal in what you
accept and all that.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Thu, 28 Jan 2010 17:27:43 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Russ Allbery <[email-masked]>
CC:  [email-masked]
Subject: Re: UUEncoding File Attachment
Sender: [email-masked]

Don't disagree with you Russ.

But I guess I still have a compassion for legacy operations and 
support. :)  Over the holidays, as I normally do every year, I go over 
the source code to review with the ideas of  "cleaning things up," 
maybe updating a piece of code or module with a new method that is 
used somewhere else perhaps, including deprecating/removing real old 
"things." Ironically, uuencoding was one of the items reviewed. But as 
it is almost always the case, sudden something reminds you that it is 
probably best to leave things alone.  I really don't care to see 
reports from long time loyal customers who pays their monthly, 
quarterly or annual product maintenance and support fees, but hardly 
heard from in a long time say after an update, "Hey, what the hell 
happen here?"  No, I rather not have that.

Thanks

-- 
Sincerely

Hector Santos
http://www.santronics.com

Russ Allbery wrote:

> Hector Santos <[email-masked]> writes:
>> Russ Allbery wrote:
> 
>>> Huh.  I suppose I should thank someone that I've never had to deal with
>>> commercial applications living that many decades in the past.
> 
>> Well, IMO, things don't have to always change just for the sake of
>> "change."
> 
> Getting rid of uuencode for MIME is not change for the sake of change.
> uuencode was a badly broken way of handling attachments in multiple ways.
> I say this as someone who has written his own uuencode and uudecode
> processors and used the protocol for various applications, mail and
> non-mail, over the years.  I'd be happy to never see it again.
> 
> It's like many pre-MIME ways of doing things that MIME now has better ways
> of doing, such as some of the Usenet article signature algorithms.  The
> old ways mostly work until you look at them funny or hit some corner case,
> and then they turn into a mess.  It's that lack of attention to detail and
> edge cases that I mean when I say poor engineering.  Most of those systems
> were designed in a much simpler time and were, at the time, following the
> "simplest thing that could possibly work" approach.  MIME is a refactoring
> to deal with all of the interoperability problems that came up in practice
> after using them for many years.
> 

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: UUEncoding File Attachment
Organization: The Eyrie
Date: Thu, 28 Jan 2010 13:07:32 -0800
Sender: [email-masked]

Hector Santos <[email-masked]> writes:
> Russ Allbery wrote:

>> Huh.  I suppose I should thank someone that I've never had to deal with
>> commercial applications living that many decades in the past.

> Well, IMO, things don't have to always change just for the sake of
> "change."

Getting rid of uuencode for MIME is not change for the sake of change.
uuencode was a badly broken way of handling attachments in multiple ways.
I say this as someone who has written his own uuencode and uudecode
processors and used the protocol for various applications, mail and
non-mail, over the years.  I'd be happy to never see it again.

It's like many pre-MIME ways of doing things that MIME now has better ways
of doing, such as some of the Usenet article signature algorithms.  The
old ways mostly work until you look at them funny or hit some corner case,
and then they turn into a mess.  It's that lack of attention to detail and
edge cases that I mean when I say poor engineering.  Most of those systems
were designed in a much simpler time and were, at the time, following the
"simplest thing that could possibly work" approach.  MIME is a refactoring
to deal with all of the interoperability problems that came up in practice
after using them for many years.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Thu, 28 Jan 2010 15:20:04 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "Carl S. Gutekunst" <[email-masked]>
CC:  [email-masked]
Subject: Re: UUEncoding File Attachment
Sender: [email-masked]

Carl S. Gutekunst wrote:

> 
> Hector Santos wrote:
>>> It was only after a large number of paying Google Apps customers 
>>> complained that support was finally added.
>>
>> Its so incredible that a project team manager would allow that.  What? 
>> He thought the complaints would go away?
> 
> That's just the maturation of Google. Originally, everyone who used 
> Gmail did so for free, and I really doubt many of them were receiving 
> Oracle reports (which are still in uuencode). Today there is a 
> significant number of commercial customers. Priorities change.

Right, but more significant, I think, it is a hitting a wider segment 
of commercial systems.  For example, operators are using a gmail.com 
account as a secondary or primary emergency and system operations 
contacts in case their own system goes down or there is a schedule 
outage, etc.  I just got a CC to my gmail.com account from verizon for 
a february 17 10 min maintenance.  That gmail.com account started a 
"JUNK" account for "junk" signups or aliasing at some forums or list 
to minimize usage of my corporate addresses.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Thu, 28 Jan 2010 11:39:41 -0800
From: "Carl S. Gutekunst" <[email-masked]>
To: Hector Santos <[email-masked]>
CC: [email-masked]
Subject: Re: UUEncoding File Attachment
Sender: [email-masked]

Hector Santos wrote:
>> It was only after a large number of paying Google Apps customers 
>> complained that support was finally added.
>
> Its so incredible that a project team manager would allow that.  What? 
> He thought the complaints would go away?

That's just the maturation of Google. Originally, everyone who used 
Gmail did so for free, and I really doubt many of them were receiving 
Oracle reports (which are still in uuencode). Today there is a 
significant number of commercial customers. Priorities change.

<csg>

Date: Thu, 28 Jan 2010 14:31:07 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Russ Allbery <[email-masked]>
CC:  [email-masked]
Subject: Re: UUEncoding File Attachment
Sender: [email-masked]

Russ Allbery wrote:

> "Carl S. Gutekunst" <[email-masked]> writes:
>> Russ Allbery wrote:
> 
>>> uuencode is still moderately common on Usenet.
> 
>> Mostly for porn, AFAIK.
> 
> Yeah, the copyright violations have mostly switched to yEnc since it has
> less bloat, so it's only the old-school porn posters who still use
> uuencode.
> 
>> Dumping undelimited uuencode "attachments" into a plain E-mail text body
>> is quite common in automated notifications from commercial applications.
> 
> Huh.  I suppose I should thank someone that I've never had to deal with
> commercial applications living that many decades in the past.

Well, IMO, things don't have to always change just for the sake of 
"change."    Sure, new applications entering the market have the 
luxury to pick and choose, but most long time established commercial 
systems do not just change things or deprecate things "just because" 
and frankly, I think it would be kinda "dumb" to drop support for 
something even if you might not always see it all the time.

But inevitable, in the mail world, once a new package (like gmail) 
reaches of level of penetration that reaches a wider server market at 
all levels, as did GMAIL, it will begin to see interoperability 
issues, as GMAIL did, and eventually has no choice but to "come 
around" to its senses.

To me, its a QA issue more than just technical.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Thu, 28 Jan 2010 14:21:11 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: "Carl S. Gutekunst" <[email-masked]>
CC:  [email-masked]
Subject: Re: UUEncoding File Attachment
Sender: [email-masked]

Carl S. Gutekunst wrote:

> 
> Hector Santos wrote:
>> I know this is an old method for file attachments, but I would be 
>> interesting to know the extent of existing systems have decided to 
>> pull support for uuencoded file attachment at the inbound side.
>>
>> I ask because I came around at least one system, GMAIL.COM, who does 
>> not support uuencoded file attachments in their WEB interface.
> 
> Gmail has supported uuencoded inbound attachments for several months. 
> Originally it did not, and while multiple end users requested support, 
> the development team refused, considering uuencode to be an obsolete 
> format and not worth the trouble to implement and support. It was only 
> after a large number of paying Google Apps customers complained that 
> support was finally added.

Its so incredible that a project team manager would allow that.  What? 
He thought the complaints would go away?

I just finished an exchange with a GMAIL person and found it does 
support it.   What was not clear is what rules were used as we found 
Thunderbird didn't like certain messages.   Begin 444, Begin 777, 
single or two quotes,  ie.

     ''
     end

The guy noted that if TBIRD found MIME-Version header it didn't 
support uuencoded attachments.  That I would loved to test/confirm 
later on.

-- 
Sincerely

Hector Santos
http://www.santronics.com

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: UUEncoding File Attachment
Organization: The Eyrie
Date: Thu, 28 Jan 2010 10:51:28 -0800
Sender: [email-masked]

"Carl S. Gutekunst" <[email-masked]> writes:
> Russ Allbery wrote:

>> uuencode is still moderately common on Usenet.

> Mostly for porn, AFAIK.

Yeah, the copyright violations have mostly switched to yEnc since it has
less bloat, so it's only the old-school porn posters who still use
uuencode.

> Dumping undelimited uuencode "attachments" into a plain E-mail text body
> is quite common in automated notifications from commercial applications.

Huh.  I suppose I should thank someone that I've never had to deal with
commercial applications living that many decades in the past.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Thu, 28 Jan 2010 10:41:05 -0800
From: "Carl S. Gutekunst" <[email-masked]>
To: Russ Allbery <[email-masked]>
CC: [email-masked]
Subject: Re: UUEncoding File Attachment
Sender: [email-masked]

Russ Allbery wrote:
> uuencode is still moderately common on Usenet.

Mostly for porn, AFAIK.

> I've not seen a uuencoded file in e-mail for years.

Dumping undelimited uuencode "attachments" into a plain E-mail text body 
is quite common in automated notifications from commercial applications.

The heuristic is indeed to look for a correct formatted "begin" line, 
and the decode what follows as long as it's correct format. Ugly indeed, 
but nearly every E-mail client I'm aware of still supports it. (And, as 
noted earlier, Gmail does now, too.)

<csg>

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: UUEncoding File Attachment
Organization: The Eyrie
Date: Thu, 28 Jan 2010 10:34:09 -0800
Sender: [email-masked]

Hector Santos <[email-masked]> writes:

> I know this is an old method for file attachments, but I would be
> interesting to know the extent of existing systems have decided to pull
> support for uuencoded file attachment at the inbound side.

> I ask because I came around at least one system, GMAIL.COM, who does not
> support uuencoded file attachments in their WEB interface.

Do you really mean a uuencoded file *attachment*, in the MIME sense, or do
you mean the historic uuencode behavior of just dumping the uuencode
output into the undelimited body of the message?  I ask because I've never
seen the former, but that's what "attachment" means in the world of
e-mail.

If you mean the latter, there's a fundamental problem here that MIME
solves, namely that it's impossible to know for certain that you're
dealing with a uuencoded file.  You can make a guess based on a line
starting with "begin " followed by some lines that look encoded, but it's
a pretty ugly heuristic and isn't reliable engineering.

uuencode is still moderately common on Usenet.  I've not seen a uuencoded
file in e-mail for years.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Thu, 28 Jan 2010 10:17:20 -0800
From: "Carl S. Gutekunst" <[email-masked]>
To: Hector Santos <[email-masked]>
CC: [email-masked]
Subject: Re: UUEncoding File Attachment
Sender: [email-masked]

Hector Santos wrote:
> I know this is an old method for file attachments, but I would be 
> interesting to know the extent of existing systems have decided to 
> pull support for uuencoded file attachment at the inbound side.
>
> I ask because I came around at least one system, GMAIL.COM, who does 
> not support uuencoded file attachments in their WEB interface.

Gmail has supported uuencoded inbound attachments for several months. 
Originally it did not, and while multiple end users requested support, 
the development team refused, considering uuencode to be an obsolete 
format and not worth the trouble to implement and support. It was only 
after a large number of paying Google Apps customers complained that 
support was finally added.

<csg>

Date: Thu, 28 Jan 2010 09:24:33 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To:  [email-masked]
Subject: UUEncoding File Attachment
Sender: [email-masked]

I know this is an old method for file attachments, but I would be 
interesting to know the extent of existing systems have decided to 
pull support for uuencoded file attachment at the inbound side.

I ask because I came around at least one system, GMAIL.COM, who does 
not support uuencoded file attachments in their WEB interface.

We still have 3rd party mail bot scripts use this format and I am 
wondering if our SMTP router to help the process by converted it to 
MIME based file attachments.

Did your software take the same action GMAIL.COM has done?

I think this is something that just feel thru the crack at GMAIL.COM 
as I can't imagine a system not being liberal in acceptance old and 
new file attachment systems, but conservative in what they send out 
using MIME.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Tue, 12 Jan 2010 12:45:17 +0000
From: Tony Finch <[email-masked]>
To: Cyrus Daboo <[email-masked]>
cc: Alexey Melnikov <[email-masked]>, John C Klensin <[email-masked]>, [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

On Thu, 7 Jan 2010, Cyrus Daboo wrote:
>
> The SRV solution is simple to implement - most OS network libraries now
> provide simple access to SRV record results. Site and domain admins
> these days are much more comfortable with setting up SRV (required for
> services like XMPP). So I think SRV represents a simple solution that
> can be adopted in a reasonable timeframe.

I have a stronger argument for the (potential) simplicity of SRV records:

I've started writing a draft describing Outlook-style MUA configuration
autoguessing. (I'm afraid the meat of it is still in my head, but once I
have it salted and packed in XML it should become available as
draft-fanf-mua-autoguess.) The exercise of describing all the probes that
an MUA needs to perform and all the awkward cases it has to be aware of
has made me realise that SRV records can make autoconfiguration MUCH
simpler for the MUA author. (The issue of APIs for SRV lookup is trivial
in comparison.)

The key point is that at the moment an autoguessing MUA cannot assume any
co-operation from the site operator, whereas we can require that the
existence of MUA SRV records on a mail domain implies a certain service
profile. This can massively reduce the amount of work that the MUA must
do to configure itself.

For example, the MUA can assume that the IMAP and POP services are
actually related to the mail domain; it can ignore imaps and pop3s and
port 25; it can assume that TLS is required; it can assume that the same
login credentials will work for POP and IMAP and message submission; none
of which are true when autoguessing.

We could perhaps guarantee that if the service operator publishes SRV
records for both IMAP and POP and both are supported by the MUA, then
everyone agrees that IMAP is the one to use.

Maybe we could also eliminate the guesswork needed to work out the user's
login credentials. Outlook tries both the whole email address and the
local part, but this can fail if (say) the user has a "friendly name"
email address (e.g. [email-masked]) which doesn't match their
login name (e.g. fanf2). Perhaps we could require that the user's email
address is a valid login name even if it is an alias of this kind.

This kind of guaranteed service profile provides a fairly strong
motivation for MUA authors to implement SRV-based autoconfiguration, at
the expense of stricter requirements on email service operators. But I
think most of the requirements are reasonable and many sites already
follow them, so probably all that a site has to do is buy a new TLS
certificate and publish a couple of SRV records. (Though I don't know if
there's decent support for the TLS server name extension.)

The only tricky requirement is email addresses as login names. If we add
that guarantee then the MUA does not need to make any trial connections to
get a correct configuration. However it is probably too much of a burden
on email service operators.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

From: Alfred =?hp-roman8?B?SM5uZXM=?= <[email-masked]>
Subject: Re: draft-daboo-srv-email and 'creative use' of DNS SRV
To: [email-masked]
Date: Sun, 10 Jan 2010 21:39:10 +0100 (MEZ)
Sender: [email-masked]

Ned Freed wrote in the message archived at
  <http://www.IMC.ORG/ietf-smtp/mail-archive/msg05917.html>:

> ...
>
> But even this isn't really necessary to specify precedence among
> protocols.  The specification could say that pop/pops/imap/imaps
> records should be considered as a group, and absent client
> configuration to the contrary should be used in the order specified
> by their individual priority fields.  For example, an imaps record
> with a higher priority than any imap record would indicate imaps
> is preferred over imap.
>
> RFC 2782 appears to allow schemes like this.  But whether it's worth
> defining such a thing is another matter.  My personal opinion is that
> it's just a bit outside my comfort zone in terms of complexity, but if
> others feel strongly that servers should be able to specify protocol
> preferences this is a way to do with it with no additional DNS records
> or supplementary protocols.

Folks,
please take a look at the draft Olafur Gudmundsson and I recently
have posted to clarify RFC 2782 (and hence intended for PS),
  http://tools.ietf.org/html/draft-gudmundsson-dnsext-srv-clarify-00

According to these clarifications (if approved), such specific rules,
*if and only if unambiguously specified* would IMO well be possible.
Absent such application-specific documentation, the standard SRV
naming and usage rules apply.  Cyrus Daboo's draft would likely be
the proper place to incorporate such specific rules, if desired.

Kind regards
  Alfred.

-- 

+------------------------+--------------------------------------------+
| TR-Sys Alfred Hoenes   |  Alfred Hoenes   Dipl.-Math., Dipl.-Phys.  |
| Gerlinger Strasse 12   |  Phone: (+49)7156/9635-0, Fax: -18         |
| D-71254  Ditzingen     |  E-Mail:  [email-masked]                     |
+------------------------+--------------------------------------------+

From: [email-masked]
Cc: [email-masked]
Date: Sat, 09 Jan 2010 09:13:02 -0800 (PST)
Subject: Re: draft-daboo-srv-email
To: Willie Gillespie <[email-masked]>
Sender: [email-masked]

> I thought I saw some discussion on this list about which option an MUA
> should choose if it was presented with IMAP, POP3, POP3S, or IMAPS.

> While, ultimately this choice is the user's, perhaps a
> _mail._tcp.example.com SRV record could be used.  The response could
> state simply "imaps" or something which would help the MUA pick a
> default option.  All the other SRV records would be used as suggested
> still.  It would just help in the auto-configuration of an MUA where a
> user doesn't know which option to pick.

This is actually can do more than provide a single default - you could have
multiple records with this prefix and use the priority fields to specify
the preference order for protocol selection.

But even this isn't really necessary to specify precedence among protocols. The
specification could say that pop/pops/imap/imaps records should be considered
as a group, and absent client configuration to the contrary should be used in
the order specified by their individual priority fields. For example, an imaps
record with a higher priority than any imap record would indicate imaps is
preferred over imap.

RFC 2782 appears to allow schemes like this. But whether it's worth defining
such a thing is another matter. My personal opinion is that it's just a bit
outside my comfort zone in terms of complexity, but if others feel strongly
that servers should be able to specify protocol preferences this is a way to do
with it with no additional DNS records or supplementary protocols.

				Ned

From: [email-masked]
Cc: [email-masked], Russ Allbery <[email-masked]>, [email-masked]
Date: Sat, 09 Jan 2010 08:37:16 -0800 (PST)
Subject: Re: draft-daboo-srv-email
To: Tony Finch <[email-masked]>
Sender: [email-masked]

> On Thu, 7 Jan 2010, [email-masked] wrote:
> >
> > > I agree with Cyrus.  Even if the market has reached a workable compromise
> > > for the time being by using well-known host names, it has done so at the
> > > cost of flexibility which SRV records could provide.  I think it would be
> > > very useful to document the naming conventions that will allow current
> > > clients to find the servers, but I don't believe that's a reason to *not*
> > > document a more robust and formal SRV record protocol as well in the hope
> > > that clients and configurations will move to that.
> >
> > Exactly right IMO.

> Thanks for all your arguments. I have been convinced. Perhaps I should be
> less negative about the likelihood of deploying improvements...

> > I also fear that an effort to do something more elaborate with XML and web
> > servers and so on carries with it a very high liklihood of failure, and even if
> > we manage to agree on something (probably at some distant point in the future),
> > it will then fail to deploy.

> Actually I'm less negative about this because it has clearer benefits.

My concern arises not from a lack of benefit - the benefits here are pretty
clear - but from the fact that when you go this far you're necessarily engaging
in a schema design exercise.

The past history of schema design work in the IETF has not been good. Partial
or even total failures are pretty common, and even when such efforts succeed
they have tended to take far more time than anyone expected them to.

> For
> example it allows an operator to override an MUA's built-in preferences.
> The big question is whether we can get the various MUA vendors to
> co-operate instead of continuing to do their own thing.

Yes, that's also an issue.

				Ned

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: draft-daboo-srv-email
Organization: The Eyrie
Date: Fri, 08 Jan 2010 13:57:52 -0800
Sender: [email-masked]

Hector Santos <[email-masked]> writes:
> Russ Allbery wrote:

>> As a site where, for various load-balancing, administrative, and
>> configuration complexity reasons, our outgoing mail servers for clients
>> are entirely distinct from the inbound servers holding the MX records
>> of the domain, I would love to see a SRV record for clients to locate
>> outbound SMTP servers.

> I missed section 3.1 which offers the _submission._tcp.domain SRV query.

> Don't know if that covers, partially or otherwise, the user needs on your
> end.

Ah, sorry, I also didn't check.  I think that covers it.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Fri, 08 Jan 2010 14:26:24 -0700
From: Willie Gillespie <[email-masked]>
Organization: Engineering System Solutions
To: [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

This is a cryptographically signed message in MIME format.

--------------ms060206070901060502070208
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

I thought I saw some discussion on this list about which option an MUA
should choose if it was presented with IMAP, POP3, POP3S, or IMAPS.

While, ultimately this choice is the user's, perhaps a
_mail._tcp.example.com SRV record could be used.  The response could
state simply "imaps" or something which would help the MUA pick a
default option.  All the other SRV records would be used as suggested
still.  It would just help in the auto-configuration of an MUA where a
user doesn't know which option to pick.

Just throwing the idea out there.  Take it or leave it, doesn't really
bother me.

Willie Gillespie

--------------ms060206070901060502070208
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

--------------ms060206070901060502070208--

Date: Fri, 08 Jan 2010 14:01:28 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Russ Allbery <[email-masked]>
CC:  [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

Russ Allbery wrote:

> Hector Santos <[email-masked]> writes:
> 
>> 2) What about outgoing?
> 
>> Are MX records considered the solution?
> 
>> Using GMAIL as an example, I think it only allows IMAP for receiving and
>> SUBMIT (port 587) for sending.  So wouldn't a SRV record be useful for
>> the case where SMTP is used for outgoing when the incoming uses IMAP?
> 
> As a site where, for various load-balancing, administrative, and
> configuration complexity reasons, our outgoing mail servers for clients
> are entirely distinct from the inbound servers holding the MX records of
> the domain, I would love to see a SRV record for clients to locate
> outbound SMTP servers.

I missed section 3.1 which offers the _submission._tcp.domain SRV query.

Don't know if that covers, partially or otherwise, the user needs on 
your end.

The only thing I see is that normally, the posting host can either be 
specific (i.e, "smart host") or MX lookup.  But I think it is MUA 
specific too.

A quick review of how TBIRD does it for "Create New Account", it will 
use the "default" smtp server setup prepared for the account settings.
So the outgoing part is probably best useful when the user is first 
installing his/her MUA.

Also, in terms of a "USER" need, the odds are good his ISP is blocking 
port 25 for anything other than the ISP email network. So for non ISP 
networks, port 587 "SHOULD" be a preferred selection if offered by the 
mail host.

In my experience, and I helped the TBIRD guys get this added, this is 
more important in the growing SOHO or wireless home network with 
additional user machines are used.

For SMTP receivers who are "strict" with the EHLO domain literal IP 
checking, this can be a problem when the MUA uses the user machine IP 
and not the NAT or public IP address for the user.

OE does not have this problem because it will use the netbios computer 
name.  TBIRD will use a bracket IP of the user' machine.

So I had the TBIRD people address this and the final simple/quick 
solution was an Advanced Configuration option (rather than add a new 
DIALOG) that allows the EHLO string to be defined.

       Tools | General | Config Editor

Type the filter

     mail.server

to see all the mail server fields.  I don't recall if I had to add te 
first time (in the first beta) and its now added automatically, but 
there would be fields called

      mail.server.server#.hostname

and if defined, it will be used as the override for the SMTP going 
transaction EHLO command.  Otherwise, it will use the FQDN for a 
public address or the private ip address of the user's machine.

For example from my SOHO machine I have my TBIRD 
mail.server.server#.hostname fields set using

          [public IP address]

Finally, before this feature was added to TBIRD, because I was using a 
SUBMIT server it may have a tight EHLO checking.  I proposed to 
Klensin and Gellens (RFC 4409) that they consider adding some notes 
for relaxing EHLO IP literal checking  because SUBMIT requires 
authentication anyway which is handled after the EHLO state.

So for our WCSMTP software, we added an option (enabled by default) to 
skip EHLO checking for PORT 587 connections based on the idea that 
authentication is required anyway, not an option like in public port 
25. That was our solution before the TBIRD changes were made.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 08 Jan 2010 13:28:18 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Cyrus Daboo <[email-masked]>
CC:  [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

Cyrus Daboo wrote:

>> 2) What about outgoing?
>>
>> Are MX records considered the solution?
>>
>> Using GMAIL as an example, I think it only allows IMAP for receiving and
>> SUBMIT (port 587) for sending.  So wouldn't a SRV record be useful for
>> the case where SMTP is used for outgoing when the incoming uses IMAP?
> 
> The current spec defines _submission._tcp in Section 3.1 which covers 
> email client submission server discovery.

Sorry, I missed that section (short section, page cut off).

Ok, so overall, a suggested logic would be:

   - Query for _pop3s, _imaps, _pop3, _imap
     (order implementation based)

   - If more than one, offer user selection.

   - Query for _submission

I think this is a simple protocol logic for implementation. Anything 
beyond that is specific to the MUA client.

Regarding SRV in general, I can provide some  personal experience. 
You might consider some of these as "encouragement" material for any 
draft update.

Until sometime in 2008, we used an NT 4.0 domain controller and DNS 
server and never had a need to upgrade. The machine served its 
isolated purpose. Hotfixes were provided per MS request but that ended 
in 2008 and last year's DNS "DoS Exploit of the year" forced the issue 
to update the DNS server.  The DNS server was moved to a Windows 2003 
enterprise box.  That move solved the security issue and also allowed 
us to use SRV records to improve our jabber (XMPP) server discovery 
process.

So I think past reasons for not using SRV are passe. At least within 
the legacy windows world, SRV records are more applicable today from a 
support standpoint because any remaining DNS 4.0 server can no longer 
operate safely and are forced to upgrade thus offering SRV query support.

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Fri, 08 Jan 2010 11:16:55 -0500
From: Cyrus Daboo <[email-masked]>
To: Hector Santos <[email-masked]>
cc: [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

Hi Hector,

--On January 8, 2010 12:27:12 AM -0500 Hector Santos 
<[email-masked]> wrote:

> This draft appears like a practical and simple proposal.  I just added a
> SRV record for our winserver.com email domain.
>
> _pop3._tcp              SRV	0 1 110	pop3.winserver.com.

OK, great!

> A few comments:
>
> 1) Mail Pickup
>
> I guess overall if I read the specs right, a MUA would do at least four
> (4) lookups?  Assuming, there is a client design preference to checked
> for a secured host first, the order would be:
>
>      _pop3s._tcp.winserver.com
>      _imap3s._tcp.winserver.com
>      _pop3._tcp.winserver.com
>      _imap._tcp.winserver.com
>
> And no decision can be made until all four are determined to allow for
> user selection idea. IOW, four queries MUST be made.  Since this would
> normally be a one time thing, it doesn't present any high volume DNS
> overhead issue. +1.

I believe DNS allows a server to return more records than were originally 
requested. What we could do is suggest that DNS servers return all the 
email related SRV record types defined if anyone of them is requested. 
However, that would require specific changes to the DNS servers to work so 
may be a non-starter.

> 2) What about outgoing?
>
> Are MX records considered the solution?
>
> Using GMAIL as an example, I think it only allows IMAP for receiving and
> SUBMIT (port 587) for sending.  So wouldn't a SRV record be useful for
> the case where SMTP is used for outgoing when the incoming uses IMAP?

The current spec defines _submission._tcp in Section 3.1 which covers email 
client submission server discovery.

-- 
Cyrus Daboo

Date: Fri, 08 Jan 2010 10:16:18 -0500
From: John C Klensin <[email-masked]>
To: Tony Finch <[email-masked]>, [email-masked]
cc: Russ Allbery <[email-masked]>, [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

Agree on both issues -- very convincing arguments for going
ahead with SRV, but, as we find ourselves moving back into a
world in which it is common for people to use multiple devices
to access the same mailbox, I think it would be worth the
investment to again see if we can make progress on
properly-secured, client-device-independent, portable full
configuration information.

   john

--On Friday, January 08, 2010 14:17 +0000 Tony Finch
<[email-masked]> wrote:

> 
> On Thu, 7 Jan 2010, [email-masked] wrote:
>> 
>> > I agree with Cyrus.  Even if the market has reached a
>> > workable compromise for the time being by using well-known
>> > host names, it has done so at the cost of flexibility which
>> > SRV records could provide.  I think it would be very useful
>> > to document the naming conventions that will allow current
>> > clients to find the servers, but I don't believe that's a
>> > reason to *not* document a more robust and formal SRV
>> > record protocol as well in the hope that clients and
>> > configurations will move to that.
>> 
>> Exactly right IMO.
> 
> Thanks for all your arguments. I have been convinced. Perhaps
> I should be less negative about the likelihood of deploying
> improvements...
> 
>> I also fear that an effort to do something more elaborate
>> with XML and web servers and so on carries with it a very
>> high liklihood of failure, and even if we manage to agree on
>> something (probably at some distant point in the future), it
>> will then fail to deploy.
> 
> Actually I'm less negative about this because it has clearer
> benefits. For example it allows an operator to override an
> MUA's built-in preferences. The big question is whether we can
> get the various MUA vendors to co-operate instead of
> continuing to do their own thing.
> 
> Tony.

Date: Fri, 8 Jan 2010 14:17:37 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
cc: Russ Allbery <[email-masked]>, [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

On Thu, 7 Jan 2010, [email-masked] wrote:
>
> > I agree with Cyrus.  Even if the market has reached a workable compromise
> > for the time being by using well-known host names, it has done so at the
> > cost of flexibility which SRV records could provide.  I think it would be
> > very useful to document the naming conventions that will allow current
> > clients to find the servers, but I don't believe that's a reason to *not*
> > document a more robust and formal SRV record protocol as well in the hope
> > that clients and configurations will move to that.
>
> Exactly right IMO.

Thanks for all your arguments. I have been convinced. Perhaps I should be
less negative about the likelihood of deploying improvements...

> I also fear that an effort to do something more elaborate with XML and web
> servers and so on carries with it a very high liklihood of failure, and even if
> we manage to agree on something (probably at some distant point in the future),
> it will then fail to deploy.

Actually I'm less negative about this because it has clearer benefits. For
example it allows an operator to override an MUA's built-in preferences.
The big question is whether we can get the various MUA vendors to
co-operate instead of continuing to do their own thing.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: draft-daboo-srv-email
Organization: The Eyrie
Date: Thu, 07 Jan 2010 22:56:17 -0800
Sender: [email-masked]

Hector Santos <[email-masked]> writes:

> 2) What about outgoing?

> Are MX records considered the solution?

> Using GMAIL as an example, I think it only allows IMAP for receiving and
> SUBMIT (port 587) for sending.  So wouldn't a SRV record be useful for
> the case where SMTP is used for outgoing when the incoming uses IMAP?

As a site where, for various load-balancing, administrative, and
configuration complexity reasons, our outgoing mail servers for clients
are entirely distinct from the inbound servers holding the MX records of
the domain, I would love to see a SRV record for clients to locate
outbound SMTP servers.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Fri, 08 Jan 2010 00:27:12 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Cyrus Daboo <[email-masked]>
CC:  [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

Cyrus Daboo wrote:

> 
> Hi Tony,
> 
> --On January 5, 2010 6:49:16 PM +0000 Tony Finch <[email-masked]> wrote:
> 
>> A few comments on draft-daboo-srv-email:
> 
> Just to update everyone on the status of this draft - there are a whole 
> bunch of comments pending from the original review of this last year. I 
> have been waiting for the SRV registry spec to be updated following the 
> heated debates at the end of last year which came to resolution at the 
> meeting in Japan. Once that spec is out I will revise srv-email 
> document. The other major point was having srv-email say very little 
> itself about certificate verification, and instead refer to 
> <http://tools.ietf.org/html/draft-saintandre-tls-server-id-check-02>. It 
> would be nice if that spec could have an explicit example of 
> verification where SRV was used to get the server hostname.
> 

This draft appears like a practical and simple proposal.  I just added 
a SRV record for our winserver.com email domain.

_pop3._tcp              SRV	0 1 110	pop3.winserver.com.

A few comments:

1) Mail Pickup

I guess overall if I read the specs right, a MUA would do at least 
four (4) lookups?  Assuming, there is a client design preference to 
checked for a secured host first, the order would be:

     _pop3s._tcp.winserver.com
     _imap3s._tcp.winserver.com
     _pop3._tcp.winserver.com
     _imap._tcp.winserver.com

And no decision can be made until all four are determined to allow for 
user selection idea. IOW, four queries MUST be made.  Since this would 
normally be a one time thing, it doesn't present any high volume DNS 
overhead issue. +1.

2) What about outgoing?

Are MX records considered the solution?

Using GMAIL as an example, I think it only allows IMAP for receiving 
and SUBMIT (port 587) for sending.  So wouldn't a SRV record be useful 
for the case where SMTP is used for outgoing when the incoming uses IMAP?

-- 
Sincerely

Hector Santos
http://www.santronics.com

From: [email-masked]
Cc: [email-masked]
Date: Thu, 07 Jan 2010 11:51:26 -0800 (PST)
Subject: Re: draft-daboo-srv-email
To: Russ Allbery <[email-masked]>
Sender: [email-masked]

> Cyrus Daboo <[email-masked]> writes:

> > I disagree. Just because clients have been forced to do things using
> > guesswork up to now does not mean we should leave them and any new
> > clients to do the same thing. I have heard from at least one developer
> > who is working on implementing SRV support for Thunderbird.

> > The SRV solution is simple to implement - most OS network libraries now
> > provide simple access to SRV record results. Site and domain admins
> > these days are much more comfortable with setting up SRV (required for
> > services like XMPP). So I think SRV represents a simple solution that
> > can be adopted in a reasonable timeframe.

> I agree with Cyrus.  Even if the market has reached a workable compromise
> for the time being by using well-known host names, it has done so at the
> cost of flexibility which SRV records could provide.  I think it would be
> very useful to document the naming conventions that will allow current
> clients to find the servers, but I don't believe that's a reason to *not*
> document a more robust and formal SRV record protocol as well in the hope
> that clients and configurations will move to that.

Exactly right IMO.

I also fear that an effort to do something more elaborate with XML and web
servers and so on carries with it a very high liklihood of failure, and even if
we manage to agree on something (probably at some distant point in the future), 
it will then fail to deploy.

So let's get the SRV mechanism in place. If we then want to work on something
more elaborate, either as a separate thing or as an enhancement to SRV, that's
fine to.

				Ned

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: draft-daboo-srv-email
Organization: The Eyrie
Date: Thu, 07 Jan 2010 11:21:19 -0800
Sender: [email-masked]

Cyrus Daboo <[email-masked]> writes:

> I disagree. Just because clients have been forced to do things using
> guesswork up to now does not mean we should leave them and any new
> clients to do the same thing. I have heard from at least one developer
> who is working on implementing SRV support for Thunderbird.

> The SRV solution is simple to implement - most OS network libraries now
> provide simple access to SRV record results. Site and domain admins
> these days are much more comfortable with setting up SRV (required for
> services like XMPP). So I think SRV represents a simple solution that
> can be adopted in a reasonable timeframe.

I agree with Cyrus.  Even if the market has reached a workable compromise
for the time being by using well-known host names, it has done so at the
cost of flexibility which SRV records could provide.  I think it would be
very useful to document the naming conventions that will allow current
clients to find the servers, but I don't believe that's a reason to *not*
document a more robust and formal SRV record protocol as well in the hope
that clients and configurations will move to that.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

Date: Thu, 7 Jan 2010 16:37:51 +0000
From: Tony Finch <[email-masked]>
To: Cyrus Daboo <[email-masked]>
cc: John C Klensin <[email-masked]>, Alexey Melnikov <[email-masked]>, [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

On Thu, 7 Jan 2010, Cyrus Daboo wrote:
>
> > You need a layer of indirection between the mail domain and the web
> > server, e.g. SRV or NAPTR, to accommodate sites like cam.ac.uk which do
> > not have a web server on their zone apex, or customers who delegate email
> > service to a service provider.
>
> Why? Just setup an HTTP server at the domain that is used for email addresses
> if you want this sort of simple configuration to work.

There may be other operational or political considerations that make it a
lot more complicated than "just".

I must say that although the Outlook 2007 autodiscover protocol is
disgusting, they have at least covered a lot of operationally difficult
cases.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

Date: Thu, 07 Jan 2010 11:31:58 -0500
From: Cyrus Daboo <[email-masked]>
To: Tony Finch <[email-masked]>
cc: John C Klensin <[email-masked]>, Alexey Melnikov <[email-masked]>, [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

Hi Tony,

--On January 7, 2010 3:54:58 PM +0000 Tony Finch <[email-masked]> wrote:

>> Frankly something akin to XRDS/Yadis (XML configuration file on an HTTP
>> server) would suffice (with perhaps something like XCAP in play too).
>> Couple that with the new .well-known URI mechanism
>> (<http://tools.ietf.org/html/draft-nottingham-site-meta-05>) and finding
>> a service would be as simple as doing an HTTP request on the primary
>> domain with request-URI .well-known/XXX.
>
> You need a layer of indirection between the mail domain and the web
> server, e.g. SRV or NAPTR, to accommodate sites like cam.ac.uk which do
> not have a web server on their zone apex, or customers who delegate email
> service to a service provider.

Why? Just setup an HTTP server at the domain that is used for email 
addresses if you want this sort of simple configuration to work. If one is 
there already just have it support .well-known or redirect to a server that 
can support it.

>> In fact I have been thinking that one option here is to register
>> .well-known/srv and then any of the registered SRV service types could
>> have meta-data stored underneath that. e.g.:
>>
>> .well-known/srv/imap
>> .well-known/srv/submission
>>
>> etc
>
> MUA configuration should be in one file. If you split it up by protocol
> then it is too easy to perpetuate problems with POP vs. IMAP
> prioritization.

Well the XML data for imap, pop3 and submission could list all the 
available mail services in addition to the actual service requested so that 
clients would only need to do HTTP request. Or we could just define 
.well-known/mail-service and have it all there.

>> If data needs to varying on a per-user basis, then defining a simple
>> RESTful api to include the userid in the request would be easy.
>
> The MUA doesn't have a userid at this point: it only knows an email
> address.

By userid I meant whatever identifier the user is expected to enter in 
their client. If something like Kerberos is in use, users may not even have 
to enter anything.

-- 
Cyrus Daboo

Date: Thu, 7 Jan 2010 15:54:58 +0000
From: Tony Finch <[email-masked]>
To: Cyrus Daboo <[email-masked]>
cc: John C Klensin <[email-masked]>, Alexey Melnikov <[email-masked]>, [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

On Thu, 7 Jan 2010, Cyrus Daboo wrote:
>
> ACAP is overkill for what is needed here.

Yes.

> Frankly something akin to XRDS/Yadis (XML configuration file on an HTTP
> server) would suffice (with perhaps something like XCAP in play too). Couple
> that with the new .well-known URI mechanism
> (<http://tools.ietf.org/html/draft-nottingham-site-meta-05>) and finding a
> service would be as simple as doing an HTTP request on the primary domain with
> request-URI .well-known/XXX.

You need a layer of indirection between the mail domain and the web
server, e.g. SRV or NAPTR, to accommodate sites like cam.ac.uk which do
not have a web server on their zone apex, or customers who delegate email
service to a service provider.

> In fact I have been thinking that one option here is to register
> .well-known/srv and then any of the registered SRV service types could have
> meta-data stored underneath that. e.g.:
>
> .well-known/srv/imap
> .well-known/srv/submission
>
> etc

MUA configuration should be in one file. If you split it up by protocol
then it is too easy to perpetuate problems with POP vs. IMAP
prioritization.

> If data needs to varying on a per-user basis, then defining a simple RESTful
> api to include the userid in the request would be easy.

The MUA doesn't have a userid at this point: it only knows an email
address.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

Date: Thu, 07 Jan 2010 10:51:19 -0500
From: Cyrus Daboo <[email-masked]>
To: Tony Finch <[email-masked]>
cc: [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

Hi Tony,

--On January 5, 2010 6:49:16 PM +0000 Tony Finch <[email-masked]> wrote:

> A few comments on draft-daboo-srv-email:

Just to update everyone on the status of this draft - there are a whole 
bunch of comments pending from the original review of this last year. I 
have been waiting for the SRV registry spec to be updated following the 
heated debates at the end of last year which came to resolution at the 
meeting in Japan. Once that spec is out I will revise srv-email document. 
The other major point was having srv-email say very little itself about 
certificate verification, and instead refer to 
<http://tools.ietf.org/html/draft-saintandre-tls-server-id-check-02>. It 
would be nice if that spec could have an explicit example of verification 
where SRV was used to get the server hostname.

-- 
Cyrus Daboo

Date: Thu, 07 Jan 2010 10:46:47 -0500
From: Cyrus Daboo <[email-masked]>
To: John C Klensin <[email-masked]>, Tony Finch <[email-masked]>, Alexey Melnikov <[email-masked]>
cc: [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

Hi John,

--On January 6, 2010 3:03:51 PM -0500 John C Klensin <[email-masked]> 
wrote:

> Such a file could also do two things that no "find the server"
> mechanism can provide: (i) deal with the POP/IMAP (or really
> POP/IMAP/Webmail) question on a per-user or per-environment
> basis and (ii) deal with multiple mailboxes, on multiple
> servers, for the same user in an efficient way.  I think both
> are realistically necessary today in a world where signing up
> for various IM or other services carries mailboxes with them.
> For that purpose, one could think of any of the approaches that
> have been mentioned, even though my mentioning ACAP was intended
> mostly as a bad joke and sad commentary about a lost opportunity.

ACAP is overkill for what is needed here.

Frankly something akin to XRDS/Yadis (XML configuration file on an HTTP 
server) would suffice (with perhaps something like XCAP in play too). 
Couple that with the new .well-known URI mechanism 
(<http://tools.ietf.org/html/draft-nottingham-site-meta-05>) and finding a 
service would be as simple as doing an HTTP request on the primary domain 
with request-URI .well-known/XXX.

In fact I have been thinking that one option here is to register 
.well-known/srv and then any of the registered SRV service types could have 
meta-data stored underneath that. e.g.:

.well-known/srv/imap
.well-known/srv/submission

etc

If we define an XML schema for resources located at those URIs we can 
provide great flexibility for client bootstrap configuration.

If data needs to varying on a per-user basis, then defining a simple 
RESTful api to include the userid in the request would be easy.

-- 
Cyrus Daboo

Date: Thu, 07 Jan 2010 10:37:10 -0500
From: Cyrus Daboo <[email-masked]>
To: Tony Finch <[email-masked]>, Alexey Melnikov <[email-masked]>
cc: John C Klensin <[email-masked]>, [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

Hi Tony,

--On January 6, 2010 5:38:04 PM +0000 Tony Finch <[email-masked]> wrote:

>> I think there is value in just being able to do easy MUA configuration
>> for an email address. Something based on DNS is going to be easy to
>> deploy. SRV RR are more widely deployed and supported, that is why
>> draft-daboo-srv-email is using them.
>
> But what is the motivation to implement and deploy the SRV record scheme?
> An MUA can do better (better coverage, easier deployment) by probing
> several common setups until it finds one that works. Outlook 2007 already
> does this (sorry, can't find a more detailed description than
> <http://support.microsoft.com/kb/287532>) and it looks like Thunderbird
> has something similar in hand. All it requires from the operator is to use
> common host names for the various servers. So I think this draft does too
> little too late.

I disagree. Just because clients have been forced to do things using 
guesswork up to now does not mean we should leave them and any new clients 
to do the same thing. I have heard from at least one developer who is 
working on implementing SRV support for Thunderbird.

The SRV solution is simple to implement - most OS network libraries now 
provide simple access to SRV record results. Site and domain admins these 
days are much more comfortable with setting up SRV (required for services 
like XMPP). So I think SRV represents a simple solution that can be adopted 
in a reasonable timeframe.

-- 
Cyrus Daboo

Date: Wed, 6 Jan 2010 21:07:06 +0000
From: Tony Finch <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
cc: [email-masked], Alexey Melnikov <[email-masked]>, John C Klensin <[email-masked]>, [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

On Wed, 6 Jan 2010, Tony Finch wrote:
>
> I tried to find some documentation on Microsoft's site describing what
> probes Outlook performs, but annoyingly I failed.

Aha: http://technet.microsoft.com/en-us/library/cc511507.aspx#BKMK_OverviewOfCommonSettingsDiscover
Reasonably sane, apart from their POP vs IMAP decision.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

Date: Wed, 06 Jan 2010 15:58:13 -0500
From: Hector Santos <[email-masked]>
Organization: Santronics Software, Inc.
To: Tony Finch <[email-masked]>
CC: Alexey Melnikov <[email-masked]>, John C Klensin <[email-masked]>, [email-masked], [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

Tony Finch wrote:

> On Wed, 6 Jan 2010, Alexey Melnikov wrote:
>> I think there is value in just being able to do easy MUA configuration for an
>> email address. Something based on DNS is going to be easy to deploy. SRV RR
>> are more widely deployed and supported, that is why draft-daboo-srv-email is
>> using them.
> 
> But what is the motivation to implement and deploy the SRV record scheme?
> An MUA can do better (better coverage, easier deployment) by probing
> several common setups until it finds one that works. Outlook 2007 already
> does this (sorry, can't find a more detailed description than
> <http://support.microsoft.com/kb/287532>) and it looks like Thunderbird
> has something similar in hand. All it requires from the operator is to use
> common host names for the various servers. So I think this draft does too
> little too late.
> 
> Tony.

Here is a link with a web cast that explains the Outlook 2007 
Automatic Configuration process.

         http://support.microsoft.com/kb/941842

Apparently, there are three process:

   Active Directory  NT/AD domains logins

   Active Discover   CAD servers w/ Exchange 2007
		    HTTP url queries, autodiscover.xml
	
   Guess Mart        Host names lookups

-- 
Sincerely

Hector Santos
http://www.santronics.com

Date: Wed, 6 Jan 2010 20:04:43 +0000
From: Tony Finch <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
cc: [email-masked], Alexey Melnikov <[email-masked]>, John C Klensin <[email-masked]>, [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

On Wed, 6 Jan 2010, Arnt Gulbrandsen wrote:
>
> Personally, I'd like a document saying "these are the SRV labels to use for
> email, should you wish to use any" with an informative by-the-way section
> saying basically "and if your SRV targets are so-and-so, outlook will be
> happy". That's not ideal, but it's my preferred compromise.

I mostly agree. However current autoguess configurers just looks for
plausible A records, e.g. mail.domain / imap.domain / smtp.domain etc.
which is why I say there's no need for SRV records.

> The Thunderbird authors are nice guys (nicer than I am at any rate) and if you
> suggest to them that if the best _imap._tcp SRV has better priority than the
> best _pop3._tcp, tb should default to IMAP, they'll probably do it.

They already think that way :-)

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

Date: Wed, 06 Jan 2010 15:03:51 -0500
From: John C Klensin <[email-masked]>
To: Tony Finch <[email-masked]>, Alexey Melnikov <[email-masked]>
cc: [email-masked], [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

--On Wednesday, January 06, 2010 17:38 +0000 Tony Finch
<[email-masked]> wrote:

> On Wed, 6 Jan 2010, Alexey Melnikov wrote:
>> 
>> I think there is value in just being able to do easy MUA
>> configuration for an email address. Something based on DNS is
>> going to be easy to deploy. SRV RR are more widely deployed
>> and supported, that is why draft-daboo-srv-email is using
>> them.
> 
> But what is the motivation to implement and deploy the SRV
> record scheme? An MUA can do better (better coverage, easier
> deployment) by probing several common setups until it finds
> one that works. Outlook 2007 already does this (sorry, can't
> find a more detailed description than
> <http://support.microsoft.com/kb/287532>) and it looks like
> Thunderbird has something similar in hand. All it requires
> from the operator is to use common host names for the various
> servers. So I think this draft does too little too late.

I'm inclined to agree with Tony, especially the moment the
question moves past "find the nearly Submission server" toward
"find the mail".  The marketplace seems to have decided that
hunting around the DNS for commonly-used naming structures and
then probing is sufficient.  Given the difficulties of finding
the "right" domain with which to associate an SRV record, it
seems unlikely that this proposal will add significant value to
that approach, even though I think it definitely would have been
a good idea some years ago.    Conversely, serious per-user
configuration, which requires multiple parameters, an understand
of security mechanisms used, locations of user mailboxes and
protocol and access mechanisms for them, etc., almost certainly
requires the ability to retrieve a configuration file (in some
plausible format), not just the identity of a host providing a
mail-specific service.  

Such a file could also do two things that no "find the server"
mechanism can provide: (i) deal with the POP/IMAP (or really
POP/IMAP/Webmail) question on a per-user or per-environment
basis and (ii) deal with multiple mailboxes, on multiple
servers, for the same user in an efficient way.  I think both
are realistically necessary today in a world where signing up
for various IM or other services carries mailboxes with them.
For that purpose, one could think of any of the approaches that
have been mentioned, even though my mentioning ACAP was intended
mostly as a bad joke and sad commentary about a lost opportunity.

Like Tony, I don't think Outlook's approach is the right answer:
I think it could be a good place to start thinking about how to
do things right, but what I remember of its actual mechanism,
refreshed by Tony's notes, is that it isn't something we (or
probably even Microsoft today) would want to standardize.  A
careful look at why ACAP never really got any traction would
also be instructive.

     john

Date: Wed, 6 Jan 2010 20:55:02 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: draft-daboo-srv-email
Cc: Tony Finch <[email-masked]>, Alexey Melnikov <[email-masked]>, John C Klensin <[email-masked]>, [email-masked]
Sender: [email-masked]

Tony Finch writes:
> If you mean its "auto account setup" feature, not its horrid 
> autodiscover protocol, then yes but with caveats: if there are 
> multiple possible configurations (e.g. pop/imap, tls/clear, 
> smtp/submission) then I don't trust Outlook's preferences to be the 
> right ones to document. The choice of hostnames and ports to probe is 
> more likely to be reasonable.

I read this as a longwinded way of saying "what outlook does is the most 
common practice, not the  best".

Personally, I'd like a document saying "these are the SRV labels to use 
for email, should you wish to use any" with an informative by-the-way 
section saying basically "and if your SRV targets are so-and-so, 
outlook will be happy". That's not ideal, but it's my preferred 
compromise.

The POP/IMAP question is IMO separate.

The Thunderbird authors are nice guys (nicer than I am at any rate) and 
if you suggest to them that if the best _imap._tcp SRV has better 
priority than the best _pop3._tcp, tb should default to IMAP, they'll 
probably do it. The Apple mail authors ditto. The Outlook authors may 
be nice guys too, but I think your chance of persuading them to change 
their POP vs. IMAP selection mechanism is basically zero, no matter 
whether the mechanism is NAPTR, ACAP, son-of-ACAP or the SRV hack 
above.

Arnt

Date: Wed, 6 Jan 2010 19:41:41 +0000
From: Tony Finch <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
cc: [email-masked], Alexey Melnikov <[email-masked]>, John C Klensin <[email-masked]>, [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

On Wed, 6 Jan 2010, Arnt Gulbrandsen wrote:
>
> Tony: Would you support a BCP documenting what Outlook does, and recommending
> that operators add CNAMEs/As/AAAAs for those exact hostnames?

If you mean its "auto account setup" feature, not its horrid autodiscover
protocol, then yes but with caveats: if there are multiple possible
configurations (e.g. pop/imap, tls/clear, smtp/submission) then I don't
trust Outlook's preferences to be the right ones to document. The choice
of hostnames and ports to probe is more likely to be reasonable.

I tried to find some documentation on Microsoft's site describing what
probes Outlook performs, but annoyingly I failed.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

Date: Wed, 6 Jan 2010 20:12:04 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: draft-daboo-srv-email
Cc: Tony Finch <[email-masked]>, Alexey Melnikov <[email-masked]>, John C Klensin <[email-masked]>, [email-masked]
Sender: [email-masked]

Tony Finch writes:
> An MUA can do better (better coverage, easier deployment) by probing 
> several common setups until it finds one that works. Outlook 2007 
> already does this (sorry, can't find a more detailed description than 
> <http://support.microsoft.com/kb/287532>) and it looks like 
> Thunderbird has something similar in hand. All it requires from the 
> operator is to use common host names for the various servers. So I 
> think this draft does too little too late.

Tony: Would you support a BCP documenting what Outlook does, and 
recommending that operators add CNAMEs/As/AAAAs for those exact 
hostnames?

(Honest question, no rhetoric. Can't expect much rhetoric from me 
anyway, I lost the taste many years ago.)

Arnt

Date: Wed, 6 Jan 2010 17:38:04 +0000
From: Tony Finch <[email-masked]>
To: Alexey Melnikov <[email-masked]>
cc: John C Klensin <[email-masked]>, [email-masked], [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

On Wed, 6 Jan 2010, Alexey Melnikov wrote:
>
> I think there is value in just being able to do easy MUA configuration for an
> email address. Something based on DNS is going to be easy to deploy. SRV RR
> are more widely deployed and supported, that is why draft-daboo-srv-email is
> using them.

But what is the motivation to implement and deploy the SRV record scheme?
An MUA can do better (better coverage, easier deployment) by probing
several common setups until it finds one that works. Outlook 2007 already
does this (sorry, can't find a more detailed description than
<http://support.microsoft.com/kb/287532>) and it looks like Thunderbird
has something similar in hand. All it requires from the operator is to use
common host names for the various servers. So I think this draft does too
little too late.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

Date: Wed, 06 Jan 2010 17:33:22 +0100
From: Alexey Melnikov <[email-masked]>
To: John C Klensin <[email-masked]>, Tony Finch <[email-masked]>
CC: [email-masked], [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

John C Klensin wrote:

>--On Tuesday, January 05, 2010 18:49 +0000 Tony Finch
><[email-masked]> wrote:
>  
>
>>...
>>To step back a bit, perhaps it would be easier to solve all
>>these problems by defining a simple (XML?) configuration
>>document which an MUA could download. This suggestion is not a
>>million miles from the autodiscover support in Microsoft
>>Outlook 2007, except that its protocol is a disgusting series
>>of hacks which is best used as an example of how not to do it.
>>http://fanf.livejournal.com/95018.html
>>
>>I wonder if NAPTR would be useful part of a clean autodiscover
>>protocol, instead of SRV.
>>    
>>
>There is, of course, always ACAP (or perhaps offspring-of-ACAP
>that used an XML config file)... assuming you can figure out how
>to find that server :-(.
>  
>
Indeed. Or some XMPP extension, etc.

I think there is value in just being able to do easy MUA configuration 
for an email address. Something based on DNS is going to be easy to 
deploy. SRV RR are more widely deployed and supported, that is why 
draft-daboo-srv-email is using them. NAPTR was mentioned in Lemonade WG 
when the draft was discussed, but there were not enough momentum to 
switch from SRV to NAPTR back then.

There are also use cases for a more complete account configuration. 
Something like ACAP/XML in foo/... is useful for that. But I don't think 
that requiring the former problem to be solved with such tool is going 
to benefit the community in short or medium term, as it might take ages 
to standardize, by which time major email clients deploy 10 different 
and incompatible solutions.

Date: Wed, 06 Jan 2010 01:41:05 -0500
From: John C Klensin <[email-masked]>
To: Tony Finch <[email-masked]>, [email-masked]
cc: [email-masked]
Subject: Re: draft-daboo-srv-email
Sender: [email-masked]

--On Tuesday, January 05, 2010 18:49 +0000 Tony Finch
<[email-masked]> wrote:

>...
> To step back a bit, perhaps it would be easier to solve all
> these problems by defining a simple (XML?) configuration
> document which an MUA could download. This suggestion is not a
> million miles from the autodiscover support in Microsoft
> Outlook 2007, except that its protocol is a disgusting series
> of hacks which is best used as an example of how not to do it.
> http://fanf.livejournal.com/95018.html
> 
> I wonder if NAPTR would be useful part of a clean autodiscover
> protocol, instead of SRV.

There is, of course, always ACAP (or perhaps offspring-of-ACAP
that used an XML config file)... assuming you can figure out how
to find that server :-(.

Date: Tue, 5 Jan 2010 18:49:16 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
cc: [email-masked]
Subject: draft-daboo-srv-email
Sender: [email-masked]

A few comments on draft-daboo-srv-email:

There's no way for a service provider to describe the order of preference
for the various protocols. For instance, we support both IMAP and POP but
we strongly prefer our users to use IMAP, however Outlook's autoguess
feature prefers POP. We could, I suppose, not deploy any _pop3 SRV
records, but then that means POP-only clients can't autoconfigure
themselves. The priority question also applies to imap/imaps and
pop3/pop3s - some paranoid firewall admins are prepared to let ports 995
and 993 through because they know passwords will be encrypted, but not 143
and 110. [Insert rant about smtps here.]

The requirement to use the "service domain" in server certificates
conflicts with current practice, which is to use the server host name.
For example our server hostnames have the form imap.hermes.cam.ac.uk,
which also appears in the certificate CNs - not hermes.cam.ac.uk. This
implies that our servers would have to support TLS Server Name
Indication to the extent of being able to present differing certificates,
which AFAIK they can't currently do.

The draft ought to recommend that clients keep configuration discovery
separate from normal operation. That is, clients should (1) do the SRV
lookups (2) make trial connections to discover TLS capabilities and login
name formats (3) ask the user ONCE to confirm that everything is correct
(4) save this information (5) when connecting to the server during normal
operation, verify that the saved configuration is still correct wrt the
real world, and if it isn't, ask the user if they want to try
reconfiguring, but do not automatically fall back to a less secure setup.

Cacheing the configuration avoids repeated login failures if the server
doesn't support full-email-address login names; it avoids repeated
questions to the user about service domain / server hostname mismatches;
it avoids some DNS attacks; and it avoids STARTTLS downgrade attacks. (The
continued existence of "TLS when available" options in most MUAs is a
disgrace.)

The specification needs to be clearer about what it means for a service
domain and a server host name to match.

It has often been noted that MUAs can do nearly as well without explicit
SRV records just by guessing. However if the following wishes were
answered then draft-daboo-srv-email might be worth implementing.

It would be nice if there were some kind of virtual domain support, e.g.
so a customer of an ISP can just redirect MUAs to their ISP's mail
servers, preferably by changing two DNS records instead of six. Even
better if this means the ISP's mail servers don't have to have TLS
certificates for each of their customer domains.

I would really like support for virtual domains which have multiple
back-end servers, for example, cam.ac.uk is served by hermes.cam.ac.uk and
admin.cam.ac.uk and medschl.cam.ac.uk and mole.bio.cam.ac.uk et cetera ad
nauseam. Ideally given an @cam email address the MUA could present the
user with a list of potential back ends. It would also be nice if we could
attach descriptive text to the domain, such as "University of Cambridge
Computing Service - Hermes email service".

To step back a bit, perhaps it would be easier to solve all these problems
by defining a simple (XML?) configuration document which an MUA could
download. This suggestion is not a million miles from the autodiscover
support in Microsoft Outlook 2007, except that its protocol is a
disgusting series of hacks which is best used as an example of how not to
do it. http://fanf.livejournal.com/95018.html

I wonder if NAPTR would be useful part of a clean autodiscover protocol,
instead of SRV.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

Date: Tue, 5 Jan 2010 14:29:55 +0000
From: Tony Finch <[email-masked]>
To: Sabahattin Gucukoglu <[email-masked]>
cc: [email-masked]
Subject: Re: Eight-Bit Line-Oriented Binary Encoding in SMTP
Sender: [email-masked]

On Mon, 4 Jan 2010, Sabahattin Gucukoglu wrote:
>
> It will have been discussed before now, but it's quite apparent that we
> want the ability to send binary data that isn't line-oriented using as
> simple sudden implementation change as possible.

Have you overlooked the "BINARYMIME" SMTP service extension (RFC 3030) and
MIME's "binary" content-transfer-encoding (RFC 2045).

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

Date: Tue, 5 Jan 2010 14:23:16 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
cc: Sabahattin Gucukoglu <[email-masked]>, [email-masked]
Subject: Re: DATA Pipelining
Sender: [email-masked]

On Mon, 4 Jan 2010, [email-masked] wrote:
>
> Attempting to avoid such bugs on the client side is an exercise in futility,
> due to the nature of TCP and IP underneath it. You cannot in general prevent
> packet fragmenation, reassembly, rebuffering, and all sort of other
> entertaining stuff from accurring, so your attempts to align your buffering to
> accomodate a broken received that doesn't handle its buffers properly are going
> to fail - the only question is when, not if.

Ned's right, of course, though the SMTP specification is vague about
exactly what a "reliable ordered data stream" consists of.

For a real instance of "entertaining stuff", I once had fun diagnosing a
bug in a Fortigate firewall that attempted to do TCP resegmentation. I
wouldn't have noticed, except that it screwed up if the segment happened
to be (n*512)+1 bytes long, causing all but the first 512 bytes of the
segment to be lost, which in turn caused the occasional message to get
stuck on the queue.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

From: [email-masked]
Cc: [email-masked]
Date: Mon, 04 Jan 2010 12:05:36 -0800 (PST)
Subject: Re: DATA Pipelining
To: Sabahattin Gucukoglu <[email-masked]>
Sender: [email-masked]

> On 2 Jan 2010, at 22:00, Tony Finch wrote:
> > On Sat, 26 Dec 2009, Sabahattin Gucukoglu wrote:
> >> As I see it, there is no reason not to pipeline both the message text
> >> lines and the final dot, because nothing is expected from the server and
> >> until the dot is received there is no change of state other than the
> >> accumulation of the DATA buffer.
> >
> > You can pipeline more than that. You can send one message per RTT if you
> > pipeline one message's data with the next message's envelope, i.e.
> > stream out all of headers, body, dot, RSET, MAIL, RCPT, DATA; then wait
> > for the server's replies; then repeat.
> >

> Sorry, I meant unextended SMTP.  I was wanting to know whether, after the
> 354, I could send multiline data buffers followed by the dot, only single
> lines, or lines then just the dot.

Yes you can. TCP is a _stream_ protocol, after all.

> The difference is simply in how much gets put into the send buffer, and the
> rationale is avoiding silly bugs that can neither be specified nor assumed
> nonexistent.

Attempting to avoid such bugs on the client side is an exercise in futility,
due to the nature of TCP and IP underneath it. You cannot in general prevent
packet fragmenation, reassembly, rebuffering, and all sort of other
entertaining stuff from accurring, so your attempts to align your buffering to
accomodate a broken received that doesn't handle its buffers properly are going
to fail - the only question is when, not if.

> > If the server supports the BDAT command then your client's sending and
> > receiving pipelines can be completely asynchronous. However if you do this
> > you massively increase the risk of message duplication caused by lost
> > connections. There's some discussion of this in
> > http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/qsmtp/draft-fanf-smtp-rfc1845bis.html
> > However it's probably easier in practice to fix this problem using some
> > server-side deduplication cleverness without protocol extensions.

> This got me thinking on the precise nature of data duplication due to long
> delays between the final dot of the DATA block and the 250.  Given that the 250
> is the first send from the server after each client wait, would it not be
> feasible for the server to treat as undelivered any transaction for which it
> could not send the final acknowledgment,

No. There are cases where an error is returned but enough of the data made it
for the client to consider the message as having been processed.

Checkpointing, server session hashes, and similar tricks are the way to solve
the duplication problem. But it seems the market has decided that the benefits
of doing this stuff don't outweigh the costs.

> thus sympathising with the client's view of things?  Are there any
> difficulties, for instance, is there the possibility that the server could send
> the acknowledgement but it not be delivered in spite of a lack of error from
> the underlying transport?

It is actually pretty common, when the client has given up and gone away, for
the server ack to go through without an error and the next read is what fails.

> >> But I also know there are implementations out there which don't like it
> >> when a command is incomplete, and try to respond to whatever is in their
> >> TCP buffer, and wondered if anybody knew how that might work for the
> >> final dot if it were stuffed at the end of some text.
> >
> > Such implementations are not allowed to advertise PIPELINING.
> >
> > The worst example I have seen recently was a Cisco PIX or ASA with SMTP
> > fuxup mode turned on. It replaces unrecognized SMTP commands with XXXX,
> > but if a known command is split between packets it cannot recognize it so
> > XXXXs it out. (More details at http://fanf.livejournal.com/102206.html).
> > Perhaps that is what you are thinking of. This is just one of many bugs in
> > the PIX SMTP implementation. Avoid it at all costs.

> Sounds terrible!  But no, I was talking about *commercial MTAs*, usually for
> Windows, which if you connect to them with telnet in stream mode will happily
> disgorge "500 Unrecognised" whenever you send them a character.  Yes, they're
> out there, really!

Command processing is actually a rather different case from data processing,
mostly because commands tend to be small and therefore don't run into 
fragmentation and other TCP/IP stuff. This allows broken implementations that
mishandle commands enough leeway to work well enough to survive in the field,
which is why PIPELINING is an extension rather than being an optimization
technique.

				Ned

Subject: Re: DATA Pipelining
From: Sabahattin Gucukoglu <[email-masked]>
Date: Mon, 4 Jan 2010 14:23:05 +0000
To: [email-masked]
Sender: [email-masked]

On 2 Jan 2010, at 22:00, Tony Finch wrote:
> On Sat, 26 Dec 2009, Sabahattin Gucukoglu wrote:
>> As I see it, there is no reason not to pipeline both the message text
>> lines and the final dot, because nothing is expected from the server =
and
>> until the dot is received there is no change of state other than the
>> accumulation of the DATA buffer.
>=20
> You can pipeline more than that. You can send one message per RTT if =
you
> pipeline one message's data with the next message's envelope, i.e.
> stream out all of headers, body, dot, RSET, MAIL, RCPT, DATA; then =
wait
> for the server's replies; then repeat.
>=20
Sorry, I meant unextended SMTP.  I was wanting to know whether, after =
the 354, I could send multiline data buffers followed by the dot, only =
single lines, or lines then just the dot.  The difference is simply in =
how much gets put into the send buffer, and the rationale is avoiding =
silly bugs that can neither be specified nor assumed nonexistent.

> If the server supports the BDAT command then your client's sending and
> receiving pipelines can be completely asynchronous. However if you do =
this
> you massively increase the risk of message duplication caused by lost
> connections. There's some discussion of this in
> =
http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/qsmtp/draft-fanf-smtp-rfc=
1845bis.html
> However it's probably easier in practice to fix this problem using =
some
> server-side deduplication cleverness without protocol extensions.
>=20
This got me thinking on the precise nature of data duplication due to =
long delays between the final dot of the DATA block and the 250.  Given =
that the 250 is the first send from the server after each client wait, =
would it not be feasible for the server to treat as undelivered any =
transaction for which it could not send the final acknowledgment, thus =
sympathising with the client's view of things?  Are there any =
difficulties, for instance, is there the possibility that the server =
could send the acknowledgement but it not be delivered in spite of a =
lack of error from the underlying transport?

>> But I also know there are implementations out there which don't like =
it
>> when a command is incomplete, and try to respond to whatever is in =
their
>> TCP buffer, and wondered if anybody knew how that might work for the
>> final dot if it were stuffed at the end of some text.
>=20
> Such implementations are not allowed to advertise PIPELINING.
>=20
> The worst example I have seen recently was a Cisco PIX or ASA with =
SMTP
> fuxup mode turned on. It replaces unrecognized SMTP commands with =
XXXX,
> but if a known command is split between packets it cannot recognize it =
so
> XXXXs it out. (More details at =
http://fanf.livejournal.com/102206.html).
> Perhaps that is what you are thinking of. This is just one of many =
bugs in
> the PIX SMTP implementation. Avoid it at all costs.

Sounds terrible!  But no, I was talking about *commercial MTAs*, usually =
for Windows, which if you connect to them with telnet in stream mode =
will happily disgorge "500 Unrecognised" whenever you send them a =
character.  Yes, they're out there, really!

Cheers,
Sabahattin

From: Sabahattin Gucukoglu <[email-masked]>
Subject: Eight-Bit Line-Oriented Binary Encoding in SMTP
Date: Mon, 4 Jan 2010 03:47:19 +0000
To: [email-masked]
Sender: [email-masked]

Hi all,

It seems to me that we have enough implementations out there now =
demonstrating:
(A) Eight-bit (8bit) transfer encoding support, as used by 8BITMIME
(B) Eight-bit cleanliness in today's SMTP environment.

There are implementations out there (Exim, several Windows servers, =
Qmail, Courier, XMail) that do "Eight-bit vain", which is a new name I =
just made up which means that they assume the whole world besides =
themselves will be happy to deposit eight-bit data, with no special =
checking or support on that data, but rely on senders never to send it =
as a means of being fully compliant to 8BITMIME.

It will have been discussed before now, but it's quite apparent that we =
want the ability to send binary data that isn't line-oriented using as =
simple sudden implementation change as possible.  That means we need to =
decouple the transport from the encoding, or declare that a server can =
accept eight bit data using the DATA command.  If this means that =
somebody comes along with a new MIME CTE, then so be it, but the two =
need not be specified together.  In fact, the SMTP extension can come =
first to allow servers to get on with the business of spewing eight-bit =
data now, including 8bit, without worrying about conversion or other =
requirements specified by 8BITMIME and, in particular, with no =
compulsive behaviour changes other than ensuring that they are not about =
to push eight-bit data down the throat of someone who isn't clearly able =
to receive it, which as I've said is how current implementations are =
treating 8BITMIME.

I looked at yEnc (www.yenc.org); without the silly headers and trailers, =
the encoding fits RFC 5321 linear limits and RFC 2045 character =
restriction requirements as used by 8bit.  It would then be a small =
matter (yeah, right!) to declare a new CTE and, as and when appropriate, =
gradually begin using this new "8BITCLEAN" extension to use the newer, =
more efficient encoding, and thus save us all from the continued =
nonsense that is allowing 33% over for quotas on advertised maximum =
attachment sizes, file sending services, added disk and network, etc, =
etc, etc.

I suppose it would be ever-so-slightly more innocuous if I suggested =
that 8BITCLEAN be not unlike 8BITMIME, to provide the conversion =
requirements.  It requires a new CTE though, so this seems like a silly =
inhibition, but it could add the gatewaying advantage.  It's not =
something I'd like to depend on, but there is the important problem of =
how new CTEs will take if we don't protect centuries-old software.  So, =
I'd love your input on that too, even though as an SMTP server code =
implementer my guess is that people would have chosen CHUNKING if it fit =
that need before now, and they didn't.  My point here is that for a lot =
of implementations, this new extension is nothing - just add an =
extension parameter in the MAIL command in the client if there's a new =
CTE or for audit purposes, and an extension keyword to EHLO responses in =
servers to inform clients they can get away with murder if they so =
choose.

And then there's always the possibility, of course, that I'm dreaming =
again and/or something important is missing.

Cheers,
Sabahattin

Date: Sat, 2 Jan 2010 22:00:15 +0000
From: Tony Finch <[email-masked]>
To: Sabahattin Gucukoglu <[email-masked]>
cc: [email-masked]
Subject: Re: DATA Pipelining
Sender: [email-masked]

On Sat, 26 Dec 2009, Sabahattin Gucukoglu wrote:
>
> As I see it, there is no reason not to pipeline both the message text
> lines and the final dot, because nothing is expected from the server and
> until the dot is received there is no change of state other than the
> accumulation of the DATA buffer.

You can pipeline more than that. You can send one message per RTT if you
pipeline one message's data with the next message's envelope, i.e.
stream out all of headers, body, dot, RSET, MAIL, RCPT, DATA; then wait
for the server's replies; then repeat.

If the server supports the BDAT command then your client's sending and
receiving pipelines can be completely asynchronous. However if you do this
you massively increase the risk of message duplication caused by lost
connections. There's some discussion of this in
http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/qsmtp/draft-fanf-smtp-rfc1845bis.html
However it's probably easier in practice to fix this problem using some
server-side deduplication cleverness without protocol extensions.

> But I also know there are implementations out there which don't like it
> when a command is incomplete, and try to respond to whatever is in their
> TCP buffer, and wondered if anybody knew how that might work for the
> final dot if it were stuffed at the end of some text.

Such implementations are not allowed to advertise PIPELINING.

The worst example I have seen recently was a Cisco PIX or ASA with SMTP
fuxup mode turned on. It replaces unrecognized SMTP commands with XXXX,
but if a known command is split between packets it cannot recognize it so
XXXXs it out. (More details at http://fanf.livejournal.com/102206.html).
Perhaps that is what you are thinking of. This is just one of many bugs in
the PIX SMTP implementation. Avoid it at all costs.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.