Subject: [ietf-smtp] SMTP server reply extensions

From: Timo Sirainen <[email-masked]>
Date: Wed, 8 Apr 2020 17:00:27 +0300

Hi,

I've recently found several reasons to add some kind of extended response metadata to SMTP/LMTP's RCPT and DATA replies. Similar to IMAP's resp-codes. Since I'm not so familiar with SMTP protocol, I was thinking I'd ask here if anyone happens to know already existing solutions for this or has some recommendations? Maybe there is already some RFC solving a similar problem?

IMAP uses "[resp-code parameters]" strings, but this doesn't seem to be used with SMTP protocol. Another option is to invent new SMTP enhanced code numbers, but especially for private extensions it seems a bit dangerous since there could be numbering conflicts. Also I'm not sure if SMTP clients would become confused when they see codes they don't understand? Also another problem with these numeric codes is how to add multiple different replies, especially since multi-line replies can't have different enhanced codes. But perhaps a single new extensible enhanced code could be invented? Something like:

RCPT TO:<[email-masked]>
250-2.1.100 key1 metadata1
250-2.1.100 key2 metadata2
250 2.1.100 OK with some extra metadata
DATA
354 OK
.
250-2.0.100 key1 metadata1
250-2.0.100 key2 metadata2
250 2.0.100 <[email-masked]> Saved and returned some metadata

To maybe clarify things, below are the 3 use cases I have so far. The first two are likely to be completely internal to Dovecot and I'm not trying make them standard.

1. IMAP-like login referrals (RFC 2221)

This RFC isn't really used nowadays, but I'm now thinking of using referrals internally in Dovecot clusters. Dovecot backend could send a referral and Dovecot proxy would reconnect to that referral instead. But the same thing is needed also for LMTP, which means that it would have to be sent in a RCPT TO reply since the redirection can be different for different users. Perhaps something like:

RCPT TO:<[email-masked]>
250 2.1.5 OK
RCPT TO:<[email-masked]>
450 4.4.4 [REFERRAL smtp://backend2/] Redirected
RCPT TO:<[email-masked]>
450 4.4.4 [REFERRAL smtp://backend3/] Redirected

2. Single instance storage for deliveries between backends in a cluster

The idea is that if backends share a storage (NFS, object storage) and there are multiple recipients, the proxy could deliver mail first to backend1, get back some metadata and use that to deliver mail to backend2, which would use the metadata to store a copy of the original mail. So for example:

RCPT TO:<[email-masked]>
250 2.1.5 OK
DATA
354 OK
.
250 2.0.0 <[email-masked]> [MAILDATA /nfs/mail/[email-masked]/INBOX/Maildir/file1] Saved

(In reality the path would likely be a base64 encoded blob)

And then in backend2:
RCPT TO:<[email-masked]> MAILDATA /nfs/mail/[email-masked]/INBOX/Maildir/file1

Now backend2 can create a hard link from [email-masked]'s mail to [email-masked]'s INBOX, saving some disk space. (Obviously only if the mails would be identical.)

3. We published a "submission token" draft a while ago: https://www.ietf.org/archive/id/draft-slusarz-extra-smtp-submission-token-00.txt

It's going to need a bunch of changes, but that draft is also trying to solve this metadata problem by inventing new enhanced status codes, e.g.:

     S: 250 2.1.13 <[email-masked]> etLBdTj1iG NRaaVe83QN Saved

The client is expected to understand 2.1.13 and to parse the reply appropriately.

The 2 and 3 cases are also an example of when it would be necessary to combine these metadata to the same DATA reply.

---

From: Jeremy Harris <[email-masked]>
Date: Wed, 8 Apr 2020 15:31:35 +0100

> 450 4.4.4 [REFERRAL smtp://backend3/] Redirected

Perhaps this?

RFC 4321 Section 4.2.3
  551  User not local; please try <forward-path>

-- 
Cheers,
  Jeremy

---

From: John C Klensin <[email-masked]>
Date: Wed, 08 Apr 2020 13:04:38 -0400

--On Wednesday, April 8, 2020 15:31 +0100 Jeremy Harris
<[email-masked]> wrote:

>> This RFC isn't really used nowadays, but I'm now thinking of
>> using referrals internally in Dovecot clusters. Dovecot
>> backend could send a referral and Dovecot proxy would
>> reconnect to that referral instead. But the same thing is
>> needed also for LMTP, which means that it would have to be
>> sent in a RCPT TO reply since the redirection can be
>> different for different users. Perhaps something like:
>> 

Jeremy, 

Thanks although I assume "4321" is a typo and your meant "5321".

The more general issue is that there has been a long-standing
principle of the SMTP design that, unless situation-specific
information is actually needed (and that use of 551 is one of
the exceptions), the codes contain most or the information and
are what is important, not the text.  That is important because
it avoids the SMTP-sender having to do natural language
processing and because it largely eliminates the need to worry
about internationalization (and hence translation) of the text
strings.  Of course the "act on the first digit only" rule is
intended to put even less burden on the SMTP-sender as well as
providing robustness against the SMTP-receiver using codes that
the SMTP-receiver has not been upgraded to understand
specifically. 

I've read through Timo's note a few times with the above as
context and I'm not sure I see the fit between what I think he
is proposing and SMTP.

First, a 4yz code implies that the sender should requeue the
message and try again later.  It is not an intermediate
processing indication of some variety with the implication that
the server is retaining responsibility for getting the message
delivered.   SMTP, by design (and long before me) does not have
such intermediate processing indications in reply codes;
inventing one or a series of them (1yz, perhaps?) would be a
rather major change to the standard and would probably break
SMTP-senders who have assumed there are only codes starting in
2, 3, 4, or 5 since the publication of RFC 821 in 1982.

Second, if the above is ignored, it would seem to me that the
right way to do at least part of this would be to use extended
status codes based on RFC 3463 and the registry established by
RFC 5428.

Finally, the information shown in that example looks a lot more
like trace information to me than like anything appropriate to a
reply code.  Normally, track information ends up on the hands of
someone on the far side of the SMTP-receiver.  But I can see no
reason (other than creating an opportunity for noise, blowback,
and spam if not done carefully) why one could not build a
Delivery Status Notification containing that type of trace
material and send it to the message originator or the
SMTP-sender.  No change to SMTP required.

Timo, if Jeremy's comment and that explanation do not answer
your question/ proposal, could you explain a bit more?

thanks,
   john

---

From: Tony Finch <[email-masked]>
Date: Wed, 8 Apr 2020 18:41:26 +0100

> 1. IMAP-like login referrals (RFC 2221)

Jeremy gave you the suggestion I was thinking of :-)

> 2. Single instance storage for deliveries between backends in a cluster

You might be able to use RFC 4468 BURL to refer to an existing copy of the
message, but I don't have a good LMTPish suggestion for getting a URL
from the back end.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Rattray Head to Berwick upon Tweed: Cyclonic 4 or 5 becoming variable 2 to 4,
then becoming east or southeast 3 to 5 later. Smooth or slight. Rain or
drizzle then fair. Good, occasionally moderate at first.

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Thu, 09 Apr 2020 20:34:08 -0400

> The idea is that if backends share a storage (NFS, object storage) and there
> are multiple recipients, the proxy could deliver mail first to backend1, get
> back some metadata and use that to deliver mail to backend2, which would use
> the metadata to store a copy of the original mail. So for example:

> Now backend2 can create a hard link from [email-masked]'s mail to
> [email-masked]'s INBOX, saving some disk space. (Obviously only if the
> mails would be identical.)

Maybe I need more caffeine, but I'm pretty convinced you'll have a hard time
getting identical mails, unless the front-end -> back-end mail fails to add
a Received: line. There's no way the Received: line for front->backend1 will
be identical to the Received: line for front->backend2.

---

From: Timo Sirainen <[email-masked]>
Date: Fri, 10 Apr 2020 15:45:45 +0300

On 8. Apr 2020, at 20.04, John C Klensin <[email-masked]> wrote:
> <[email-masked] <mailto:[email-masked]>> wrote:
> 

This looks like it could work for the first use case. Although I'm still wondering a bit whether to use the deprecated source routes:

551  User not local; please try <@backend2:[email-masked]>

Or if it should just rewrite the address in some way to contain the backend name as the domain (or local-part):

551  User not local; please try <""@backend2>
551  User not local; please try <[email-masked]>

There are still the other two issues though.

> 2, 3, 4, or 5 since the publication of RFC 821 in 1982.

The 4yz codes were used only for the first use case, which I think can be solved with the 551 result. And I suppose if I still wanted to do it in another way, 5yz code would have been more appropriate since that's what 551 is also doing.

> status codes based on RFC 3463 and the registry established by

I think you means some other RFC than 5428? Seems unrelated.

Anyway, the extended status codes couldn't at least fully be used for the other two examples I had, since it's not possible to give two different status codes in a single reply. But I suppose since the 2nd example is just internal communication data then maybe it could be just hacked in some way. I was just hoping for a something that could be used in a generic way to add any kind of key-value metadata pairs to replies, but it's beginning to sound like there's no nice SMTP-way of doing that.

> SMTP-sender.  No change to SMTP required.

I think you're still talking about the first example? The reply code was meant from LMTP backend server to the LMTP proxy server in front of it, so the proxy would instead deliver the mail to a different backend's LMTP server (user was just moved from backend1 to backend2). Sending DSN to the LMTP proxy that would match it to that specific ongoing LMTP session would get pretty complicated to implement.

---

From: Timo Sirainen <[email-masked]>
Date: Fri, 10 Apr 2020 15:56:23 +0300

> be identical to the Received: line for front->backend2.

True, I didn't remember that LMTP also normally adds a Received: header. But I don't think it's really necessary to add them, and if this feature is implemented then we could just require turning them off in backends. It's just an internal implementation detail how the mails are delivered after MTA has accepted them. I know some of our customers already have disabled the Received headers because they don't want any of the internal server names visible in the mails. All the tracing information exists in the log files anyway.

---

From: Ned Freed <[email-masked]>
Date: Fri, 10 Apr 2020 08:43:51 -0700 (PDT)

I guess I'm missing something pretty fundamental here, because I don't see why
any of this is necessary.

We have a model for extending SMTP/LMTP that includes private extensions:
Advertise the extension in the EHLO/LHO response, then enable the extension
either with a new command or parameter. There's essentially no limit on what an
extension can do, and it certainly includes enabling the use of structured
information in certain replies.

Of course it makes sense to reuse existing structured syntax when possible. As
for reusing existing enhanced status code, I'm really not on board with that -
I think new values are the right way to do it, and that once again we've been
guilty of integer-hoarding. (We really should have reserved a range for private
use, but there are enough values that it's difficult to get excited about
this.)

FWIW, we have the same redirect/referral issue in our software when a user has
been moved, but the LMTP server only knows it's the wrong place for a given
recipient. So what we do is signal that fact using a special private reply
code.

As for data deduplication, we do that with store hashes independent of
ownership, so no need for anything at this level.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Fri, 10 Apr 2020 09:07:03 -0700 (PDT)

> information in certain replies.

I should also note that the use of an enabling command or parameter here
isn't strictly necessary, especially in the case of LMTP: Since reply text
is normally unstructured, all that's needed is an indicator from the server
that it's going to include structured information in some replies. This
has worked just fine in the case of ENHANCEDSTATUSCODES.

				Ned

---

From: Timo Sirainen <[email-masked]>
Date: Fri, 10 Apr 2020 21:18:46 +0300

> information in certain replies.

Sure, I was just hoping to find a way to avoid designing a horribly non-SMTP like extension, even if it was a private one.

> use, but there are enough values that it's difficult to get excited about

So you are recommending not using the 551 code for the redirection purpose?

Hmm. Now thinking further about this, I'm not sure 551 would be enough for my purposes either. I think I'm going to need multiple different things returned in the reply. So maybe I'll do it with a new private response code also. Any recommendations what private codes to use? Maybe x.y.100 and over?

> recipient. So what we do is signal that fact using a special private reply

Until now Dovecot's proxies have been responsible for knowing which backend is correct and backends haven't done any extra verifications. But I'm now doing some redesigning of how clustering works and this extra information in replies can be useful in some situations.

---

From: Ned Freed <[email-masked]>
Date: Fri, 10 Apr 2020 17:04:43 -0700 (PDT)

> So you are recommending not using the 551 code for the redirection purpose?

I don't this the SMTP code you use matters much - the enhanced status code is
what I would look at to determine if there's additional information. But if
your information fits the 551 syntax, sure, why not? Although in our case I
think we use 450 since we regard this as a temporary failure. (AFAIK there is
no 551 temporary equivalent with the same semantics.)

> Hmm. Now thinking further about this, I'm not sure 551 would be enough for my
> purposes either. I think I'm going to need multiple different things returned
> in the reply. So maybe I'll do it with a new private response code also. Any
> recommendations what private codes to use? Maybe x.y.100 and over?

We used .99.

				Ned