Subject: [ietf-dkim] DKIM/ADSP edge case writeup at CircleID

From: "MH Michael Hammer (5304)" <[email-masked]>
Date: Tue, 24 Mar 2009 10:01:44 -0400

I posted a write up on an issue with DKIM/ADSP and missing message-ids
at CircleID that might be of interest to people:
 
http://www.circleid.com/posts/20090323_exploring_dkim_adsp_edge_cases_mi
ssing_message_id/

I'm going to try to spend a little more time focusing on some of the
implementation "gotchyas" related to DKIM/ADSP for a while.

Mike

_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Hector Santos <[email-masked]>
Date: Tue, 24 Mar 2009 11:35:21 -0400

> implementation "gotchyas" related to DKIM/ADSP for a while.

Michael,

It is bad practice for receiving software to be adding an originating 
header such as a Message-ID.  Especially, is this receiver software 
has gain some DKIM smarts (its active developed software).

The BCP for all exporting software to add the Messages-ID for its own 
tracing and dupe checking purposes.

Yes, the problem sometimes today with increase dumb mail creation 
software that only pushes out:

     From:
     To:
     Subject:
     Date:

and some that only do:

     To:
     Date:

with the presumption the exporting software will add the rest.

If the exporting software also has DKIM software, then it needs to be 
compliant here with 2822/5322.

But overall, I think this is just another issue regarding policy and 
domain mail expectations.

Just consider, even without DKIM, there is a POLICY idea where a 
receiver can check the From: domain for what is expected as far as 
level of mail compliants:

    v=ietf  rfc5322=required

That might means the standard headers are expected.

It might also have:

    v=ietf  rfc5322=required msgid=example.domain.com

which might say that the message-id RSH must have example.domain.com 
as the source.

We can further extend it to add ADSP support

    v=ietf  rfc5322=required msgid=example.domain.com dkim=all

Anyway, I am not quite sure if there an issue here with Message-ID 
because it is pretty well established.

The problem is protocol consistency.  The signer within the domain 
network needs to make sure it adds the missing headers.  And receivers 
really should not be adding one - especially if it is as you say, DKIM 
compliant.

-- 
Sincerely

Hector Santos
http://www.santronics.com

---

From: Hector Santos <[email-masked]>
Date: Tue, 24 Mar 2009 11:37:55 -0400

> software that only pushes out:

And also:

       From:
       Subject:

which is what I have seen in a few instances with "news letters"

-- 
Sincerely

Hector Santos

---

From: Mark Martinec <[email-masked]>
Date: Tue, 24 Mar 2009 16:54:16 +0100

> implementation "gotchyas" related to DKIM/ADSP for a while.

I promised the following list to Michael, but it is probably of a
wider interest, so I'm posting it here.

For more than a year I've been interested in cases where apparently
a genuine DKIM signature is broken as received at our mailer (with
about 1000 active users), so I started collecting such cases.

It's a bit tricky to find out what could be a reason for a failure,
but with practice some of these blunders can be guessed, typically
by trying to reconstruct the original message until a signature
becomes valid. Sometimes a combination of DKIM and a DomainKeys
signatures helps to see what went wrong, sometimes a 'z' tag helps,
sometimes I've been asking the sending site for joined troubleshooting
or I could reproduce it by using the same type of a suspect MTA,
and sometimes just plain guessing did the job.

So here is my list. Each entry reflect an actual case of received mail.
Some of these may have been fixed meanwhile by the sending domain,
so I'm not claiming that all of them still apply for the named domain.

- signing Bcc header field (which gets stripped away by MTA);

- signing a Return-Path header field (e.g.: yahoo-inc.com, [email-masked]);
  Apart from the fact that a verifyier at MTA may not see the Return-Path
  at all, and that signing envelope info is not something DKIM was supposed
  to do, the actual failure reason in this case was that a signer signed
  a Return-Path header field like: "Return-Path: [email-masked]", while
  the verifier saw a reconstructed header field with address in angle
  brackets like "Return-Path: <[email-masked]>" as required by RFC 2822;

- adding a local Sender header field and signing it, then posting to a
  mailing list, which may be DKIM friendly, but still is required to
  strip the original Sender header field and replace it with its own;
  My pointing the blame in this case goes to RFC 2822, which does not
  allow more than one occurrence of a Sender header field. Allowing
  multiple Sender header field (new ones appended at the top) would
  avoid the issue. This is a reason why our site is not signing
  the Sender header field (except for mailing list fanout);

- forwarding (by MailServer or Microsoft SMTPSVC?) replaces Message-ID,
  adds "FW:" to a Subject, replaces Date, but keeps original signature
  and Received header fields

- some mailing lists strip Received header fields (lighttpd, bugtraq)
  (although I should add that a breakage due to a stripped but signed
  Received header field is much less common that other breakages, like
  a mangled To header field)

- with DomainKeys: no h tag, but does not provide a Message-ID,
  which is inserted by a receiving MX prior to validation,
  e.g. from [email-masked], [email-masked]
  (still true at least for [email-masked])

- signature includes Message-ID in h tag, but there was no Message-ID in
  the original message at the time of signing. When a receiving MX inserts
  a missing header field, it breaks the signature.

- missing or misplaced public key, e.g. signs as
  s=mail, d=members.stockhouse.com, but key at d=stockhouse.com;
  or: jpl.nasa.gov key, lungusa.org, christopherreeve.org found
    under kintera.com;
  or: forgets a selector in DNS name: _domainkey.travian.com;
  or: ci._domainkey.ci.freewebs.com with d=freewebs.com;

- syntax errors in public key:
    * missing ';' between tags (stardock.com),..
    * a '+' in a published base64-encoded p tag is converted to a space:
      default._domainkey.biofeedbackinternational.com,
      k1._domainkey.mcsv22.net
      (looks like a web GUI blunder)

- signed 'To', but fetchmail(?) rewrites 'To' into a
  (Recipient list suppressed)

- sendmail reformats long lists of addresses in a To header field,
  which is why our site is not signing a To header field;

- some mailers add a space after a colon, e.g. rewriting a
  "Subject:foo" into a "Subject: foo"

- Mailman rewrites 'Reply-To: <addr>' into 'Reply-To: addr'
  and 'Reply-To: "Display Name" <addr>' into 'Reply-To: Display Name <addr>'

- Postfix strips bare CR at end-of-line which invalidates a signature in a
  message with embedded bare CRs at eol (e.g. with altermime html disclaimers)
  (this is just one case of a garbage-in / garbage-out principle, the lesson
  to be learned is that a mail to be signed should be syntactically correct)

- mail provider is mangling a Date header field:
    original: Date: Mon, 4 Aug 2008 17:43:42 -0400 (EDT)
    mangled:  Date: Mon, 04 Aug 2008 17:43:42 -0400 (EDT)

- system time on the signing host is few minutes into the future,
  dkim-milter considers it an invalid signature

- resend munge (at Cern)

- PerlMX munge

- eSafe munge: eSafe SMTP Relay / Microsoft SMTPSVC at gen-energija.si

- PMDF munger (rcum.uni-mb.si)

- eBay and PayPal: signs non-existent Resent-From, preventing resending

- the new yahoo.com DKIM signatures with c=relaxed/relaxed appears
  to get a body canonicalization wrong (signature h is ok, bh wrong)
  If someone is interested I have a trully small enough example,
  suitable for troubleshooting (several empty lines, followed by
  the last line with several spaces)

Mark

---

From: SM <[email-masked]>
Date: Tue, 24 Mar 2009 15:48:55 -0700

>so I'm not claiming that all of them still apply for the named domain.

[snip]

>- signing a Return-Path header field (e.g.: yahoo-inc.com, [email-masked]);

This generally occurs with a specific MTA.  It is not a RFC compliant behavior.

>   a missing header field, it breaks the signature.

That header field is a SHOULD.  It is not optional unless your view 
of implementation is restricted to "MUST".  That can be fixed at the 
message submission stage.

>- missing or misplaced public key, e.g. signs as

[snip]

>- syntax errors in public key:

These two problems are generally caught during testing.

>   which is why our site is not signing a To header field;

Do that cause a verification failure?  If so, can you send me a test 
case off-list?

>   "Subject:foo" into a "Subject: foo"

This is a MTA specific issue.

>   dkim-milter considers it an invalid signature

There is a ClockDrift setting to deal with that.  People generally do 
not notice this problem when they debug verification failures.

Regards,
-sm

---

From: "J.D. Falk" <[email-masked]>
Date: Tue, 24 Mar 2009 17:40:06 -0600

> This generally occurs with a specific MTA.  It is not a RFC compliant behavior.

Pointing to an RFC rarely mitigates real-world concerns.

-- 
J.D. Falk
Return Path Inc
http://www.returnpath.net/
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: John Levine <[email-masked]>
Date: 25 Mar 2009 01:09:28 -0000

>Pointing to an RFC rarely mitigates real-world concerns.

Indeed.  We need deployment guides, but it's not clear to me whether
they'd better be IETF documents or written somewhere else.

I don't think they'd be very complicated or controversial, mostly a checklist
of remember to do this, and not to do that.

R's,
John

---

From: SM <[email-masked]>
Date: Tue, 24 Mar 2009 21:49:07 -0700

>Pointing to an RFC rarely mitigates real-world concerns.

I commented on why the problem occurs.  I could argue that header 
field should not be present in a RFC 5322 message at the signing 
stage.  RFC 4871 lists some header fields that should be signed.  It 
also contains a list of header fields that should not be signed.  The 
Return-Path header field is listed in there.

If you believe this is a real-world concern that should be addressed, 
you could specify that the Return-Path header field must not be 
included in the signature.

Regards,
-sm

---

From: Hector Santos <[email-masked]>
Date: Wed, 25 Mar 2009 07:28:52 -0400

Mark,

Good review list.

Sounds to me that people are implementing DKIM, and I hate to use the 
term "blindly", but thats exactly how it appears with this review. 
This doesn't help with the confidence that DKIM can be successful 
outside the direct 1 to 1 communications where there is less 
expectation for integrity issues and in my view, where the focus needs 
to be to give DKIM some real power, rather than trying to make this 
work broadly with Mailing List Server and 3rd party signers.

But I think yo are showing the extent of how passthru mail is being 
reworked when the long tradition is to leave passthru mail alone - it 
only creates problems and you provided a good review of this 
unfortunate trend over the years.

The DKIM specification clearly says what headers SHOULD be considered 
and what SHOULD NOT.  I always felt there should a default and that h= 
was optional (to reduce the bandwidth), but having h= should help 
remove any guessing.

I have inline comments:

> - signing Bcc header field (which gets stripped away by MTA);

Thats crazy (signing BCC header).

Talking about Errata, this should be a MUST NOT sign.  This is the 
type of spec changes that are really worth the causes.  Not trying 
work in 3rd party signatures and reputations services which produces 
all kinds of 3-4 years of already discussed headaches.  This should of 
been a separate I-D.

>   brackets like "Return-Path: <[email-masked]>" as required by RFC 2822;

Same thing, RFC 4871 says this header SHOULD NOT be signed.  Again, 
another case thats shows  BLIND signing!

>   the Sender header field (except for mailing list fanout);

I think its a MLS/DKIM that has not been worked out. Another case 
where a MLS needs to be paying attention to what its going to strip 
and resign, and talking about "taking responsibility?" the originating 
domain should be more prudent about what it signs and where it 
"blindly" sends his mail! I don't see how a domain can expect the 
integrity of the signature to be safe knowing its going to be 
destroyed in nearly all MLS systems.

>   and Received header fields

I really, really, really hate middle ware that destroys passthru mail 
- it should be against the law! <g>

>   a mangled To header field)

Another MLS issue, especially when a MLS has gains DKIM smarts, it 
should be more conscious of such things.

>   a missing header field, it breaks the signature.

But if it was signed with one, then someone stripped it before the MX 
got it.

>       (looks like a web GUI blunder)

Another case of blind signing.  IMO, if failure is going to be 
tolerated, DKIM will never work well.

>   and 'Reply-To: "Display Name" <addr>' into 'Reply-To: Display Name <addr>'

wow.

>   to be learned is that a mail to be signed should be syntactically correct)

This is an issue for many.  We are talking about the MS-DOS, UNIX vs 
MAC storage and/or display world.  This is one of the reasons why we 
have not implemented yet.

>     mangled:  Date: Mon, 04 Aug 2008 17:43:42 -0400 (EDT)

Another case where passthru mail SHALL NOT BE touched!

>   dkim-milter considers it an invalid signature.

Ha! I just ran into a similar situation with OpenSSL creating a 
self-signed certificate valid 1 hour from now :-)  Didn't have time to 
figure that one out but to use another version.

> - PMDF munger (rcum.uni-mb.si)

Oh brother!

> - eBay and PayPal: signs non-existent Resent-From, preventing resending

Another case of BLIND signing!  Read the freaking specs!!

Anyway good list. Wow!  Maybe not as bad as this list makes it, but to 
me, it shows DKIM - a mail integrity technology is not going to work 
very well in a open distribution environment. Not direct 1 to 1 with 
less expectation for failure.

But if we don't give a hoot about failure, then whats the point of 
DKIM in the first place.

Bad guys will simply not sign or just fake it even if they know will fail.

Who is hurt by all this?

    The SMTP mail receivers!!

who are being asked to put up with this tolerance for failure, and 
only look for Mr. GoodBar in a crowd of misfits!  So if you see 3 
messages from the same domain:

    1 - valid signature
    2 - invalid signature
    3 - no signature

where 2 and 3 are the same according to RFC 4871, and the signature is 
expected to be there by the DOMAIN with no way of exposing that fact, 
I find it hard to shallow why the sending domain will want only the 
GOOD MAIL to be accepted without no instructions on handling 2 or 3.

If DKIM is going to have a chance to provide a payoff, we need to 
finally provide the power for exclusive domains, the high-value 
domains to better protect their 1 to 1 communications.

In this environment, most of those items in your list will not be 
tolerated and the mail will be expected to fail.

-- 
Sincerely

Hector Santos

---

From: Hector Santos <[email-masked]>
Date: Wed, 25 Mar 2009 08:28:14 -0400

> included in the signature.

I think this is the sort of "errata" material is worth the cause.

 From my experience, Return-Path should not be included in the 
signature.  But then again, we started as a UUCP/UUCICO system where 
"From " was part of the model and was replaced with Return-Path only 
for the router for failure to delivery purposes.  But the router does 
not resend it back out. For us, its "meta" data.

But over the years, you can see the evolution of more and more systems 
adding it and keeping it there all the way to the MUA to see.

So in my book, due to the history behind it, it is probably better to 
have a MUST NOT for this. SHOULD NOT is probably not strong enough for 
a mail integrity application.  Signing return-path might cause issues 
as Mark Martinec highlighted with non DKIM-aware software

But there are others in the SHOULD NOT list, for example, BCC, that 
really should be in a MUST NOT list simply because it is not expected 
to be there otherwise we are violating User Expectation for privacy.

Legally speaking, in the USA, this would be US EPCA protected item. 
BCC definitely should never be signed and it should not be in the 
headers after the mail is submitted for processing.  The distribution 
list should be expanded and the 8222 headers prepared properly for the 
  non-blind and blind distribution.   Definitely a process that should 
be done any DKIM SIGNING design consideration.

-- 
Sincerely

Hector Santos

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 26 Mar 2009 10:32:01 -0000

> Another case of BLIND signing!  Read the freaking specs!!

Not necessarily. Signing a non-existent header is a valid way of
preventing it being added subsequently, and maybe that is what you want
(e.g. in this case if the mail is "for original recipient's eyes only").
Not that Ebay and Paypal were necessarily trying to do that, although they
are the sort of organisations that just might want to do it in specific
situations.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5
_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: Hector Santos <[email-masked]>
Date: Thu, 26 Mar 2009 07:40:15 -0400

Charles Lindsey wrote:
> On Wed, 25 Mar 2009 11:28:52 -0000, Hector Santos 
> <[email-masked]> wrote:
> 
> 
> Not that Ebay and Paypal were necessarily trying to do that, although 
> they are the sort of organisations that just might want to do it in 
> specific situations.

Good point Charles.

I guess I can see benefits of signing an non-existing header with the
intent to preempt some downlink injection.  But only from the
standpoint of the intent to force a failure handling process. i.e, 
eBay, Paypal and entities of the like do not expect these failures to 
be ignored.  Possible example is Reply-To.  They might not want a 
Reply-To and will rely on From: for any user feedback.  So they sign 
an non-existing Reply-to, this preventing any replays with an injected 
Reply-to for MUAs to use.

I can see that.

-- 
Sincerely

Hector Santos

---

From: "MH Michael Hammer (5304)" <[email-masked]>
Date: Thu, 26 Mar 2009 09:03:30 -0400

> -----Original Message-----
> From: [email-masked] [mailto:ietf-dkim-
> [email-masked]] On Behalf Of Hector Santos
> Sent: Thursday, March 26, 2009 7:40 AM
> To: IETF-DKIM
> Subject: Re: [ietf-dkim] DKIM/ADSP edge case writeup at CircleID
> 
> Charles Lindsey wrote:

Hector,

This is exactly why I said in the article:

"Some might assert that an organization should never DKIM sign a
non-existent message-id header. At this time it is not clear, at least
to me, that this is absolutely true. The implications of signing versus
not signing under these circumstances certainly merit a healthy
discussion before a verdict is reached." 

I've seen little if any (systematic) discussion of the various cases
(for all headers) and how unsigned non-existent (at time of injection to
the mail stream or signing) might be abused at a later point. Most of
the discussion is about how things SHOULD function, not how they might
be abused.

Mike

---

From: Hector Santos <[email-masked]>
Date: Thu, 26 Mar 2009 11:19:42 -0400

> the discussion is about how things SHOULD function, not how they might

I agree.   As I call it "protocol consistency". Simply put, we don't 
have it here.

<SOAP>

I think there were many discussions in the past when SSP was still 
part of the picture.  SSP is what sold DKIM to me and others. The 
strong early emphasis and marketing presentation points wrt to SSP was 
the deciding difference over DKEYS.

ADSP watered it down and since we no longer have a true champion of 
policy based DKIM implementations speaking on our behalf, our voices 
and concerns are ignored and stamped out.

I will say, if ADSP is not part of the picture, I will not bother with 
DKIM. Maybe, if someone came with a DNS DKIM ZONE, that had only a 
listing of domains with ALWAYS SIGN, but with nothing like this, pure 
DKIM processing will be a waste of time.

I should of never gave up on DSAP (DKIM Signature Authorization 
Protocol) and kept on it. I only did so after being told SSP would be 
more considerate of the concerns outlined in DSAP.   If I knew SSP was 
going to be taken over by ADSP especially by someone who never 
believed in POLICY to begin with, which made the whole process, well, 
stink, no doubt I would had never given up on DSAP.

Oh well, as long as DKIM-BASE remains open and not locked down to 
specific accessors and reputation trust services, then at least there 
is still hope for new I-D and inventions to happen.  Maybe then I will 
introduce DSAP again.

While there might be some folks here that despise SPF, and even among 
those who support it and know its not 100% perfect, it did prove one 
thing:

     The industry desire to accept the idea of a DOMAIN EMAIL
     POLICY Discovery process solidified by the millions of domains
     and receivers that support SPF.

There is no doubt about that.

</SOAP>

-- 
Sincerely

Hector Santos