Subject: Re: [ietf-dkim] DKIM/ADSP edge case writeup at CircleID

From: Mark Martinec <[email-masked]>
Date: Thu, 26 Mar 2009 03:14:24 +0100

Hector Santos wrote:
> The problem is protocol consistency.  The signer within the domain
> network needs to make sure it adds the missing headers.

There is probably a general agreement that a message when submitted
to a signer should contain all mandatory header fields (From and Date)
as required by RFC 5322, along with all the SHOULD ones (Message-ID,
and some others in special cases according to section 3.6).
And it should be syntactically correct in other aspects too (line length,
allowed characters, ...), otherwise a garbage-in garbage-out principle
applies.

It also seems prudent to supply a To, even though it is not required by
the RFC 5322, as many MTAs or MUAs expect it and/or supply it if missing.

> And receivers really should not be adding one - especially if it is as you
> say, DKIM compliant.

I don't agree with the above. I think the core issue is the fact that
there is a great semantic difference between including an absent
header field name into the 'h' tag, and including an existing one.
The fact that DKIM uses the same syntactic mechanism to express
each is just a coincidence.

Requiring that some header field must not be added at all is a much
stronger requirement. It may be justifyable, but not casually. The
decision to assert such a strong requirement does not automatically
follow from a general site policy that some header field should be
signed when present.

Appliance/sw GUI that does so is misguiding the administrator.
Requiring a header field not to ever be present should be an expert setting.
A checkbox like 'sign this header field' should have a semantics 'sign it
if present' (at least by default).

> Sounds to me that people are implementing DKIM, and I hate to use the
> term "blindly", but thats exactly how it appears with this review.
> This doesn't help with the confidence that DKIM can be successful
> outside the direct 1 to 1 communications where there is less
> expectation for integrity issues and in my view, where the focus needs
> to be to give DKIM some real power, rather than trying to make this
> work broadly with Mailing List Server and 3rd party signers.

Most of the cases on my list deals with direct mail, not mailing lists.

Just some discipline would prevent several of these failures:
at least some minimal testing when deploying, keeping up with
reasonably recent versions of software, avoiding software which
ignores standards, and avoiding software which modifies an
otherwise correct passing mail for no good reason, often just
because someone though a message might look neater with
one or another gratuitous edit.

It would also help if it were easier to report signer mistakes to the
guilty part. With large ISPs or commerces I find it particularly difficult
or futile.

> The DKIM specification clearly says what headers SHOULD be considered
> and what SHOULD NOT.  I always felt there should a default and that h=
> was optional (to reduce the bandwidth), but having h= should help
> remove any guessing.

The invention of an 'h' tag made a tremendous improvement in survivability
of a DKIM signature over the initial DomainKeys drafts.

> > - signature includes Message-ID in h tag, but there was no Message-ID in
> >   the original message at the time of signing. When a receiving MX
> > inserts a missing header field, it breaks the signature.
>
> But if it was signed with one, then someone stripped it before the MX
> got it.

Not in cases I observed. There was no Message-ID at the time of signing.
It's easy to prove - just deleting the late-added Message-ID makes a
signature valid again.

> > - Postfix strips bare CR at end-of-line which invalidates a signature in
> > a message with embedded bare CRs at eol (e.g. with altermime html
> > disclaimers) (this is just one case of a garbage-in / garbage-out
> > principle, the lesson to be learned is that a mail to be signed should be
> > syntactically correct)
>
> This is an issue for many.  We are talking about the MS-DOS, UNIX vs
> MAC storage and/or display world.  This is one of the reasons why we
> have not implemented yet.

That was a lone incident of bare CR characters finding their way into
a message. It was due to a bug in a disclaimer-adding software.

I don't think that MS/Unix/Mac differences in line endings will present
a share of failures worth mentioning. So far I have yet to find a single
case of such breakage. The DKIM rfc is clear on it, and as long as
implementers do it by the book, this is a non-issue.

> Anyway good list. Wow!  Maybe not as bad as this list makes it, but to
> me, it shows DKIM - a mail integrity technology is not going to work
> very well in a open distribution environment. Not direct 1 to 1 with
> less expectation for failure.
> But if we don't give a hoot about failure, then whats the point of
> DKIM in the first place.
> Bad guys will simply not sign or just fake it even if they know will fail.
> Who is hurt by all this?
>     The SMTP mail receivers!!
> who are being asked to put up with this tolerance for failure, and
> only look for Mr. GoodBar in a crowd of misfits!

Despite my list of failures, I'm generally pleasantly surprised with
the survivability of DKIM signatures. Despite limited present use, signing
already offers genuine benefits, like easier and more reliable whitelisting,
reputation-based scoring, non-repudiation when filing abuse reports, etc.

> From my experience, Return-Path should not be included in the
> signature.  But then again, we started as a UUCP/UUCICO system where
> "From " was part of the model and was replaced with Return-Path only
> for the router for failure to delivery purposes.  But the router does
> not resend it back out. For us, its "meta" data.

Even though the listed case of a signed Return-Path was probably due
to a mistake and not an intentional deed, I found it interesting and
intriguing to use such approach of signing envelope information.
It's intriguing as the envelope sender address travels out-of-band with
a message, yet it is reassembled when LDA inserts the Return-Path.
If only the signer used the proper angle-bracketed address form as
specified by RFC2822, the signature would even have a chance of
surviving (a verifier at a MTA would need to provide a pseudo Return-Path).
I'm not suggesting it would be a good idea ... just some food for thought.

SM wrote:
> >- signature includes Message-ID in h tag, but there was no Message-ID in
> >   the original message at the time of signing. When a receiving MX
> > inserts a missing header field, it breaks the signature.
>
> That header field is a SHOULD.  It is not optional unless your view
> of implementation is restricted to "MUST".  That can be fixed at the
> message submission stage.

True. But a signer sw had no right to require that the header field
must not be added later, at least not in a default setting, without
explicitly asked to do so.

> >- sendmail reformats long lists of addresses in a To header field,
> >   which is why our site is not signing a To header field;
>
> Do that cause a verification failure?  If so, can you send me a test
> case off-list?

Sent.

> Thanks.  You mentioned a long list of addresses in the "To" header
> field.  I assumed that the problem was due to length of the header
> field.  That's not the case.  There is the "COMMAIZE" FEATURE in
> dkim-milter 2.8.0 which addresses the verification problem you
> described.

The approach of anticipating a munge only helps when the feature
is used in a signer in conjunction with the MTA which is expected to
do the modification. It does not help if a message is signed by some
other mailer and just happens to pass through a header-modifying MTA
on its was out (e.g. an outbound mail submission mailer for a SOHO
at an ISP), or on it way in (like a backup MX relay).

> I suggest that you sign the To: and Cc: header fields to prevent abuse. :-)

The To and Cc header fields are purely informational, they do not
affect mail delivery, and many (but not all) recipients have already
learned not to be confused by them - be it due to the use of Bcc,
or mailing lists, or just by a daily exposure to spam.

>From my position (verifying at a MTA, using the outcome as input
to whitelisting, to reputation schemes, and to classification)
a signed To and Cc brings no additional value. And being a likely
reason for a signature failure, I prefer it not to be signed.

> >- system time on the signing host is few minutes into the future,
> >   dkim-milter considers it an invalid signature
>
> There is a ClockDrift setting to deal with that.  People generally do
> not notice this problem when they debug verification failures.

Right, the default is 300 seconds. A MTA and a signing host should
be running NTP. Setting time manually and letting it free-run
is at least unrespectful and confusing to recipients or senders,
or can cause mail misclassification/blocking at worst.

John Levine wrote:
> Indeed.  We need deployment guides, but it's not clear to me whether
> they'd better be IETF documents or written somewhere else.
> I don't think they'd be very complicated or controversial, mostly a
> checklist of remember to do this, and not to do that.

Yes, a checklist of do-s/don't-s could prevent some of the more
common mistakes. Also, producers of signing appliances or sw
should make it hard for an average administrator to misconfigure
the signer. Unusual settings belong to 'expert settings' and
need an 'are you sure' and 'are your really sure' safety fuses.
Default settings should be good for most setups (a signature
requiring some header field not to be added later is not
generally a safe setting). 

  Mark

_______________________________________________
NOTE WELL: This list operates according to 
http://mipassoc.org/dkim/ietf-list-rules.html

---

From: SM <[email-masked]>
Date: Wed, 25 Mar 2009 20:19:26 -0700

Hi Mark,
>if present' (at least by default).

It's a bit more complicated than that.  Some header fields should be 
signed even when they are absent or else you leave the door open to 
abuse.  There are the "visible" header fields and some header fields 
which can be used to "repurpose" the message.  If you don't sign 
them, you leave the door open for abuse.

>guilty part. With large ISPs or commerces I find it particularly difficult

There is an I-D about DKIM reporting.

>of a DKIM signature over the initial DomainKeys drafts.

Agreed.

>It's easy to prove - just deleting the late-added Message-ID makes a

Section 5.5 of RFC 4871 recommends signing the Message-ID if it is 
present in the message being signed.  Instead of trying to make a 
signature valid again, it is better to fix the problem at the (signer) source.

>implementers do it by the book, this is a non-issue.

I've come across such cases.  I agree with you that this is a 
non-issue as there should always be CRLF at the end of the lines.

>explicitly asked to do so.

I prefer to see such issues addressed through guidelines instead of 
forcing people "to do the right thing" as we might not agree on what 
the correct behavior should be.  Default settings are a matter of 
tradeoffs.  The user-base might not find what we believe is right 
suitable for their environment.

>at an ISP), or on it way in (like a backup MX relay).

Yes, that would be a problem.  I don't have a solution for that. :-)

>or mailing lists, or just by a daily exposure to spam.

There is still room for social engineering.

>or can cause mail misclassification/blocking at worst.

The verifying host should also be running NTP.

Regards,
-sm

---

From: HLS <[email-masked]>
Date: Thu, 26 Mar 2009 09:18:38 -0400

> the RFC 5322, as many MTAs or MUAs expect it and/or supply it if missing.

Many systems, including ours, leave level of compliance up to the
operator.  Some systems, will abort the transaction (at DATA) when the
headers did not to either 822, 2822/5322.  Others will do the same
post SMTP (accept first). Some systems behavior as drop off points for
printers, faxing, text messagings, etc using simple clients where the
addressing is purely at the envelope.  Not the headers. Generally,
these systems required authenticated clients. The submission protocol
may also enforce the injection of headers.

I'm old school though, I really really really hate middle ware that
fool with passthru mail. <g> so if this is an original submission,
that is one thing, you have more ethical engineering acceptance and
authority to "adjust" the mail.  With relays, other than adding the
required trace headers, I always leaned on not "adjusting" the
headers.  It is really a taboo to do so.  I believe RFC 5322 reflects
this traditional practice.

> each is just a coincidence.

Well, I am quickly coming to cerment the reality that much of the
philosophical and discipline conflicts depends on how one expects to
use DKIM.

I come from an Fault Tolerance background - AI, Expert Systems,
process controllers, simulators, modeling, etc. where fault detection
is very much part of the working success of a system.

So if you (speaking in general) can't define how to handle DKIM
faults, I find to see how one can succesfully work in the handling of
DKIM whitelisting without eliminating or reducing the abuse of it.

So even if one wants to use DKIM for whitelisting, you still need to
protects against the faults of the WhiteListing models.

> of a DKIM signature over the initial DomainKeys drafts.

But thats because DKEYS signed all headers.  I was referrig to a
default set of headers.  For example, no h=, might mean the basic
fundamental headers:

   From:
   To:
   Subject:
   Date:

at the very least, something 100% of all mail systems have, and 99.99%
of all rendering devices ergonimically show to users.

[Note: I'm not proposing to make it optional]

> It's easy to prove - just deleting the late-added Message-ID makes a

Ok, I was scatching my head by what you mean by MX.  Do you mean the
MDA?  Sure, the MX does not have to be a MDA, it can be a router.

This all goes back to my first point above.  It is final destination
or a relay.  Problems begin when relays begin to tamper with mail.
But if its final destination, why would a DKIM-aware reciever add
something before it attempts to validate the signature, if any?

> implementers do it by the book, this is a non-issue.

Well, that depends on implementations and the frameworks of backends.
The variety is to great to expand on here.  But I will say,  this
issue has raised a barrier to adoption for us.   That is why the cost
of implementation DKIM must have a payoff if we are to justify the
design changes that are required to support it.   We are not going to
add DKIM for marketing reasons.  We don't need it.  We are not going
to add it for someone elses good (a trust service) unless there is a
Fault Handling involved.  But that just us.  I like to think others
are in similar positions, but I can't change the fact that DKIM would
be a very expensive implementation effort for us.

> the survivability of DKIM signatures.

Yes, which keeps people interested in its promise.

> Despite limited present use, signing already offers genuine benefits, like
> easier and more reliable whitelisting, reputation-based scoring, non-repudiation
> when filing abuse reports, etc.

You see, that is what I am talking about.  What about failures? <g>

This is really two different schools of though.  But in my view, its
all really part of the same paradigm,  failure and success are mirror
images of the same thing.  Using a chemisty methapor,  there are left
sided and right sided modular structure of the same modulars. One is
bad for humans (not ediable), one is.  We are a right-handed world.
:But they have the same molecular composition. :-)

> intriguing to use such approach of signing envelope information.

One of the reasons it is hard to work in these list issues is because
a good portion of it deals with implementation issues.  I think we
need to focus on the general aspect that applies to all as a maximum
(or possibly minimum).    Spell of the standards, the BCP and work
from there.

-- 
hls