Subject: [jose] JOSE and signed REST requests

From: Anders Rundgren <[email-masked]>
Date: Tue, 2 Aug 2016 07:32:08 +0200

Hi All,

I was recently involved in an inter-bank payment project based on a REST API.

Since my role was "cryptography" I recommended the following approach
http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html
since an operation is defined not only by the message payload, but also by the HTTP verb, URI, and header parameters.

The only related standards effort I'm aware of is this:
https://tools.ietf.org/html/draft-cavage-http-signatures-05

Unfortunately the methods above get rather awkward if you have a system where requests are supposed to be embedded in other messages or just proxied to another server.

I would rather have dropped REST in favor of transport-independent schemes using self-contained JSON-encoded signed message objects.

WDYT?

Anders

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 02 Aug 2016 08:12:03 +0200

> [...] an operation is defined not only by the message payload, but
> also by the HTTP verb, URI, and header parameters.

I'm don't think that this will help you with your payment project, but,
you said REST and didn't specifically ask for HTTP, so here is another
standards effort you might want to be aware of:

draft-selander-ace-object-security–05

In the Berlin CoRE WG meeting, there was in-room consensus to adopt this
as a WG document (to be confirmed on the mailing list), so this should
not be too far out.  [Sorry about the WG name confusion -- ACE is
focusing on how to do OAuth in constrained-node networks, but the
protected CoAP exchanges themselves actually are a CoRE WG issue, where
we use COSE formats for the actual protection.]

> I would rather have dropped REST in favor of transport-independent
> schemes using self-contained JSON-encoded signed message objects.

[Pet peeve: HTTP is not a transport protocol.]
This removes the REST abstraction that is quite useful to the
application designer.  Having the application designer create the
security protocol around signed message objects also has increased
potential for mistakes around freshness, context binding etc.  You get
to verify your whole application-layer message interaction against the
security objectives; you need a larger amount of luck with that than
with protected REST primitives.

Grüße, Carsten

---

From: Justin Richer <[email-masked]>
Date: Tue, 2 Aug 2016 06:43:49 -0400

There's also this approach:

https://tools.ietf.org/html/draft-ietf-oauth-signed-http-request-02

It's more limited than a general HTTP signing mechanism, but as a 
consequence it's more robust for systems that mess with the HTTP message 
in transit (which we know happens in the real world).

  -- Justin

On 8/2/2016 1:32 AM, Anders Rundgren wrote:
> since an operation is defined not only by the message payload, but 
> Unfortunately the methods above get rather awkward if you have a 
> system where requests are supposed to be embedded in other messages or 
> just proxied to another server.
>
> _______________________________________________
> jose mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jose

---

From: Martin Thomson <[email-masked]>
Date: Tue, 2 Aug 2016 21:10:26 +1000

Clearly this is a windmill that many like to tilt at.  After seeing
the long procession of failed attempts, my conclusion was that signing
HTTP messages in a generic fashion was an enterprise doomed to fail.

The main problem is that it's really hard to work out what to protect.
Header fields are especially difficult to protect because they lack a
stable encoding and they change in transit.  The first means that even
the most careful spec ends up with spurious signature failures; the
second means that things that should be protected, can't be.  In the
end, you get something with byzantine complexity, an overly
restrictive applicability, security failures, or combinations of all
three.

Taking the OAuth work and the "q" parameter:
 - the method for creating a canonical encoding of the query string is
quite complex,
 - but it's not complex enough yet because it doesn't handle percent-encoding;
 - also, if the "q" parameter is omitted, but the query string happens
to be security critical you get failures.

Clearly the more limited focus helps here, but the point is that you
have to accept those limitations, which is a whole new set of problems
for an implementation.  Say you successfully validate a request, that
means that you cannot pass anything that was unvalidated to the next
processing stage or you create a side-channel.  And are you sure that
there isn't some significance to the information you erased during the
validation process that might then be used by the next stage?

On 2 August 2016 at 20:43, Justin Richer <[email-masked]> wrote:
> consequence it's more robust for systems that mess with the HTTP message in
> transit (which we know happens in the real world).
>
>  -- Justin
>
>
>
>> since an operation is defined not only by the message payload, but also by
>> the HTTP verb, URI, and header parameters.
>>
>> Unfortunately the methods above get rather awkward if you have a system
>> where requests are supposed to be embedded in other messages or just proxied
>> to another server.
>>
>> I would rather have dropped REST in favor of transport-independent schemes
>> using self-contained JSON-encoded signed message objects.
>>
>> WDYT?
>>
>> Anders
>>

---

From: Justin Richer <[email-masked]>
Date: Tue, 2 Aug 2016 07:17:54 -0400

Those limitations are exactly what drove the work in this direction. I 
completely agree that it's always going to be a trade off, and we went 
in the direction of not having a universal solution but instead having a 
limited solution that would work for many (but not all) use cases. I've 
yet to see a universal solution that doesn't break under simple 
counter-examples.

The fact that you can have unsigned elements that could be critical to 
your application is one such limitation. In those cases, you need to 
make sure that not only do the signature and hash validate, but also 
that the elements you care about were protected by it. That doesn't mean 
you can't pass on other parameters to other layers, but it does mean 
that you can't guarantee they were left alone in transit. This is an 
important note in the security considerations section, along with 
parameter ordering.

Side note about the percent encoding, is the spec not clear enough about 
that as input for the hash calculation? Happy to take suggested text.

  -- Justin

---

From: Martin Thomson <[email-masked]>
Date: Tue, 2 Aug 2016 21:24:31 +1000

On 2 August 2016 at 21:17, Justin Richer <[email-masked]> wrote:
> that as input for the hash calculation?

It is very much underspecified.  You talk about strings, but header
fields are sequences of octets.  Percent-encoding is not mentioned at
all and URL normalization [RFC7230] is not mentioned either.  The text
on duplicated entries needs work (as noted).

> Happy to take suggested text.

I'm glad to hear it, but I apologize that I really can't spare the
time.  Hopefully I've given you a few pointers that you can use.

---

From: Sergey Beryozkin <[email-masked]>
Date: Tue, 2 Aug 2016 14:34:12 +0300

Hi Justin, Anders

in Apache CXF we have the filters for signing the outgoing payload.
Short overview:
http://cxf.apache.org/docs/jax-rs-jose.html#JAX-RSJOSE-JOSEJAX-RSFilters
JWS:

http://cxf.apache.org/docs/jax-rs-jose.html#JAX-RSJOSE-JWS

This is much less complete compared the http-request-02 work but we dpo 
focus on the integrity of the payload. I think it will be interesting 
for us to combine the http-request-02 (for ex the optional protection of 
the headers, etc) with the streaming approach employed to sign the 
data... Seems like a good opportunity for me to start looking at the
the http-request-02/etc work.

Thanks, Sergey

---

From: Mike Jones <[email-masked]>
Date: Tue, 2 Aug 2016 13:36:01 +0000

As background, people should be aware that at IETF 96, members of the OAuth working group expressed a preference to abandon work on the OAuth http signing spec or put it on hold.  This discussion and the results are recorded near the end of the minutes at https://www.ietf.org/proceedings/96/minutes/minutes-96-oauth.

So anyone considering taking a dependency on this spec should take into account that it will likely never become an RFC.

				-- Mike

-----Original Message-----
From: jose [mailto:[email-masked]] On Behalf Of Sergey Beryozkin
Sent: Tuesday, August 2, 2016 4:34 AM
To: [email-masked]
Subject: Re: [jose] JOSE and signed REST requests

Hi Justin, Anders

This is much less complete compared the http-request-02 work but we dpo focus on the integrity of the payload. I think it will be interesting for us to combine the http-request-02 (for ex the optional protection of the headers, etc) with the streaming approach employed to sign the data... Seems like a good opportunity for me to start looking at the the http-request-02/etc work.

Thanks, Sergey

On 02/08/16 13:43, Justin Richer wrote:
> consequence it's more robust for systems that mess with the HTTP 
> message in transit (which we know happens in the real world).
>
>  -- Justin
>
>
>> system where requests are supposed to be embedded in other messages 
>> or just proxied to another server.
>>

---

From: Justin Richer <[email-masked]>
Date: Tue, 2 Aug 2016 10:13:00 -0400

Or that there will be enough interest in the spec that it will either spur the OAuth working group to finish what it started, or the work will move elsewhere.

 — Justin

> On Aug 2, 2016, at 9:36 AM, Mike Jones <[email-masked]> wrote:
>

---

From: Anders Rundgren <[email-masked]>
Date: Tue, 2 Aug 2016 17:19:17 +0200

Thanx everybody for providing so much information, I really appreciate it!

FWIW, the self-contained "RESTless" request scheme I mentioned would in a HTTP context look like this:

     POST /transact HTTP/1.1                           // Always POST and to a single URL/partner
     User-Agent: Java
     Host: payments.mybank.com
     Content-Type: application/json
     Content-Length: 1234

     {
       "@context":"some-name-space",
       "@qualifier":"some-object-type",                  // "Operation"
          .
      other message properties
          .
       "signature":{}                                    // Covers the entire JSON object
     }

Including the target URL in the message seems like a reasonable addition which I have yet to perform.

Anders

---

From: Hoai Viet Nguyen <[email-masked]>
Date: Tue, 2 Aug 2016 18:16:00 +0200

Hi all,

we have developed an authentication scheme which address this topic. Our
scheme is a generic security approach for REST that relies on the same
abstraction level as the architectural style itself. The developed
authentication contains a general notation for describing REST message
elements including header fields and the payload. We used this abstract
notation to specify a policy defining security-relevant header fields
which require to be signed and verified by a corresponding signature and
verification algorithm. Both algorithms are also proposed by our work.
These generic authentication scheme containing the policy and the two
algorithms forms a guideline for implementing message authentication
technologies for REST-based protocols of any kind such as HTTP or CoAP.

Our work has been published in two papers. The first paper includes the
generic authentication scheme and the HTTP implementation. 
https://doi.org/10.1007/978-3-319-19210-9_8

The second paper contains the CoAP implementation of our authentication
scheme. This paper has been presented at the International Workshop on
Secure Internet of Things (SIOT) which has been held in conjunction with
the ESORICS 2015.
https://dx.doi.org/10.1109/SIOT.2015.8

In further works, we will also focus on concepts enabling intermediate
systems to interacting with signed REST messages. This includes the
consideration of changing header fields in transit, a problem which has
been addressed by Martin Thomson.

Regards and looking forward for your feedback,
Viet

Am 02.08.16 um 13:17 schrieb Justin Richer:
> in the direction of not having a universal solution but instead having
> a limited solution that would work for many (but not all) use cases.
> I've yet to see a universal solution that doesn't break under simple
> counter-examples.
>
> that the elements you care about were protected by it. That doesn't
> mean you can't pass on other parameters to other layers, but it does
> mean that you can't guarantee they were left alone in transit. This is
> an important note in the security considerations section, along with
> parameter ordering.
>
> Side note about the percent encoding, is the spec not clear enough
> about that as input for the hash calculation? Happy to take suggested
> text.
>
>  -- Justin
>
>
>>> message in
>>>> I was recently involved in an inter-bank payment project based on a
>>>> REST
>>>> API.
>>>>
>>>> also by
>>>> system
>>>> schemes

---

From: Anders Rundgren <[email-masked]>
Date: Tue, 2 Aug 2016 21:38:11 +0200

On 2016-08-02 18:16, Hoai Viet Nguyen wrote:

Hi Viet,

It is quite possible that I didn't follow the links properly but I actually ended up having to purchase the papers :-(

I was recently advised to buy an X.9 standard which is supposed to cover the security solutions required by the US financial industry.  Sounds pretty interesting, right?
However, I did not.  Not because a few hundred bucks represents a "budgetary problem", but due to the fact that paid for standards are difficult to handle (without violating the copyright) in a world powered by open source.

Regards
Anders

> Hi all,
>
> we have developed an authentication scheme which address this topic. Our scheme is a generic security approach for REST that relies on the same abstraction level as the architectural style itself. The developed authentication contains a general notation for describing REST message elements including header fields and the payload. We used this abstract notation to specify a policy defining security-relevant header fields which require to be signed and verified by a corresponding signature and verification algorithm. Both algorithms are also proposed by our work. These generic authentication scheme containing the policy and the two algorithms forms a guideline for implementing message authentication technologies for REST-based protocols of any kind such as HTTP or CoAP.
>
> Our work has been published in two papers. The first paper includes the generic authentication scheme and the HTTP implementation.
> The second paper contains the CoAP implementation of our authentication scheme. This paper has been presented at the International Workshop on Secure Internet of Things (SIOT) which has been held in conjunction with the ESORICS 2015.
> In further works, we will also focus on concepts enabling intermediate systems to interacting with signed REST messages. This includes the consideration of changing header fields in transit, a problem which has been addressed by Martin Thomson.
>
>> Those limitations are exactly what drove the work in this direction. I completely agree that it's always going to be a trade off, and we went in the direction of not having a universal solution but instead having a limited solution that would work for many (but not all) use cases. I've yet to see a universal solution that doesn't break under simple counter-examples.
>>
>> The fact that you can have unsigned elements that could be critical to your application is one such limitation. In those cases, you need to make sure that not only do the signature and hash validate, but also that the elements you care about were protected by it. That doesn't mean you can't pass on other parameters to other layers, but it does mean that you can't guarantee they were left alone in transit. This is an important note in the security considerations section, along with parameter ordering.
>>
>> Side note about the percent encoding, is the spec not clear enough about that as input for the hash calculation? Happy to take suggested text.
>>
>>  -- Justin
>>
>>