Subject: [Netconf] WG Last Call on one more change to RFC4742bis

From: "Bert (IETF) Wijnen" <[email-masked]>
Date: Mon, 29 Nov 2010 16:47:41 +0100

Dear WG participants,

This is a 2 week WG Last Call for WG has consensus on
a late proposal to add proper/decent framing to the
SSH transport layer.

Please comment/respond before 14 December 2010

as you know, rfc4742bis has already been submitted to our AD
for and to be published as a PS RFC, obsoleting rfc4742.

At our IETF session in Beijing, there was more discussion on
the EOM sequence and on possible solutions in a future version.
That resulted in some hallway discussions to try and figure out
if we could find a way to do proper framing now, while being
backwards compatible with the rfc4742. Your WG chairs were
only willing to consider this if we can find real WG consensus
on such a change. We have had some offline mail discussions
with most (in fact we think all) implementers of the code, and
as a result, it seems there is a proposal on the table now that
we as a WG can reach consensus on. The proposal is below,
in the form of proposed TEXT edits to the latest
draft-ietf-netconf-rfc4742bis-04.txt. We have asked our
AD to hold that document for a short period of time, to see
if we can indeed get consensus on the below. If we do, then
we want to update the draft-ietf-netconf-rfc4742bis-04.txt
document before our AD sends it to IETF Last Call.

As you can see, the proposal makes it a little bit harder
to do just an SSH-session and then do cut-and-paste input.
We discussed this also, and the implementers now believe that
the proposed solution for proper/decent framing is acceptable
and that most implementation can/will provide a simple
scripting front-end to allow for some form of cut-and-paste.

Bert and Mehmet
---------
Here follow the proposed text-edits to
draft-ietf-netconf-rfc4742bis-04.txt

OLD (in 3.1):

    S:<?xml version="1.0" encoding="UTF-8"?>
    S:<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    S:<capabilities>
    S:<capability>
    S:       urn:ietf:params:xml:ns:netconf:base:1.0
    S:</capability>
    S:<capability>
    S:       urn:ietf:params:ns:netconf:capability:startup:1.0
    S:</capability>
    S:</capabilities>
    S:<session-id>4<session-id>
    S:</hello>
    S: ]]>]]>

    C:<?xml version="1.0" encoding="UTF-8"?>
    C:<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    C:<capabilities>
    C:<capability>
    C:       urn:ietf:params:xml:ns:netconf:base:1.0
    C:</capability>
    C:</capabilities>
    C:</hello>
    C: ]]>]]>

NEW:

    S:<?xml version="1.0" encoding="UTF-8"?>
    S:<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    S:<capabilities>
    S:<capability>
    S:       urn:ietf:params:xml:ns:netconf:base:1.1
    S:</capability>
    S:<capability>
    S:       urn:ietf:params:ns:netconf:capability:startup:1.0
    S:</capability>
    S:</capabilities>
    S:<session-id>4<session-id>
    S:</hello>
    S: ]]>]]>

    C:<?xml version="1.0" encoding="UTF-8"?>
    C:<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    C:<capabilities>
    C:<capability>
    C:       urn:ietf:params:xml:ns:netconf:base:1.1
    C:</capability>
    C:</capabilities>
    C:</hello>
    C: ]]>]]>

Delete this from 3.1:

    As the previous example illustrates, a special character sequence,
    ]]>]]>, MUST be sent by both the NETCONF client and the NETCONF
    server after each XML document in the NETCONF exchange.  This
    character sequence MUST NOT appear in an XML document.

NEW (right after the first paragraph in Section 4.):

4.1.  Framing protocol

   The previous version of this document defined the character sequence
   "]]>]]>" as a message separator, under the assumption that it could
   not be found in well-formed XML documents.  However, this assumption
   is not correct.  It can legally appear in XML attributes, comments,
   and processing instructions.  In order to solve this problem, and at
   the same time be compatible with existing implementations, this
   document defines the following framing protocol.

   The<hello>  message MUST be followed by the character sequence
   "]]>]]>".  Upon reception of the<hello>  message, the receiving
   peer's SSH Transport layer conceptually passes the<hello>
   message to the Messages layer.  If the :base:1.1 capability is
   advertised by both peers, the chunked transfer mechanism (see
   Section 4.1.1) is used for the remainder of the NETCONF
   session.  Otherwise, the old end-of-message based mechanism
   (see Section 4.1.2) is used.

4.1.1.  Chunked Framing Mechanism

   This mechanism encodes all NETCONF messages with a chunked
   framing.  Specifically, the message follows the ABNF [RFC 5234]
   rule Chunked-Message:

       Chunked-Message = 1*chunk
                         end-of-chunks

       chunk           = LF HASH chunk-size LF
                         chunk-data
       chunk-size      = 1*DIGIT1 0*DIGIT
       chunk-data      = 1*OCTET

       end-of-chunks   = LF HASH HASH LF

       DIGIT1          = %x31-39
       DIGIT           = %x30-39
       HASH            = %x23
       LF              = %x0A
       OCTET           = %x00-FF

    The chunk-size field is a string of decimal digits indicating the
    number of octets in chunk-data.  Leading zeros are prohibited, and
    the maximum allowed chunk-size value is 4294967295.

    As an example, the message:

      <rpc message-id="102"
           xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
        <close-session/>
      </rpc>

    could be encoded as:

      #4
      <rpc
      #18
       message-id="102"

      #79
           xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
        <close-session/>
      </rpc>
      ##

   Conceptually, the SSH Transport layer encodes messages sent by the
   Messages layer, and decodes messages received on the SSH channel
   before passing them to the Messages layer.

   If an error occurs during the decoding process, the peer MUST
   terminate the NETCONF session by closing the corresponding SSH
   channel.

4.1.2.  End-of-message Framing Mechanism

   This mechanism exists for backwards compatibility with
   implementations of previous versions of this document.  It is only
   used when the remote peer does not advertise the :base:1.1
   capability.

   When this mechanism is used, the special character sequence
   "]]>]]>", MUST be sent by both the NETCONF client and the NETCONF
   server after each message (XML document) in the NETCONF exchange.
   Conceptually, the SSH Transport layer passes any data found in
   between the "]]>]]>" characters to the Messages layer.

OLD (section 4):

    To continue the example given above, a NETCONF over SSH session to
    retrieve a set of configuration information might look like this:

    C:<?xml version="1.0" encoding="UTF-8"?>
    C:<rpc message-id="105"
    C: xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    C:<get-config>
    C:<source><running/></source>
    C:<config xmlns="http://example.com/schema/1.2/config">
    C:<users/>
    C:</config>
    C:</get-config>
    C:</rpc>
    C: ]]>]]>

    S:<?xml version="1.0" encoding="UTF-8"?>
    S:<rpc-reply message-id="105"
    S: xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    S:<config xmlns="http://example.com/schema/1.2/config">
    S:<users>
    S:<user><name>root</name><type>superuser</type></user>
    S:<user><name>fred</name><type>admin</type></user>
    S:<user><name>barney</name><type>admin</type></user>
    S:</users>
    S:</config>
    S:</rpc-reply>
    S: ]]>]]>

NEW:

    The following example shows a capability exchange.  Data sent by the
    NETCONF client are marked with "C:" and data sent by the NETCONF
    server are marked with "S:".

    C: #271
    C:<?xml version="1.0" encoding="UTF-8"?>
    C:<rpc message-id="105"
    C:      xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    C:<get-config>
    C:<source><running/></source>
    C:<config xmlns="http://example.com/schema/1.2/config">
    C:<users/>
    C:</config>
    C:</get-config>
    C:</rpc>
    C:
    C: ##

    S: #404
    S:<?xml version="1.0" encoding="UTF-8"?>
    S:<rpc-reply message-id="105"
    S:            xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    S:<config xmlns="http://example.com/schema/1.2/config">
    S:<users>
    S:<user><name>root</name><type>superuser</type></user>
    S:<user><name>fred</name><type>admin</type></user>
    S:<user><name>barney</name><type>admin</type></user>
    S:</users>
    S:</config>
    S:</rpc-reply>
    S:
    S: ##

OLD (Section 5):

    C:<?xml version="1.0" encoding="UTF-8"?>
    C:<rpc message-id="106"
    C: xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    C:<close-session/>
    C:</rpc>
    C: ]]>]]>

    S:<?xml version="1.0" encoding="UTF-8"?>
    S:<rpc-reply id="106"
    S: xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    S:<ok/>
    S:</rpc-reply>
    S: ]]>]]

NEW:

    C: #141
    C:<?xml version="1.0" encoding="UTF-8"?>
    C:<rpc message-id="106"
    C:      xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    C:<close-session/>
    C:</rpc>
    C:
    C: ##

    S: #140
    S:<?xml version="1.0" encoding="UTF-8"?>
    S:<rpc-reply id="106"
    S:            xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    S:<ok/>
    S:</rpc-reply>
    S:
    S: ##

OLD (Section 6):

    RFC 4742 assumes that the EOM sequence cannot appear in any XML
    document, which turned out to be mistaken.  The EOM sequence can
    cause operational problems and open space for attacks if sent
    deliberately in an XML document.  It is however believed that the
    associated threat is not very high.  This document keeps using the
    EOM sequence to avoid incompatibility with existing implementations.
    Implementations of SSH transport must exercise extra care and make
    sure the EOM sequence is not present in an XML document.

NEW:

    RFC 4742 assumes that the EOM sequence "]]>]]>" cannot appear in
    any well-formed XML document, which turned out to be mistaken.  The
    EOM sequence can cause operational problems and open space for
    attacks if sent deliberately in RPC messages.  It is however
    believed that the associated threat is not very high.  This
    document still uses the EOM sequence for the initial<hello>
    message to avoid incompatibility with existing implementations.
    When both peers implement this specification, a proper framing
    protocol is used for the rest of the NETCONF session, to avoid
    injection attacks.

---

From: Andy Bierman <[email-masked]>
Date: Mon, 29 Nov 2010 08:14:06 -0800

> Please comment/respond before 14 December 2010

This seems OK.
The NEW examples for section 4 and 5 (at the end)
have an extra NL in them after the last chunk and before
the EOM (##).  They should be the same as the example in 4.1.

Andy

> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf
>
>

---

From: Andy Bierman <[email-masked]>
Date: Mon, 29 Nov 2010 08:20:48 -0800

> the EOM (##). They should be the same as the example in 4.1.

oops -- got it backwards but the examples are confusing.
The example in 4.1.1.1 does not show the old encoding
so we cannot tell if </rpc> was followed by a new line or not.

In the examples that have OLD and NEW encoding, it is clear that
the rpc end tag is followed by a new line so the extra line in
the new encoding is correct.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 30 Nov 2010 12:58:54 +0100 (CET)

Hi,

It seems some white-spaces got messed up somewhere in transit from me
to Bert to the ML, so here's a slightly updated version of the diffs:

/martin

OLD (in 3.1):

   S: <?xml version="1.0" encoding="UTF-8"?>
   S: <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
   S:   <capabilities>
   S:     <capability>
   S:       urn:ietf:params:xml:ns:netconf:base:1.0
   S:     </capability>
   S:     <capability>
   S:       urn:ietf:params:ns:netconf:capability:startup:1.0
   S:     </capability>
   S:   </capabilities>
   S:   <session-id>4<session-id>
   S: </hello>
   S: ]]>]]>

   C: <?xml version="1.0" encoding="UTF-8"?>
   C: <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
   C:   <capabilities>
   C:     <capability>
   C:       urn:ietf:params:xml:ns:netconf:base:1.0
   C:     </capability>
   C:   </capabilities>
   C: </hello>
   C: ]]>]]>

NEW:

   The following example shows a capability exchange.  Data sent by the
   NETCONF client are marked with "C: " and data sent by the NETCONF
   server are marked with "S: ".

   S: <?xml version="1.0" encoding="UTF-8"?>
   S: <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
   S:   <capabilities>
   S:     <capability>
   S:       urn:ietf:params:xml:ns:netconf:base:1.1
   S:     </capability>
   S:     <capability>
   S:       urn:ietf:params:ns:netconf:capability:startup:1.0
   S:     </capability>
   S:   </capabilities>
   S:   <session-id>4<session-id>
   S: </hello>
   S: ]]>]]>

   C: <?xml version="1.0" encoding="UTF-8"?>
   C: <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
   C:   <capabilities>
   C:     <capability>
   C:       urn:ietf:params:xml:ns:netconf:base:1.1
   C:     </capability>
   C:   </capabilities>
   C: </hello>
   C: ]]>]]>

Delete this from 3.1:

4.1.  Framing protocol

  The <hello> message MUST be followed by the character sequence
  "]]>]]>".  Upon reception of the <hello> message, the receiving
  peer's SSH Transport layer conceptually passes the <hello>
  message to the Messages layer.  If the :base:1.1 capability is
  advertised by both peers, the chunked transfer mechanism (see
  Section 4.1.1) is used for the remainder of the NETCONF
  session.  Otherwise, the old end-of-message based mechanism
  (see Section 4.1.2) is used.

      DIGIT1          = %x31-39
      DIGIT           = %x30-39
      HASH            = %x23
      LF              = %x0A
      OCTET           = %x00-FF

     <rpc message-id="102"

   could be encoded as:

     #4
     <rpc
     #18
      message-id="102"
     
     #79

OLD (section 4):

   C: <?xml version="1.0" encoding="UTF-8"?>
   C: <rpc message-id="105"
   C: xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
   C:   <get-config>
   C:     <source><running/></source>
   C:     <config xmlns="http://example.com/schema/1.2/config">
   C:      <users/>
   C:     </config>
   C:   </get-config>
   C: </rpc>
   C: ]]>]]>

   S: <?xml version="1.0" encoding="UTF-8"?>
   S: <rpc-reply message-id="105"
   S: xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
   S:   <config xmlns="http://example.com/schema/1.2/config">
   S:     <users>
   S:       <user><name>root</name><type>superuser</type></user>
   S:       <user><name>fred</name><type>admin</type></user>
   S:       <user><name>barney</name><type>admin</type></user>
   S:     </users>
   S:   </config>
   S: </rpc-reply>
   S: ]]>]]>

NEW:

   The following example shows a capability exchange.  Data sent by the
   NETCONF client are marked with "C: " and data sent by the NETCONF
   server are marked with "S: ".

   C: #271
   C: <?xml version="1.0" encoding="UTF-8"?>
   C: <rpc message-id="105"
   C:      xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
   C:   <get-config>
   C:     <source><running/></source>
   C:     <config xmlns="http://example.com/schema/1.2/config">
   C:      <users/>
   C:     </config>
   C:   </get-config>
   C: </rpc>
   C:
   C: ##

   S: #404
   S: <?xml version="1.0" encoding="UTF-8"?>
   S: <rpc-reply message-id="105"
   S:            xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
   S:   <config xmlns="http://example.com/schema/1.2/config">
   S:     <users>
   S:       <user><name>root</name><type>superuser</type></user>
   S:       <user><name>fred</name><type>admin</type></user>
   S:       <user><name>barney</name><type>admin</type></user>
   S:     </users>
   S:   </config>
   S: </rpc-reply>
   S:
   S: ##

OLD (Section 5):

   C: <?xml version="1.0" encoding="UTF-8"?>
   C: <rpc message-id="106"
   C: xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
   C:   <close-session/>
   C: </rpc>
   C: ]]>]]>

   S: <?xml version="1.0" encoding="UTF-8"?>
   S: <rpc-reply id="106"
   S: xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
   S:   <ok/>
   S: </rpc-reply>
   S: ]]>]]

NEW:

   C: #141
   C: <?xml version="1.0" encoding="UTF-8"?>
   C: <rpc message-id="106"
   C:      xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
   C:   <close-session/>
   C: </rpc>
   C:
   C: ##

   S: #140
   S: <?xml version="1.0" encoding="UTF-8"?>
   S: <rpc-reply id="106"
   S:            xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
   S:   <ok/>
   S: </rpc-reply>
   S:
   S: ##

OLD (Section 6):

NEW:

   document still uses the EOM sequence for the initial <hello>

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 30 Nov 2010 13:07:24 +0100 (CET)

Hi again,

I accidentally hit send before the text was ready.  Here is the new
text.

/martin

OLD (in 3.1):

NEW:

Delete this from 3.1:

4.1.  Framing protocol

      DIGIT1          = %x31-39
      DIGIT           = %x30-39
      HASH            = %x23
      LF              = %x0A
      OCTET           = %x00-FF

     <rpc message-id="102"

   could be encoded as:

     #4
     <rpc
     #18
      message-id="102"
     
     #79

OLD (section 4):

NEW:

   C: #271

   S: #404

OLD (Section 5):

NEW:

   C: #141

   S: #140

OLD (Section 6):

NEW:

---

From: Phil Shafer <[email-masked]>
Date: Tue, 30 Nov 2010 10:01:49 -0500

>I accidentally hit send before the text was ready.  Here is the new

Absolutely, completely, and in all other ways digging it.

Thanks,
 Phil

---

From: Kent Watsen <[email-masked]>
Date: Tue, 30 Nov 2010 11:03:03 -0800

I very much like the chunked strategy being in 4742bis, just a few nits:

1. In section 4.1.1, would following be more clear?

      Chunked-Message = 1*chunk end-of-chunks
      chunk           = LF "#" chunk-size LF chunk-data
      chunk-size      = %x00000000-FFFFFFFF
      chunk-data      = 1*OCTET
      end-of-chunks   = LF "##" LF

    The three mechanisms I'm using to collapse this are:
      1. leverage OCTET and LF already being defined in RFC 5234
      2. inline the HASH definition as a "#" character
      3. define chunk-size using a 4-byte hex value

2. update to section 4.1.2

   "It is only used when the remote peer does not advertise the :base:1.1 capability."

   Should the language match that used in 4.1 about the :base:1.1 capability being
   advertised by both peers?  I understand that it's assumed that the implementer
   is presumably already advertising ":base:1.1, so it's really just the remote side
   that matters, but it took me a few seconds to realize that, which suggests that the
   the text could be clarified

3. in the examples in sections 4 and 5, is there a missing LF at the beginning of each message?  For
   instance, should it be

     C:
     C: #271

  instead of:

     C: #271

Thanks,
Kent

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Martin Bjorklund
Sent: Tuesday, November 30, 2010 7:07 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [Netconf] WG Last Call on one more change to RFC4742bis

Hi again,

/martin

OLD (in 3.1):

NEW:

Delete this from 3.1:

4.1.  Framing protocol

      DIGIT1          = %x31-39
      DIGIT           = %x30-39
      HASH            = %x23
      LF              = %x0A
      OCTET           = %x00-FF

     <rpc message-id="102"

   could be encoded as:

     #4
     <rpc
     #18
      message-id="102"
     
     #79

OLD (section 4):

NEW:

   C: #271

   S: #404

OLD (Section 5):

NEW:

   C: #141

   S: #140

OLD (Section 6):

NEW:

---

From: Andy Bierman <[email-masked]>
Date: Tue, 30 Nov 2010 12:18:26 -0800

Hi,

We do not want to allow leading zeros.
We do not want to use hex encoding.
I like the original proposal posted by Martin better than this rewrite.

Andy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Kent Watsen
Sent: Tuesday, November 30, 2010 11:03 AM
To: Martin Bjorklund; [email-masked]
Cc: [email-masked]
Subject: Re: [Netconf] WG Last Call on one more change to RFC4742bis

     C:
     C: #271

  instead of:

     C: #271

Thanks,
Kent

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Martin Bjorklund
Sent: Tuesday, November 30, 2010 7:07 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [Netconf] WG Last Call on one more change to RFC4742bis

Hi again,

/martin

OLD (in 3.1):

NEW:

Delete this from 3.1:

4.1.  Framing protocol

      DIGIT1          = %x31-39
      DIGIT           = %x30-39
      HASH            = %x23
      LF              = %x0A
      OCTET           = %x00-FF

     <rpc message-id="102"

   could be encoded as:

     #4
     <rpc
     #18
      message-id="102"
     
     #79

OLD (section 4):

NEW:

   C: #271

   S: #404

OLD (Section 5):

NEW:

   C: #141

   S: #140

OLD (Section 6):

NEW:

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 30 Nov 2010 21:25:37 +0100 (CET)

Hi,

>       1. leverage OCTET and LF already being defined in RFC 5234

I don't think it is legal to just use the definitions from Appendix B
of RFC 5234.  They must be copied.

>       2. inline the HASH definition as a "#" character

This is a matter of taste, it doesn't change the protocol.  I don't
have a strong opinion.

>       3. define chunk-size using a 4-byte hex value

This is not what we had in mind.  We wanted the size to be encoded
using decimal digits as a string, e.g. the size 42 would be encoded as
the string "42", not as the single byte 42.

>      C: <?xml version="1.0" encoding="UTF-8"?>

Yes this is correct.

/martin

---

From: Kent Watsen <[email-masked]>
Date: Tue, 30 Nov 2010 13:02:42 -0800

> of RFC 5234.  They must be copied.

I wasn't sure either, I checked RFC 5234 and didn't find a statement saying they had to be copied but I do see other RFCs doing it, so I guess it's standard practice to copy the core rules

> This is a matter of taste, it doesn't change the protocol.  I don't

It only mattered so long as we didn't copy the Core Rules; so long as we do, then any attempt to compress the syntax is lost

> the string "42", not as the single byte 42.

Ah, good point, thanks.

> Yes this is correct.

Whew, at least I made one helpful comment ;)