Subject: [Netconf] escaping EoM

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 08 Nov 2010 15:22:25 +0800

Hi,

Wes Hardaker probably found a counter-example against the EoM escaping
method:

]]>]]>]]>

This is because the middle "]]" is a common part of two chunks that are
candidates for escaping.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 08 Nov 2010 09:34:24 +0100 (CET)

> This is because the middle "]]" is a common part of two chunks that are

I think this works, but the alg. description is not really correct.
On the receiving side, the code needs to scan for ]]>]]> and ]]>]]] at
the same time.  When ]]>]]] is found, it is replaced with ]]>]], and
scanning continues.

So ]]>]]>]]> is replaced with ]]>]]]>]]>.  At the receiving side,
]]>]]] is found and replaced with ]]>]], then scanning continues with
>]]>, which is left untouched, resulting in ]]>]]>]]>, which we
>started with.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 08 Nov 2010 17:57:17 +0800

> >]]>, which is left untouched, resulting in ]]>]]>]]>, which we

I think the point of this method was to make sure the EoM marker doesn't
appear in the NETCONF message body so that the receiving side can
identify the frame limits a priori. This is not satisfied here.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Per Hedeland <[email-masked]>
Date: Mon, 08 Nov 2010 10:58:22 +0100

On 2010-11-08 09:34, Martin Bjorklund wrote:
> the same time.

Yeah, I guess that should be spelled out - you can't scan for ]]>]]>
until you find it before you scan for ]]>]]] - but you don't want to do
that anyway, since it would require buffering the whole PDU.

>  When ]]>]]] is found, it is replaced with ]]>]], and
> scanning continues.

Yes (i.e. it continues after the replacement).

>> ]]>, which is left untouched, resulting in ]]>]]>]]>, which we

Right. So perhaps a more elaborate description should be:

1 Transmission of PDU:

a) Look at the "next" 5 input characters.

b) If they are "]]>]]", replace with "]]>]]]", advance 5 positions in
input and go to a).

c) Otherwise, advance 1 position in input and go to a).

d) At end of input, append "]]>]]>".

2. Reception of PDU:

a) Look at the "next" 6 input characters.

b) If they are "]]>]]>", EoM has been found - discard EoM and "deliver"
PDU.

c) If they are "]]>]]]", replace with "]]>]]", advance 6 positions in
input and go to a).

d) Otherwise, advance 1 position in input and go to a).

(An actual implementation would presumably look at those 5/6 characters
one-by-one, allowing for the "Otherwise" case to be selected as soon as
any character in the first 4/5 doesn't match.)

--Per

---

From: Per Hedeland <[email-masked]>
Date: Mon, 08 Nov 2010 11:16:04 +0100

> identify the frame limits a priori. This is not satisfied here.

No, the (unstated) point of the method was only to provide a really
transparent transport. An additional (unstated) requirement was to
support streaming of the PDU. "Identifying the frame limits a priori" is
obviously at odds with streaming - it wasn't considered. But perhaps the
method could be "refined" to do "]]>" -> "]]>>" instead, at the cost of
some "unnecessary" rewriting.

--Per

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 08 Nov 2010 18:32:13 +0800

> support streaming of the PDU. "Identifying the frame limits a priori" is

What do you mean by "streaming the PDU"? If we don't care about
layering, the most straightforward method on the receiver side would be
to parse the XML stream properly right away - which has to be done in
any case, sooner or later - and the EOM would then become a non-issue.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Per Hedeland <[email-masked]>
Date: Mon, 08 Nov 2010 11:51:05 +0100

On 2010-11-08 11:32, Ladislav Lhotka wrote:
> What do you mean by "streaming the PDU"?

The sending side of the transport should be able to put bits on the wire
without waiting for the complete PDU to arrive from the Messages layer.
The receiving side of the transport should be able to deliver bits to
the Messages layer without waiting for the complete PDU to arrive on the
wire. s/PDU/Message/ I guess.

> If we don't care about

That may be true - the context of the suggestion was that we did care
about layering, and that the transport would be ignorant of XML.

--Per

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 08 Nov 2010 23:09:54 +0800

> wire. s/PDU/Message/ I guess.

The multibyte EOM makes this complicated because the problematic
sequences may be split between subsequent frames. For this particular
purpose, the single-byte EOM that must not occur at all in the message
is IMO much easier to deal with.
 
Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Per Hedeland <[email-masked]>
Date: Mon, 08 Nov 2010 16:57:56 +0100

On 2010-11-08 16:09, Ladislav Lhotka wrote:
> sequences may be split between subsequent frames.

Yes, somewhat - this is the case even without the escaping, of course.

> For this particular

*If* it can be guaranteed that the single-byte EOM never will occur in a
message (and hence won't need escaping), yes - I think there was some
mention of the possibility of unchecked user input including it.

But anyway, my understanding is that the WG has decided to postpone the
discussion/decision of any changes needed to the currently defined
mechanism - it is probably not useful to keep debating the relative
merits of different alternatives until there is an actual "target" for
the discussion.

--Per

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 09 Nov 2010 00:11:32 +0800

> mention of the possibility of unchecked user input including it.

No escaping is needed at all, any message containing it will be rejected
as not being well-formed.

> merits of different alternatives until there is an actual "target" for

Sure, but Wes had a good point with his example.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Phil Shafer <[email-masked]>
Date: Mon, 8 Nov 2010 10:50:21 -0500

>is IMO much easier to deal with.

+1, with another appeal to find a better way before this protocol
becomes widely deployed.

Thanks,
 Phil

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 8 Nov 2010 17:24:52 +0100

> +1, with another appeal to find a better way before this protocol

And your proposal how to transition without breaking existing systems
is? Or is the argument the number is not big yet, so we have a flag
day?

/js

PS: If we ever fix this, we should consider adding a mechanism in
    which client and server exchange an ASCII CRLF terminated line
    with version information etc. before they start throwing XML
    documents at each other. This will enable us to actually change
    the version in the XML namespace at some point in time - currently
    that is not possible. SSH pretty much does it this way and it is a
    cool solution to versioning issues - greet each other at the most
    basic language before you engage into a more complex dialogue.

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Phil Shafer <[email-masked]>
Date: Mon, 8 Nov 2010 11:16:31 -0500

>is? Or is the argument the number is not big yet, so we have a flag

Create a new ssh subsystem named "netconf-eom" that uses the single
byte 0x04 as the EOM, reachable over the existing NETCONF port.

While this is not the real message framing that I proposed, Lada's
idea of using a single non-XML byte is by far the more simple
solution, allows the existing transport to operation in parallel,
and allows the "]]>]]>" to be deprecated asap.  The cost is opening
a second connection if the server does not implement the new eom
mechanism, and clients can cache this knowledge of what transport
mapping each server implements to reduce this cost.

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 09 Nov 2010 00:46:37 +0800

> mapping each server implements to reduce this cost.

Right, and it does fix a bug, so it is well within the mandate of
4742bis.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Mon, 08 Nov 2010 08:54:08 -0800

>      basic language before you engage into a more complex dialogue.

I disagree.
We have a <hello> exchange in 4741bis that determines if both peers agree
to use base:1.1, base:1.0, or no agreement and the session is closed.

There is no flag day if the new transport uses a different subsystem.

There are other issues though.

A new server is going to need to listen on the old and new subsystem.
This is problematic in openssh:

    Port 830
    Subsystem netconf /usr/sbin/netconf-subsystem
    Subsystem netconf-eom /usr/bin/netconf-eom-subsystem

This sort of hack is needed so the netconf server will know which subsystem was
actually requested by the client.

Another concern: Is base:1.0 over netconf-eom allowed/required/optional?
That is more complexity and more server variants than I want.
By the time this is done there will be base:1.0, base:1.1, and base:2.0,
plus 2 different subsystems.  Will all combinations be supported?

IMO, the operational problems created by EOM changes are almost as bad as the
operational problems perceived to exist already with the ]]>]]> sequence.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 09 Nov 2010 01:12:04 +0800

> operational problems perceived to exist already with the ]]>]]> sequence.

This will be true only till the first exploit. In contrast, the new
transport means controlled damage.

Lada

> 
> 
> Andy
> 
> 
> _______________________________________________
> Netconf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netconf

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 8 Nov 2010 18:50:50 +0100

> mapping each server implements to reduce this cost.

Late evening idea: What does RFC4741 say concerning empty messages?
Would it work to send ]]>]]> or even ]]>]]>]]>]]> at the very
beginning of a session, that is before the <hello/>, as an indicator
that a new framing mechanism is supported? Would current 4741bis
implementations simply skip over this? If so, we could perhaps avoid
introducing a new subsystem.

/js

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 08 Nov 2010 19:04:07 +0100 (CET)

Phil Shafer <[email-masked]> wrote:
> solution,

Yes it is the simplest idea.  Note though that 0x04 is legal in XML
1.1, so if we ever want to support XML 1.1, this will break.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 08 Nov 2010 19:23:58 +0100 (CET)

> Late evening idea: What does RFC4741 say concerning empty messages?

It says that messages are XML documents, and xml documents cannot be
empty:

$ touch foo
$ xmllint foo
foo:1: parser error : Document is empty

^
foo:1: parser error : Start tag expected, '<' not found

^

> implementations simply skip over this?

Nope, sorry.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 08 Nov 2010 22:14:00 +0100 (CET)

> 1.1, so if we ever want to support XML 1.1, this will break.

Correcting myself: 0x04 is allowed, but not unescaped; only as a
character reference.  So 0x04 should be ok.  Sorry about the noise.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Mon, 8 Nov 2010 13:37:44 -0800

Hi Martin,

Illegal in XML 1.1?
Then this is not a lasting solution, is it?
Being tied to a someday-obsolete version of XML is not an option.
What if libxml2 stops supporting XML 1.0 in the future?

Andy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Martin Bjorklund
Sent: Monday, November 08, 2010 10:04 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [Netconf] escaping EoM

/martin

---

From: "Ersue, Mehmet (NSN - DE/Munich)" <[email-masked]>
Date: Tue, 9 Nov 2010 03:16:37 +0100

Folks,

please consider that the current discussion on EOM handling is not for
1.0.
So, we may think a bit broader than the 1.0 borders.

We can use the discussion result for a 2.0 though there is no plan or
charter for this work yet.

Cheers,
Mehmet

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of ext Andy Bierman
> Sent: Monday, November 08, 2010 10:38 PM
> To: Martin Bjorklund; [email-masked]
> Cc: [email-masked]

---

From: Phil Shafer <[email-masked]>
Date: Mon, 8 Nov 2010 22:11:12 -0500

>charter for this work yet.

Why is this not for 1.1 (aka -bis)?

Thanks,
 Phil

---

From: "Ersue, Mehmet (NSN - DE/Munich)" <[email-masked]>
Date: Tue, 9 Nov 2010 06:51:06 +0100

An appropriate solution produces a version incompatible with 1.0 
and needs to be addressed by more than one -bis (4741, 4742).
So, this is effectively 2.0.

Cheers,
Mehmet

> From: ext Phil Shafer [mailto:[email-masked]]
> Sent: Tuesday, November 09, 2010 4:11 AM
> To: Ersue, Mehmet (NSN - DE/Munich)
> Cc: ext Andy Bierman; Martin Bjorklund; ext Per Hedeland; Ladislav
> Lhotka; [email-masked]