Subject: Re: [Netconf] ssh EOM and netconf 1.1

From: "Andy Bierman" <[email-masked]>
Date: Wed, 10 Nov 2010 12:33:45 +0800

I think Wed asked what problem we are solving here.  I would like to know as well.

Using the old framing at all means a hacker can still figure out how to inject rpcs into the message stream using legal XML.

Using a magic PI is a complicated hack.  Not sure why it helps.

A client would have to wait until the server hello was processed before sending the first rpc to use the correct framing.  

Wed is right.  Mixing framing and data doesn't work. 

Andy

----- Reply message -----
From: "Juergen Schoenwaelder" <[email-masked]>
Date: Wed, Nov 10, 2010 11:07
Subject: [Netconf] ssh EOM and netconf 1.1
To: <[email-masked]>

Hi,

I am trying to find a solution that allows to fix the unfortunate EOM
marker without requiring a new SSH subsystem. Would the following
work:

a) We allocate a new EOM marker (e.g. 0x04) we believe is better than
   what we have now.

b) NETCONF client and server exchange hello messages in the
   traditional way, using the ]]>]]> EOM marker.

c) If a peer announces the capability urn:ietf:params:netconf:base:1.1,
   this is taken as the signal to switch from the ]]>]]> EOM marker to
   the new EOM marker (e.g. 0x04).

Since the hello exchange does not carry any user supplied data, the
usage of the traditional EOM marker during the hello exchange should
not be a big deal.

Would something like this work?

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
_______________________________________________
Netconf mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netconf

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 10 Nov 2010 05:52:22 +0100

On Wed, Nov 10, 2010 at 12:33:45PM +0800, Andy Bierman wrote:
 
> Using the old framing at all means a hacker can still figure out how
> to inject rpcs into the message stream using legal XML.

It is hard to inject something into the <hello>. We won't fix the
existing 1.0 transport, we can however fix the 1.1 transport.
 
> A client would have to wait until the server hello was processed
> before sending the first rpc to use the correct framing.

Yep. Big deal? Isn't it a good thing to wait until you know the
receiver will understand what you are saying. Isn't that why we have
the <hello> in the first place?

> Wed is right.  Mixing framing and data doesn't work. 

I don't understand. You can look at this like the message layering
figuring out the peer is 1.1 and telling the transport to switch to
1.1 framing. If you take this viewpoint, there is no mixing of framing
and data.

/js

---

From: Andy Bierman <[email-masked]>
Date: Tue, 9 Nov 2010 20:54:06 -0800

Oops – I meant Wes, not Wed – my Android phone changed the spelling.
Thought I caught all those, but I guess not.  Need to disable that broken feature.

Andy

From: [email-masked] [mailto:[email-masked]] On Behalf Of Andy Bierman
Sent: Tuesday, November 09, 2010 8:34 PM
To: Juergen Schoenwaelder; [email-masked]
Subject: Re: [Netconf] ssh EOM and netconf 1.1

Andy

----- Reply message -----
From: "Juergen Schoenwaelder" <[email-masked]>
Date: Wed, Nov 10, 2010 11:07
Subject: [Netconf] ssh EOM and netconf 1.1
To: <[email-masked]>

Hi,

/js

---

From: Andy Bierman <[email-masked]>
Date: Tue, 9 Nov 2010 21:02:13 -0800

I do not agree we have a problem that needs to be solved in 1.1.
In theory, data injection can still happen after the <hello>, so the alleged
data injection problem is not solved.

Switching framing during the session seems complicated and prone to
bugs and interoperability problems.

A client can certainly send a <get-config> on <running> before
knowing the capabilities.  NETCONF allows the client to send as
many <rpc> messages in a row as it wants.  I do not want to break
this feature.

We already agreed to punt this to 2.0.
That is the WG consensus.  I see no reason to change that decision.

Andy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Juergen Schoenwaelder
Sent: Tuesday, November 09, 2010 8:52 PM
To: Andy Bierman
Cc: [email-masked]
Subject: Re: [Netconf] ssh EOM and netconf 1.1

 

/js

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 10 Nov 2010 06:30:24 +0100

> bugs and interoperability problems.

I do not see why.

> many <rpc> messages in a row as it wants.  I do not want to break

The 1.1 transport can still accept ]]>]]> framing for corner cases
like this one. The point is to move to a more robust framing without
much hassle such as new subsystems.

> We already agreed to punt this to 2.0.  That is the WG consensus.  I
> see no reason to change that decision.

I like to understand whether there is a _technical_ problem with this
proposal. Any?

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 10 Nov 2010 13:34:41 +0800

> data injection problem is not solved.

I'd say this approach removes 99 % of the security risk so it is IMO a
very good thing.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Tue, 9 Nov 2010 22:52:45 -0800

What technical problem does this solution address?
It still relies on ]]>]]>.  That seems broken to me.
The solution should not rely on ]]>]]> whatsoever.

Is the problem to be solved:
  a) operators want to use the value ]]>]]> in NETCONF content
     so a different EOM is needed
  b) protect against data injection attacks
  c) allow a raw SSH terminal session to be used for NETCONF

IMO, (a) is of no interest at all.  The binary data type can be
used to address this problem.  

Only a real framing solution addresses (b).  We would force a session
to terminate at the first hint of a data injection attack if we cared
about solving (b), but we are not doing that.

None of the solutions addresses (c).

Marshall Rose used to force WGs to
  1) define the problem to be solved
  2) get the WG to agree there is a problem to be solved
  3) discuss solutions

Without (1) and (2) done, we are wasting time on (3).

Andy

-----Original Message-----
From: Juergen Schoenwaelder [mailto:[email-masked]] 
Sent: Tuesday, November 09, 2010 9:30 PM
To: Andy Bierman
Cc: Andy Bierman; [email-masked]
Subject: Re: [Netconf] ssh EOM and netconf 1.1

I do not see why.

/js

---

From: "Ersue, Mehmet (NSN - DE/Munich)" <[email-masked]>
Date: Wed, 10 Nov 2010 07:57:42 +0100

Juergen,

do I understand correctly that you suggest, the secure transport layer
should interpret the NETCONF hello message and based on this should
decide which EOM has to be used.
The capability exchange is AFAIU to negotiate the NETCONF capabilities.
Do you really want that SSH transport interprets XML content of a
NETCONF message?

Independently of the technical clearness I have a process-related
concern.
We had this discussion already many times and we all agreed to keep it
as it is now in 4742bis document.
As document shepherd I announced the discussion result and that I give
the document now over to Dan requesting to start the publication
process.

In spite of everything I think there might be still room to do changes
now or during IETF LC.
However, I would agree with such a change only if there are no
objections in the workgroup.

Mehmet

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of ext Juergen Schoenwaelder
> Sent: Wednesday, November 10, 2010 6:30 AM
> To: Andy Bierman
> Cc: [email-masked]

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 10 Nov 2010 15:34:19 +0800

>      so a different EOM is needed

No.

>   b) protect against data injection attacks

Yes, but also against simple DoS attacks based on forcing the devices to
close the session abruptly. If this happens simultaneously to several
dozen routers, it could mean a serious disruption.

>   c) allow a raw SSH terminal session to be used for NETCONF

I don't get how this ability is affected by the proposed change.

> Only a real framing solution addresses (b).  We would force a session

Real framing would be the best but I understand it is now out of
question. I prefer to fix the hole now with an imperfect but compatible
patch.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Tue, 9 Nov 2010 23:48:50 -0800

Hi,

We are not protecting against data injection attacks at all unless
there is real framing.  So (b) is not solved.

There is no DoS attack possible.
The session sending the 'fake' EOM is the only one affected.
Dropping your own NETCONF session does not cause the device to stop working.

IMO, (b) is the only problem worth solving here, and only
Phil's solution does that, using a new netconf-eom subsystem.

As Wes pointed out, if the operator can somehow get 0x4 accepted into
content by the buggy client, then the server is going to interpret
the 0x4 as the end of a message, and any data after that will be
interpreted as a new message.

I think we should get a new security area advisor and listen to
what that person has to say about addressing this security problem.
That might help get the IESG to approve the solution as well.

I object to changing 4742bis to address this problem, especially when
the solution doesn't actually work.

Andy

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ladislav Lhotka
Sent: Tuesday, November 09, 2010 11:34 PM
To: Andy Bierman
Cc: [email-masked]
Subject: Re: [Netconf] ssh EOM and netconf 1.1

No.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 10 Nov 2010 09:00:29 +0100 (CET)

> there is real framing.  So (b) is not solved.

[...]

> Phil's solution does that, using a new netconf-eom subsystem.

I disagree.  It protects against one form of injection, but there are
others.  Specifically, it protects against an injection of the EOM,
followed by a new rpc.  It does not protect against injection of
carefullt constructed XML tags that modifies the current operation,
as descibed in
http://www.ietf.org/mail-archive/web/netconf/current/msg04078.html

The only solution that works is that the client checks the input.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 10 Nov 2010 16:16:57 +0800

> there is real framing.  So (b) is not solved.

It makes them more difficult to do because a simple check with a
non-validating parser (an existing library function) will catch 0x04,
unlike the existing EoM. Note that injection attacks are possible with
any framing (e.g. adding nodes to a config), but being able to terminate
one message and inject an entire forged message is the most dangerous
case.
 
> Dropping your own NETCONF session does not cause the device to stop working.

I can have a configuration script that opens many sessions and sends the
same piece of (infected) config to many devices.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 10 Nov 2010 12:13:15 +0100

> Do you really want that SSH transport interprets XML content of a

Here is how I look at it:

  The NETCONF message layer tells the transport, after the <hello> has
  been processed, that the other end is running 1.1 and that the
  transport can use the new EOM marker which can't appear in valid XML
  content. The transport itself uses the traditional EOM until it gets
  notified by the message layer - the transport does not have to parse
  the XML nor does it have to understand capability announcements.

> objections in the workgroup.

I wanted to find out whether there is a technical solution that allows
us to move away from the traditional EOM marker, which does not have
the properties we assumed it has, without requiring a new subsystem or
port number transports.

I understand your procedural concern and as WG chair it is your job to
have this concern. At this point in time however, I like to stay out
of such a discussion - I am only interested to understand the solution
space, trying to find a solutions that restores the originally assumed
behaviour (an EOM marker that never appears in valid XML), which is
backwards compatible, and which does not require a new subsystem or
port number.

/js

---

From: "Ersue, Mehmet (NSN - DE/Munich)" <[email-masked]>
Date: Wed, 10 Nov 2010 12:17:26 +0100

Understood. Please elaborate "tells the transport".

Cheers,
Mehmet

> From: ext Juergen Schoenwaelder [mailto:j.schoenwaelder@jacobs-
> university.de]
> Sent: Wednesday, November 10, 2010 12:13 PM
> To: Ersue, Mehmet (NSN - DE/Munich)

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 10 Nov 2010 12:26:33 +0100

> Understood. Please elaborate "tells the transport".

    transport->switch_to(VERSION1_1);

or

    fprintf(client, "%", (version == VERSION1_1) ? NEWEOM : EOLEOM);

or

    ....

This is obviously implementation specific - we do not have ASIs. I am
sure an implementer will find a way how to implement this.

/js

---

From: Andy Bierman <[email-masked]>
Date: Wed, 10 Nov 2010 17:28:01 -0800

>      fprintf(client, "%", (version == VERSION1_1) ? NEWEOM : EOLEOM);

This is a serious layering violation and bad software design.

It would prevent NETCONF from being implemented using transport layer code
unless it could be specially hacked to support NETCONF.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 10 Nov 2010 17:32:48 -0800

On 11/10/2010 12:00 AM, Martin Bjorklund wrote:
> Andy Bierman<[email-masked]>  wrote:
>> Hi,
>>

So none of the proposed solutions actually does an adequate job
of protecting against data injection attacks?  Then that proves
to me that none of the solutions should be adopted.  Maybe by 2.0
someone will come up with a solution that actually works.
The server cannot rely on the client to be bug-free and always send
the right content and never send the EOM by mistake.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 11 Nov 2010 02:57:53 +0100

> This is a serious layering violation and bad software design.

I must have missing the specification of the interface between the
transport layer and the message layer you claim is violated. ;-)
 
> It would prevent NETCONF from being implemented using transport
> layer code unless it could be specially hacked to support NETCONF.

???

/js

---

From: Andy Bierman <[email-masked]>
Date: Wed, 10 Nov 2010 18:17:47 -0800

>> layer code unless it could be specially hacked to support NETCONF.

IMO this is terrible protocol and software design.
The solution:
   1) creates unwarranted coupling between protocol layers
   2) assumes the transport layer code can be modified to support NETCONF message
      layer hacks
   3) creates timing issues between the layers.  The SSH transport layer would
      be forced to deliver the first message and then wait to be told what
      EOM to look for after MSG #1.  Meanwhile, new data is arriving and it has to
      be buffered until the message layer tells the transport layer how to
      do its job.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 11 Nov 2010 03:29:15 +0100

On Wed, Nov 10, 2010 at 06:17:47PM -0800, Andy Bierman wrote:
 
>   2) assumes the transport layer code can be modified to support
>      NETCONF message layer hacks

I am not aware of anything else that would use the transport layer
code. Any solution to fix the EOM requires changes.

>   3) creates timing issues between the layers.  The SSH transport
>      layer would be forced to deliver the first message and then
>      wait to be told what EOM to look for after MSG #1.  Meanwhile,
>      new data is arriving and it has to be buffered until the
>      message layer tells the transport layer how to do its job.

Not true. I answered that before.

/js

---

From: "Bert (IETF) Wijnen" <[email-masked]>
Date: Thu, 11 Nov 2010 13:21:12 +0800

It seems we're not really communicating (at least not effectively).
Can we get together at 5pm today?

If Martin and Phil could dial in via skype or so, that would be great.

I'll try to arrange a breakout room
Bert

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 11 Nov 2010 13:57:40 +0800

> Can we get together at 5pm today?

Unfortunately I can't be there, I bought the evening trip to the Great
Wall and will be leaving at 4pm.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: "Bert (IETF) Wijnen" <[email-masked]>
Date: Thu, 11 Nov 2010 16:44:22 +0800

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 11 Nov 2010 11:30:38 +0100

> >I'll try to arrange a breakout room

There was no clear outcome. Anyway, I thought I write up how I look
at things after the short discussion. Perhaps it helps, perhaps
not. Note, I specifically do not consider procedural questions.

* What:

  We need to find agreement whether the possible code injection issues
  are a problem worth to solve or not, independently of when and
  how. If we believe there is no problem to solve, stop.

* When:

  Otherwise, we have to decide when to solve the problem. Solving it
  now may have a chance to replace the old framing before larger scale
  deployments of NETCONF. Solving it later, when NETCONF is deployed
  at a larger scale, means we will have to live much longer with old
  framings in the field.

* How:

The how has two aspects - the transport versioning aspect and the
question how a more robust framing looks like.

** Transport Versioning

   The SSH and TLS transport layers do not provide a mechanism for
   versioning, except using diffent SSH subsystems or port
   numbers. Using SSH subsystems is relatively painful, since lots of
   security processing is lost by trying probing availability of SSH
   subsystems. Alternative is to allocate new port numbers (but that
   does not help on systems that also provide NETCONF access over the
   default SSH port number).

   One proposal to solve the transport versioning issue is by having
   the initial <hello> exchange always happen in the old format and to
   change the transport behaviour by having the message layer tell the
   transport which version is used after having processed the
   capabilities contained in the <hello> message.

** Robust Framing 

   There have been several proposals for framing alternatives. Lada
   summarized them, see his slides. 

Again, this is how _I_ look at the situation.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 11 Nov 2010 21:45:10 +0800

> not. Note, I specifically do not consider procedural questions.

It is an objective description of the situation, I fully agree.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Thu, 11 Nov 2010 15:53:57 -0800

>> I'll try to arrange a breakout room

Here are the issues that I have with this entire discussion:

What are we fixing? what problem are we trying to solve?
People skip over this part as if it were irrelevant, but to me
the solutions are irrelevant if we cannot agree on the problem.

It seems the problem is that XML allows ]]>]]> to appear in
an XML document, but NETCONF does not allow it.  The current
4742bis text says so.

So a real actual debatable issue is whether or not it is reasonable
to require a NETCONF client application to make sure this sequence
is not sent in an XML document.  If so, then the current 4742bis solution
is fine and no changes are needed to anything.

If it is not reasonable to require a client to check for ]]>]]>
and escape it (e.g., change > to &gt;), then a new framing mechanism
is needed.

In that case, a new framing mechanism is needed.
If any character sequence whatsoever is used within the content,
then the client must implement it correctly.
IMO, it is a purist's argument to say that a client must not
need to check for otherwise legal XML.  The client must escape
the magic marker no matter what it is.  Solutions which require
the client to send some exact XML fragment are flawed, because
from a purist POV we cannot restrict valid XML in any way at all.
A magic PI or comment cannot be used because it is legal for
a 4741 client to send them, and if a 4741bis server misinterprets
the sequence, the wrong NETCONF version will be used.

Way down the list, after a new framing mechanism is picked, there is the
issue of coexistence with old servers.  Then a new debatable issue exists,
on how to do this.  One approach is to use a new subsystem and another
is to use a magic PI or XML comment before the <hello>.  The base:1.1
capability in the <hello> could also be used.  Using the same subsystem
means the old message framing is used for the first message, but maybe
not used after that, depending on if both sides agree to use base:1.1 or not.

Using the old framing at all in the protocol means a hacker can still
inject messages just the same as now, except only for the first message,
instead of any message.

So there is a debate whether using a new subsystem (e.g. netconf-eom)
is too much burden or not.  IMO, this is the safest and cleanest way
to support both 4741 and 4741bis servers.  A new client would first try
netconf-eom and then netconf if that did not work.  Over time, there
will likely be only netconf-eom servers so the client will never need
to try the old subsystem.

Using 2 subsystems keeps the problem localized to the transport layer code.
There are no new special interactions between the message and transport code needed.
There are no buffering requirements for the transport, waiting to be told
which transport framing mechanism has been negotiated in the operations layer.
It is also very clear to implementors what to do because SSH is being used as designed.
Using a magic PI or comment would be a hack that people who know SSH may not expect.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 12 Nov 2010 09:34:54 +0800

> the solutions are irrelevant if we cannot agree on the problem.

I believe what we are fixing is a security hole.

> an XML document, but NETCONF does not allow it.  The current

This is also quite unsystematic. A transport layer spec should not state
rules concerning message contents. Does it mean that I can use "]]>]]>"
when using TLS but not when using SSH?

On a practical side, XML well-formedness can be checked with
off-the-shelf tools while the search for "]]>]]>" requires a special
implementation.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Thu, 11 Nov 2010 17:58:13 -0800

> off-the-shelf tools while the search for "]]>]]>" requires a special

Let's do a little reality check...
We are not really concerned with XML attributes in NETCONF content
because we are standardizing on YANG, and YANG does not allow them.
There is only 1 place in the entire protocol where a client could
possibly provide an XML attribute and that is in the <rpc> element.

There are no implementations at all that allow the user to specify
XML attributes to add to <rpc> and I doubt this will ever happen because
it serves no use case at all.  This NETCONF feature is for the client
implementation.  I do not know of any application implementation that
actually uses it at all, let alone provides a user API to set these attributes.

Therefore it seems quite reasonable to keep the current 4742bis text
and require a client to make sure ]]>]]> is not sent inside a NETCONF message.
In theory XML allows in in a PI or XML attribute.  In reality, NETCONF
does not use these XML features so a client should not ever need to send them.

Andy

>> and escape it (e.g., change>  to&gt;), then a new framing mechanism
>> is needed.
>>
>> is to use a magic PI or XML comment before the<hello>.  The base:1.1
>> capability in the<hello>  could also be used.  Using the same subsystem

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 12 Nov 2010 13:00:43 +0800

> actually uses it at all, let alone provides a user API to set these attributes.

For a cracker, the easiest way is to try to smuggle an XML comment like
this to _any_ place of the XML document:

<!-- ]]>]]><rpc message-id="101" xmlns="...">...</rpc> -->

This is a perfectly valid XML, any parser will be happy to completely
ignore the comment. If this is not a huge security hole, then I don't
know what else is.

Lada 

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C