Subject: [clue] Kick-Off Item 5: description of composed pictures

From: Stephan Wenger <[email-masked]>
Date: Fri, 14 Oct 2011 23:02:53 -0400

Hi,

Here is the kickoff message as agreed in our F2F.

We are in the process of defining a mechanism that allows us to tag in room
"coordinates" of captures to RTP streams.  In its most trivial incarnation,
that would allow us to assign the decoded RTP streams to devices such as
individual monitors.

RTP <id#1> comes from system 1, camera coordinates x, y, z, upper right x,
y, z, lower left x, y, z
RTP <id#2> comes from system 1, camera coordinates (something else)

Note that we probably need a third and even a fourth point for faithfully
specifying a capture in 3D space, but that's not the subject of this email,
so don't beat me up on it right now.

As for the capabilities of a renderer (not defined in the framework yet), it
should be clear that a monitor can trivially display more than one decoded
RTP stream.  Insofar, the mapping should be per part of a virtual display
wall (or however you want to call it), and not per physical screen.  I would
hope that this is uncontroversial.

Things become more tricky when there is a video mixing MCU in the picture,
or when an endpoint creates a single RTP stream that contains information
from multiple camera devices (through tiling, picture-in-picture).  Here, we
cannot meaningfully tag the a single geographic information to a single RTP
stream simply because the multiple video sequences were mixed at the sample
level, and the encoder didn't have a clue that it is actually encoding data
stemming from more than one camera.  However, the endpoint system (or the
MCU) know the geographic metadata associated with the composed picture.  So
what I'm proposing is that we allow a single RTP stream to have more than
one geometric information about the capture involved.  For video, one would
probably have to associate regions of the reconstructed video stream (in the
spatial dimension) with capture geographic information as currently proposed
in the framework.

Example: a sending endpoint creating ONE RTP stream which includes TWO
captures, tiled horizontally

RTP <id#2> {
    bitstream coordinates x=0 to 959, y=0, to 1080 , comes from system 1,
camera coordinates x, y, z, x, y,z x, y, z
    bitstream coordinates x=960 to 1919, y=0 to 1080, comes from system 1,
camera coordinates (something else)
}

Architecturally, this system would best need to be recursive, as it ought to
be possible to throw an already composed image from MCU1 at the input of
MCU2, which scales it down and composes it with other images to other ports.

Clear enough?

Thanks,
Stephan

---

From: Michael Hammer <[email-masked]>
Date: Sat, 15 Oct 2011 13:03:07 -0400

Stephan,

You use monitor as both input and output device.  I hope that is not
embedded in the scheme.
The rest of this email seems to focus only on input definitions as it
probably should.

Don't understand the comment about 3-D followed by coordinates in 3-D.  It
seems like 3-D is fundamental.

Do I understand correctly that the camera coordinates define the location of
the lense, while the bitstream is abstraction of the pixel layout for some
abstraction (via ray-tracing?) of a window?

Or is the camera coordinates defining some virtual window in space that
through some changes in focal length could be a closeup and those farther
from the camera and narrower field of view versus wider angel and closer to
the lense with greater field of view?

When talking about spatial things, perhaps a picture would help save a
thousand words.

Also, I would think that what a renderer or MCU would do could be a second
discussion once the capture scheme is better understood.  (Understand that
the impacts of those might drive some iterations on the basic scheme.)

Mike

On Fri, Oct 14, 2011 at 11:02 PM, Stephan Wenger <[email-masked]> wrote:

> As for the capabilities of a renderer (not defined in the framework yet),
> it should be clear that a monitor can trivially display more than one
> decoded RTP stream.  Insofar, the mapping should be per part of a virtual
> display wall (or however you want to call it), and not per physical screen.
>  I would hope that this is uncontroversial.
>
> Architecturally, this system would best need to be recursive, as it ought
> to be possible to throw an already composed image from MCU1 at the input of
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>
>

---

From: Stephan Wenger <[email-masked]>
Date: Wed, 19 Oct 2011 13:35:20 -0700

Mike,

Not sure this is going through to the list, due to the size of the
attachment.  But let's try.

Attached is a little hand-drawing that hopefully helps understanding what I
am shooting for.
Consider a three way conference.  The two "capturing" rooms, depicted on the
left side, have two cameras each, capturing scenes one and two (for the
first room), and four and five (for the second room), respectively.  The
captured content is encoded by encoders marked "E" to create RTP streams 1,
2, 4, and 5.
RTP streams 1 and 2 (from the first room) terminate an an MCU, which
composes them into a composed picture 3, encoded into RTP stream 3.  The
composed picture contains 1 and 2 side-by-side, plus grey bars on top and
bottom.  
The monitors of the receiving third endpoint are shown on the right side.
Streams 4 and 5 go straight to the two decoders of the main projection wall.
Stream 3 (created by the MCU) goes through a side decoder to a third monitor
(could be the monitors that many telepresence rooms have for each
participant, a monitor normally dedicated for shared content, what not).
This monitor shows the composed image.

The CLUE architecture allows perfectly well to describe the capture
properties of streams 4 and 5.  However, at present, the MCU hides the
capture properties of streams 1 and 2.  Similarly, the MCU hides the render
properties of the decoder/display of stream 3.  Accordingly, the geographic
parameters of captures 1 and 2 are not known at decoder 3, and, if that
decoder would want to do something a bit more sophisticated than just
decoding and displaying on the screen, it couldn't because it lacks info.

One of this "more sophisticated" things could, for example be as follows.
Suppose the endpoint capturing 4 and 5 drops out of the call.  The video
wall of the rendering endpoint goes blank.  It would be desirable if the
info from 1 and 2 would be rendered on the video wall instead of the side
screen.  One sensible way to do that would be that the stream 3 is feed into
both decoders responsible for the video wall, but "cut out" and upscale the
content from the composed picture created by the MCU, before rendering.
(There may be technical reasons for the presence of the MCU beyond the fact
that there are only three decoders at the receiving side).  However, that's
not possible as the nature of stream 3 (being composed form 1 and 2) is not
visible at the rendering endpoint.

My proposal is to add information related to the captures inside a composed
stream (beyond that one flag that says the stream is composed).

Clearer?

Stephan    

From:  Michael Hammer <[email-masked]>
Date:  Sat, 15 Oct 2011 13:03:07 -0400
To:  Stephan Wenger <[email-masked]>
Cc:  "[email-masked]" <[email-masked]>
Subject:  Re: [clue] Kick-Off Item 5: description of composed pictures

Stephan,

 

 
 
Mike

On Fri, Oct 14, 2011 at 11:02 PM, Stephan Wenger <[email-masked]> wrote:
> RTP <id#1> comes from system 1, camera coordinates x, y, z, upper right x, y,
> z, lower left x, y, z
> specifying a capture in 3D space, but that's not the subject of this email, so
> don't beat me up on it right now.
> 
> should be clear that a monitor can trivially display more than one decoded RTP
> stream.  Insofar, the mapping should be per part of a virtual display wall (or
> however you want to call it), and not per physical screen.  I would hope that
> this is uncontroversial.
> 
> Things become more tricky when there is a video mixing MCU in the picture, or
> when an endpoint creates a single RTP stream that contains information from
> multiple camera devices (through tiling, picture-in-picture).  Here, we cannot
> meaningfully tag the a single geographic information to a single RTP stream
> simply because the multiple video sequences were mixed at the sample level,
> and the encoder didn't have a clue that it is actually encoding data stemming
> from more than one camera.  However, the endpoint system (or the MCU) know the
> geographic metadata associated with the composed picture.  So what I'm
> proposing is that we allow a single RTP stream to have more than one geometric
> information about the capture involved.  For video, one would probably have to
> associate regions of the reconstructed video stream (in the spatial dimension)
> with capture geographic information as currently proposed in the framework.
> 
> be possible to throw an already composed image from MCU1 at the input of MCU2,
> which scales it down and composes it with other images to other ports.
> 
> Clear enough?
> 
> Thanks,
> Stephan
> 
>

---

From: Michael Hammer <[email-masked]>
Date: Thu, 20 Oct 2011 14:14:27 -0400

Stephan,

What I am getting from this description is that you want the aggregated feed
composed of original feeds 1 and 2 to have sufficient data about the
original feeds to enable an output site to do a de-aggregation of feed 3
back into its components and to put on available displays as desired.

That sort of goal seems reasonable.  That would require that the mixing or
aggregation process leaves the feed 1 and 2 segregated within the feed 3.  I
imagine that some mixers might not do that, but it would provide
flexibility.  So, you might need a de-composable flag in addition to
distinguish if that can be done.  Anyway, that slides into the design.

Mike

On Wed, Oct 19, 2011 at 4:35 PM, Stephan Wenger <[email-masked]> wrote:

> Consider a three way conference.  The two "capturing" rooms, depicted on
> the left side, have two cameras each, capturing scenes one and two (for the
>  Streams 4 and 5 go straight to the two decoders of the main projection
> wall.  Stream 3 (created by the MCU) goes through a side decoder to a third
> monitor (could be the monitors that many telepresence rooms have for each
> Do I understand correctly that the camera coordinates define the location
> of the lense, while the bitstream is abstraction of the pixel layout for
> some abstraction (via ray-tracing?) of a window?
>
>> We are in the process of defining a mechanism that allows us to tag in
>> room "coordinates" of captures to RTP streams.  In its most trivial
>> incarnation, that would allow us to assign the decoded RTP streams to
>> devices such as individual monitors.
>>

---

From: Stephan Wenger <[email-masked]>
Date: Thu, 20 Oct 2011 12:05:01 -0700

Hi Mike,

> imagine that some mixers might not do that, but it would provide flexibility.
> So, you might need a de-composable flag in addition to distinguish if that can
> be done.  Anyway, that slides into the design.

The use case I was referring to is that the MCU performs video mixing in the
traditional way MCUs perform tiling (that is: reconstruction of the video
streams, aggregation on the pixel level, encoding).  In other words, feed 3
is a single RTP stream containing a single coded video stream.  To decompose
that stream, you need spatial information in the renderer domain where each
image is located in the tiled composed image.  That is what I want to add,
in addition to the "decomposable" flag you are suggesting (which would
simply be an indication that the spatial information is present for this RTP
stream).

Stephan

Date:  Thu, 20 Oct 2011 14:14:27 -0400

Stephan,

 
Mike

On Wed, Oct 19, 2011 at 4:35 PM, Stephan Wenger <[email-masked]> wrote:
> Mike,
> 
> Not sure this is going through to the list, due to the size of the attachment.
> But let's try.
> 
> Attached is a little hand-drawing that hopefully helps understanding what I am
> shooting for.
> left side, have two cameras each, capturing scenes one and two (for the first
> room), and four and five (for the second room), respectively.  The captured
> content is encoded by encoders marked "E" to create RTP streams 1, 2, 4, and
> 5.
> RTP streams 1 and 2 (from the first room) terminate an an MCU, which composes
> them into a composed picture 3, encoded into RTP stream 3.  The composed
> picture contains 1 and 2 side-by-side, plus grey bars on top and bottom.
> (could be the monitors that many telepresence rooms have for each participant,
> a monitor normally dedicated for shared content, what not).  This monitor
> shows the composed image.
> 
> The CLUE architecture allows perfectly well to describe the capture properties
> of streams 4 and 5.  However, at present, the MCU hides the capture properties
> of streams 1 and 2.  Similarly, the MCU hides the render properties of the
> decoder/display of stream 3.  Accordingly, the geographic parameters of
> captures 1 and 2 are not known at decoder 3, and, if that decoder would want
> to do something a bit more sophisticated than just decoding and displaying on
> the screen, it couldn't because it lacks info.
> 
> Suppose the endpoint capturing 4 and 5 drops out of the call.  The video wall
> of the rendering endpoint goes blank.  It would be desirable if the info from
> 1 and 2 would be rendered on the video wall instead of the side screen.  One
> sensible way to do that would be that the stream 3 is feed into both decoders
> responsible for the video wall, but "cut out" and upscale the content from the
> composed picture created by the MCU, before rendering.  (There may be
> technical reasons for the presence of the MCU beyond the fact that there are
> only three decoders at the receiving side).  However, that's not possible as
> the nature of stream 3 (being composed form 1 and 2) is not visible at the
> rendering endpoint.
> 
> You use monitor as both input and output device.  I hope that is not embedded
> in the scheme.
> The rest of this email seems to focus only on input definitions as it probably
> should.
>  
> through some changes in focal length could be a closeup and those farther from
> the camera and narrower field of view versus wider angel and closer to the
> lense with greater field of view?
>  
> discussion once the capture scheme is better understood.  (Understand that the
> impacts of those might drive some iterations on the basic scheme.)
>  
> Mike
> 
>> multiple camera devices (through tiling, picture-in-picture).  Here, we
>> what I'm proposing is that we allow a single RTP stream to have more than one
>> geometric information about the capture involved.  For video, one would