Subject: [clue] Origin and dimensions

From: John Leslie <[email-masked]>
Date: Wed, 12 Oct 2011 06:16:42 -0400

Listening yesterday, I'm not sure we all have the same understanding
of mathematical modeling of cameras and microphones.

   Inherently, a camera needs six (or more) dimensions: three for where
it is, and three for where it's pointed. A microphone needs three for
where it is, and two for where it's pointed. (The difference is that
the camera has a focus length -- anything closer or farther will be
out of focus.)

   The canonical representation IMHO should be cartesian for position
and polar for where it's pointed. Thus X, Y, and Z in millimetres is
ideal, while pan and tilt in degrees feels right.

   YMMV, of course... To me pan-tilt-zoom is so cheap today that I
expect it to dominate eventually; also, various calculations such as
gap-correction and panorama-stitching make more sense in polar.

   For microphones, recall that all microphones have sensitivity
patterns, and "cardioid" tends to dominate in practice: thus, where
the microphone is pointed _does_ matter.

   I well remember when Apple was starting on MacOS they made a big
thing about mathematical modeling of their display. Many folks scoffed
at this; but IMHO the decision to be crisp and correct in their model
really paid off.

   I would urge that we _have_ a canonical representation which is
crisp and correct, even if it not mandatory.

--
John Leslie <[email-masked]>

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 12 Oct 2011 07:33:48 -0400

Our systems use microphone arrays, the pickup pattern is not necessarily
fixed (though we are generating a proper stereo capture). In the case of
audio, we should not presume that an "audio capture" comes from a single
cardiod microphone.

I don't favor polar for where captures are aimed, because of the origin
problem stated yesterday.  We need to easily identify adjacency of video
captures even if the cameras are not co-located.  Polar works for individual
cameras, and it also works if all the cameras are in the same position.  But
it is not easy to tell how two fields of view are related if you use have
cameras that are not co-located and you use each camera as the origin for
its polar field of view.  And polar is *not* natural if the origin is not
the camera location.

BTW, not sure about the need for focus length in general, but if we do need
it we should account for depth of field (a range of distance, not just a
single point).   Also, we'd need to deal with auto-focus cameras.

Regards,
Stephen Botzko

On Wed, Oct 12, 2011 at 6:16 AM, John Leslie <[email-masked]> wrote:

> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

---

From: Marshall Eubanks <[email-masked]>
Date: Wed, 12 Oct 2011 07:39:45 -0400

Dear John;

A very good and clear description here.

> the camera has a focus length -- anything closer or farther will be

Yes, and they both have nominally 2-D fields of view as well, which
can change if they zoom, so that is technically
8 and 7, although I think for microphones it is more common to use
polar lobe widths (i..e,
1 width parameter can suffice), so 8 and 6.

That assumes that people are not using arrays of microphones or 3-D
cameras. Arrays of microphones
(linear or otherwise) can point in multiple directions at once, so
that can be dealt with as separate
virtual microphones (as I don't think we want to get into microphone
phase analysis, and that might
be hardwired in the system and inaccessible otherwise).

3-D cameras are of course doublets, with a fixed separation and
orientation, so so can be treated as 8 parameters plus 3
more (the physical separation of the cameras plus the angular
orientation of the line separating them). In principle a
3-D camera can also be dealt with as simply 2 cameras (and thus 16
parameters), but treating them as a doublet reduces that to 11
(as you can assume that they are looking in the same location with the
same zoom and focus). I strongly suspect that 3-D compression and
display will deal with the doublet in a hard-wired fashion.

> ideal, while pan and tilt in degrees feels right.

Exactly.

> gap-correction and panorama-stitching make more sense in polar.

If parameters are fixed or irrelevant, you can simply ignore them.

> crisp and correct, even if it not mandatory.

In my opinion, if we don't, we are not future proofed, and will regret it later.

Regards
Marshall

---

From: John Leslie <[email-masked]>
Date: Wed, 12 Oct 2011 12:14:33 -0400

> audio, we should not presume that an "audio capture" comes from a single

   Obviously, not all microphones are cardioid: at a minimum, you'd want
to distinguish a "generic cardioid" from a "generic omnidirectional". It
may make sense to have a few "generic" sensitivity patterns (which will
likely be ignored by most renderers today anyway).

   And for a microphone array, if you don't want to generate an actual
sensitivity pattern (for the array), IMHO it's reasonable to specify none.

> problem stated yesterday.

   Mea culpa! I didn't intend to talk of a _single_ polar coordinate
system at the origin of the cartesian system -- rather a polar system
for each camera centered at the XYZ of that camera. (Where the zero
pan and tilt points is arbitrary: I didn't mean to advocate anything
specific.)

> We need to easily identify adjacency of video captures even if the
> cameras are not co-located.

   Actually, this _is_ easy, so long at our definitions are crisp and
complete.

> Polar works for individual cameras, and it also works if all the cameras
> are in the same position. But it is not easy to tell how two fields of
> view are related if you use have cameras that are not co-located and
> you use each camera as the origin for its polar field of view.

   Actually, it _is_ simple: transform both to cartesian, then compare.
(There are trade-offs here: I can imagine cases where all measurements
are done in cartesian, and there is no benefit from any transformation
into a polar system.)

> And polar is *not* natural if the origin is not the camera location.

   Agreed.

> BTW, not sure about the need for focus length in general, but if we do
> need it we should account for depth of field (a range of distance,
> not just a single point).

   Depth-of-field is pretty mushy: IMHO it's different for different
renderers. But in principle, you're right.

> Also, we'd need to deal with auto-focus cameras.

   Alas! I haven't thought of a way to deal with a camera which does that
unless _it_ reports its focus-length.

---

From: Marshall Eubanks <[email-masked]>
Date: Wed, 12 Oct 2011 12:21:48 -0400

Dear John;

>> audio, we should not presume that an "audio capture" comes from a single

More importantly, it is common to hide microphone array details in firmware, and
even the direction of the maximum sensitivity can be changed
automatically. We may only
be able to go so far here.

> sensitivity pattern (for the array), IMHO it's reasonable to specify none.

The default should be omnidirectonal.

> pan and tilt points is arbitrary: I didn't mean to advocate anything

That is what makes sense and how I interpreted what you said.

> unless _it_ reports its focus-length.

I am not so sure we would want to communicate constant mm level
changes in focus that autofocus tends to produce.
A default I think makes more sense here.

Marshall