Subject: [siprec] Lossless recording/HA recording/media clipping? Open issues to SIPRECers?

From: "Yanqiang (Michael)" <[email-masked]>
Date: Thu, 13 Feb 2014 08:15:46 +0000

Hi guys,

	Recently there are discussions about several similar issues in mailing list. The 'lossless recording', 'HA recording', 'media clipping' are key words. Gerben had mentioned ' Lossless Recording' at http://www.ietf.org/mail-archive/web/siprec/current/msg03893.html (not 1st version for the issue but the explicit version in this WG). And Simon had mentioned 'media clipping' at http://www.ietf.org/mail-archive/web/siprec/current/msg03839.html . Are those issues still open for us?

	I tried to collect all the issues and solutions to make them easy-to-discuss:

1. UDP Package loss in transmission
	The solution might be:
	a. FEC
	b. draft-ietf-avtext-rtp-duplication-05
	b. RTP over TCP to same SRS?

2. media clipping, esp for video clipping caused by I-frame lost
	The solution might be:
	a. FIR (RFC 5104). buffering in SRC (ready for full frame refresh), check and start from I-frame in RS
	b. a persistent RS...(already have in SIPREC)

3. SRS is not ready to receive or goes down in the mid of RS
	The solution might be:
	a. buffering in SRC(a small duration or all session), then detect and resend
	b. have two independent parallel SRS, with synchronization mechanism 
	c. other ways? Switch to backup SRS (still loss during switching)? 

Then, let's back to current documents.

1. I think RFC6341 covers all issues in different way.
	a. Use Case 10: No loss of media recording can occur, including during failure of an SRS (issue #1&#3)
	b. REQ-005: The mechanism MUST support the ability to record a CS without loss of media of RS (for example, clipping media at the beginning of the CS) (issue #2)
	c. REQ-021: The mechanism MUST NOT prevent high-availability deployments (issue #1#2#3)

2. draft-siprec-architecture and draft-siprec-protocol documents seem not cover enough details about how to handle these issues.
	a. The architecture document might reconsider the description in Section Introduction to avoid ambiguity: The Replicated Media is required to be sent in real-time to the SRS and is *not buffered* by the SRC to allow for real-time analysis of the media by the SRS.
	b. The protocol document has section 8.1.7 for the use of FIR for generating a full I-frame...Any more details needed?

Are those the quality of implementation issues, not a standardization issues? Any comments?

Thanks,
Michael

---

From: "Dave Higton" <[email-masked]>
Date: Thu, 13 Feb 2014 14:23:47 +0000

That looks to me like a complete collection.

Dave

-----Original Message-----
From: siprec [mailto:[email-masked]] On Behalf Of Yanqiang (Michael)
Sent: 13 February 2014 08:16
To: [email-masked]
Subject: [siprec] Lossless recording/HA recording/media clipping? Open issues to SIPRECers?

Hi guys,

Thanks,
Michael
_______________________________________________
siprec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/siprec

NICE Systems UK Limited ("NICE") is registered in England under company number, 3403044. The registered office of NICE is at Tollbar Way, Hedge End, Southampton, Hampshire SO30 2ZP.

Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.

Monitoring: NICE may monitor incoming and outgoing e-mails.

Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.

---

From: "Dave Higton" <[email-masked]>
Date: Thu, 13 Feb 2014 14:46:33 +0000

Here are a few assumptions on topics 1 and 3 for everybody to check out:

* The network is engineered to have all the required bandwidth.

* We can only cure occasional dropped packets.

* If there is a hard failure, e.g. a network switch dies, no simple retry-based mechanism is going to solve it.  For those cases you need redundancy, eliminating all single points of failure.

* We want to use an existing, tried and trusted, protocol; we don't want to engineer a new one.

* We don't have significant numbers of implementations of lossless recording in the field, so whatever we do is going to be a change from what we do now.

* Any SRC that implements SIPREC is engineered with enough RAM to buffer the media for long enough to deal with some retries.

* SIPREC should cover lossless recording in enough detail that people can implement to it, referring to other RFCs where necessary.

Based on the above assumptions, I'm suggesting that TCP is a tried and trusted solution to the problem of occasional dropped packets to the SRS.  TCP is usually discounted for RTP because of the increase in latency.  Clearly that is not a consideration for recording, because recording is inherently a time delay mechanism.

RFC 4571 would appear to pretty much cover what is necessary for RTP over TCP, including an update to SDP to specify TCP.

Comments?  Better suggestions?

Dave

-----Original Message-----
From: siprec [mailto:[email-masked]] On Behalf Of Yanqiang (Michael)
Sent: 13 February 2014 08:16
To: [email-masked]
Subject: [siprec] Lossless recording/HA recording/media clipping? Open issues to SIPRECers?

Hi guys,

Thanks,
Michael

---

From: Brian Rosen <[email-masked]>
Date: Thu, 13 Feb 2014 10:15:32 -0500

That’s pretty radical.

Are you assuming that you record at least twice (two SRSs)?  That is a required part of HA, no matter what.

Then we have to determine if we need to improve the reliability of a single instance of an SRS.

That’s a matter of the final required reliability, and is also related to how many copies you are making = the larger the number of SRSs, the lower the reliability of each one of them needed to get the overall reliability.

What’s the goal?  4 nines?  5?

If you are trying to achieve 4 nines, you have 2 instances, then you need a fairly high reliability for each one of the instances, and we may need something like TCP to accomplish that.

If you have 3 instances, you might not need it.  Having said that, the math to determine what any real implementation needs is non trivial.

We might look at this as a tool box, support mechanisms to both increase reliability of a single instance and support more than one instance, and let the system designer determine how to achieve their reliability requirements with the tools.

Brian

---

From: "Dave Higton" <[email-masked]>
Date: Thu, 13 Feb 2014 16:00:20 +0000

We're talking about two different issues here.  I'm talking about preventing clipping and loss of occasional packets.  You're talking about redundancy and high availability.  Simply adding redundant recording is going to do nothing to prevent clipping and loss of occasional packets, AFAICS.

Recording systems for NG911, for example, are going to have to address both issues.

Dave

-----Original Message-----
From: Brian Rosen [mailto:[email-masked]] 
Sent: 13 February 2014 15:16
To: Dave Higton
Cc: [email-masked]
Subject: Re: [siprec] Lossless recording/HA recording/media clipping? Open issues to SIPRECers?

That's pretty radical.

That's a matter of the final required reliability, and is also related to how many copies you are making = the larger the number of SRSs, the lower the reliability of each one of them needed to get the overall reliability.

What's the goal?  4 nines?  5?

Brian

---

From: "Parthasarathi R" <[email-masked]>
Date: Thu, 13 Feb 2014 22:09:42 +0530

Hi Dave,

Initial Clipping shall be avoided in case RS callflow is designed in a
better way. Please note that I'm not talking only about persistent RS. If
required, we will add more example callflow to illustrate the same. 

Thanks
Partha

> From: siprec [mailto:[email-masked]] On Behalf Of Dave Higton
> Sent: Thursday, February 13, 2014 9:30 PM
> To: [email-masked]
> We're talking about two different issues here.  I'm talking about
> preventing clipping and loss of occasional packets.  You're talking
> about redundancy and high availability.  Simply adding redundant
> recording is going to do nothing to prevent clipping and loss of
> occasional packets, AFAICS.
> 
> That's a matter of the final required reliability, and is also related
> to how many copies you are making = the larger the number of SRSs, the
> lower the reliability of each one of them needed to get the overall
> reliability.
> 
> If you are trying to achieve 4 nines, you have 2 instances, then you
> need a fairly high reliability for each one of the instances, and we
> may need something like TCP to accomplish that.
> 
> If you have 3 instances, you might not need it.  Having said that, the
> math to determine what any real implementation needs is non trivial.
> 
> We might look at this as a tool box, support mechanisms to both
> increase reliability of a single instance and support more than one
> instance, and let the system designer determine how to achieve their
> reliability requirements with the tools.
> 
> Brian
> 
> > * If there is a hard failure, e.g. a network switch dies, no simple
> retry-based mechanism is going to solve it.  For those cases you need
> redundancy, eliminating all single points of failure.
> >
> > * We don't have significant numbers of implementations of lossless
> recording in the field, so whatever we do is going to be a change from
> what we do now.
> >
> > * Any SRC that implements SIPREC is engineered with enough RAM to
> buffer the media for long enough to deal with some retries.
> >
> > * SIPREC should cover lossless recording in enough detail that people
> can implement to it, referring to other RFCs where necessary.
> >
> > Based on the above assumptions, I'm suggesting that TCP is a tried
> and trusted solution to the problem of occasional dropped packets to
> the SRS.  TCP is usually discounted for RTP because of the increase in
> latency.  Clearly that is not a consideration for recording, because
> recording is inherently a time delay mechanism.
> >
> > RFC 4571 would appear to pretty much cover what is necessary for RTP
> over TCP, including an update to SDP to specify TCP.
> >
> > 	Recently there are discussions about several similar issues in
> mailing list. The 'lossless recording', 'HA recording', 'media
> clipping' are key words. Gerben had mentioned ' Lossless Recording' at
> http://www.ietf.org/mail-archive/web/siprec/current/msg03893.html (not
> 1st version for the issue but the explicit version in this WG). And
> Simon had mentioned 'media clipping' at http://www.ietf.org/mail-
> archive/web/siprec/current/msg03839.html . Are those issues still open
> for us?
> >
> > 	a. FIR (RFC 5104). buffering in SRC (ready for full frame
> refresh), check and start from I-frame in RS
> > 	b. REQ-005: The mechanism MUST support the ability to record a CS
> without loss of media of RS (for example, clipping media at the
> beginning of the CS) (issue #2)
> > 2. draft-siprec-architecture and draft-siprec-protocol documents seem
> not cover enough details about how to handle these issues.
> > 	a. The architecture document might reconsider the description in
> Section Introduction to avoid ambiguity: The Replicated Media is
> required to be sent in real-time to the SRS and is *not buffered* by
> the SRC to allow for real-time analysis of the media by the SRS.
> > 	b. The protocol document has section 8.1.7 for the use of FIR for
> generating a full I-frame...Any more details needed?
> >
> > NICE Systems UK Limited ("NICE") is registered in England under
> company number, 3403044. The registered office of NICE is at Tollbar
> Way, Hedge End, Southampton, Hampshire SO30 2ZP.
> >
> > Confidentiality: This communication and any attachments are intended
> for the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE. If this communication has come to you in
> error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> >
> > Viruses: Although we have taken steps toward ensuring that this e-
> mail and attachments are free from any virus, we advise that in keeping
> with good computing practice the recipient should ensure they are
> actually virus free.
> >

---

From: Brian Rosen <[email-masked]>
Date: Thu, 13 Feb 2014 12:17:28 -0500

Huh.  You said you were addressing points 1 and 3 (packet loss and SRS failure).  Clipping is point 2.

Confusion.  I was addressing mostly 3, but usually if you have two independent copies of a stream, you could get 1 covered as long as you can somehow merge the streams (since the packets will have the same samples in them).

Brian

---

From: "Dave Higton" <[email-masked]>
Date: Thu, 13 Feb 2014 17:34:54 +0000

Ah, my apologies.  I wasn't paying enough attention to the contents of the issues.  I saw video in 2 and dismissed it as "I don't know enough about this", whereas it's actually about clipping of any media stream.

Perhaps I should start again...

But I believe that redundancy does not address clipping at all, and cannot be relied upon to fix occasional loss of packets.

Dave

-----Original Message-----
From: Brian Rosen [mailto:[email-masked]] 
Sent: 13 February 2014 17:17
To: Dave Higton
Cc: [email-masked]
Subject: Re: [siprec] Lossless recording/HA recording/media clipping? Open issues to SIPRECers?

Brian

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 13 Feb 2014 13:49:17 -0500

> But I believe that redundancy does not address clipping at all, and cannot be relied upon to fix occasional loss of packets.

Redundancy could fix the occasional loss of packets, with some effort, 
if the redundant paths are completely independent. (Synchronize the two 
recordings after the fact, with each filling holes in the other. This 
assumes that probability of dropping the same packet on both paths is low.)

I guess addressing clipping requires local buffering, and sufficient 
bandwidth to catch up once you have a connection. Agree it doesn't have 
anything to do with redundancy, except in cases where it is quicker to 
establish a connection to one path than the other.

	Thanks,
	Paul

---

From: Brian Rosen <[email-masked]>
Date: Thu, 13 Feb 2014 14:51:40 -0500

Agree

On Feb 13, 2014, at 1:49 PM, Paul Kyzivat <[email-masked]> wrote:

> Redundancy could fix the occasional loss of packets, with some effort, if the redundant paths are completely independent. (Synchronize the two recordings after the fact, with each filling holes in the other. This assumes that probability of dropping the same packet on both paths is low.)
Yes, that was what I was trying to say

> 
> I guess addressing clipping requires local buffering, and sufficient bandwidth to catch up once you have a connection. Agree it doesn't have anything to do with redundancy, except in cases where it is quicker to establish a connection to one path than the other.
Yeah, but you need enough buffering for the worst path.