Subject: [Gen-art] timing of reviews

From: Jari Arkko <[email-masked]>
Date: Fri, 24 May 2013 01:28:48 +0300

Thanks again for all your hard work in doing the reviews. They make it possible for me to concentrate on reviewing just those documents where there are problems or I have deeper expertise. And the quality of the protocol specifications coming out of the IETF is obviously very important, particularly for protocols that are gaining significant use.

As you may have seen, the IESG and the community has been wondering if there'd be something that we could do about the IETF process, in the sense that there's quite many things happening at the very end of the document's life cycle. This results in some surprises, and often also moves some important decisions out of the working group and to author/shepherd/AD hands. A while ago we met for the IESG retreat and wanted to experiment with three specific changes:

- sending work back to the WGs when significant changes are needed, and making the WG the central place of the edits
- moving some directorate reviews earlier, without impact reviewer effort too much
- inviting some of the shepherds onto tele chats

I am writing to you in order to discuss the second item. How big of a change would it be to have Gen-ART reviews be invoked during WGLC, while the documents are still in the working groups? The goal of the reviews would still be the same, e.g., I would be checking that the reviews were positive and/or that the issues brought up have been properly addressed.

There are important details to consider, however, and I would like to get your feedback on how you would seem them having an effect, and what would be the best way to organise this, if we decide to go ahead with the change for the Gen-ART.

Triggering the review would have to be done by something else than IETF last call announcement. Is the best approach is to have the WG chairs manually request for this? Note that sometimes there are multiple WGLCs. I presume that it would be preferable to have a Gen-ART review be done only once at this stage, as otherwise the work load would increase. The chairs may have some idea of whether they are likely to need another WGLC before they start one. 

There may be possibly bigger changes and time lag between the first Gen-ART review and the one that checks that the changes are ok.

Some specs may not make it through WGLC, and a review at that stage may increase the effort you guys are putting in, by reviewing those specs that will fail.

Jari

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 23 May 2013 19:43:51 -0400

In principle I am happy perform reviews in conjunction with WG last 
call, even though that would probably produce a small increase in loading.

There is however a complication.  Many WG email lists are "ember only", 
for good reason.  I do not wish to end up on all those lists.  Which 
means that the moderator has to let the message through, and then 
probably remember to whitelist the reviewer.  Ad folk have to keep 
copying the reviewer explicitly with responses.  I think this can be 
done, but we should remember such procedural glitches.

Yours,
Joel

On 5/23/2013 6:28 PM, Jari Arkko wrote:
> Thanks again for all your hard work in doing the reviews. They make
> it possible for me to concentrate on reviewing just those documents
> where there are problems or I have deeper expertise. And the quality
> of the protocol specifications coming out of the IETF is obviously
> very important, particularly for protocols that are gaining
> significant use.
>
> As you may have seen, the IESG and the community has been wondering
> if there'd be something that we could do about the IETF process, in
> the sense that there's quite many things happening at the very end of
> the document's life cycle. This results in some surprises, and often
> also moves some important decisions out of the working group and to
> author/shepherd/AD hands. A while ago we met for the IESG retreat and
> wanted to experiment with three specific changes:
>
> - sending work back to the WGs when significant changes are needed,
> and making the WG the central place of the edits - moving some
> directorate reviews earlier, without impact reviewer effort too much
> I am writing to you in order to discuss the second item. How big of a
> change would it be to have Gen-ART reviews be invoked during WGLC,
> while the documents are still in the working groups? The goal of the
> reviews would still be the same, e.g., I would be checking that the
> reviews were positive and/or that the issues brought up have been
> properly addressed.
>
> There are important details to consider, however, and I would like to
> get your feedback on how you would seem them having an effect, and
> what would be the best way to organise this, if we decide to go ahead
> with the change for the Gen-ART.
>
> Triggering the review would have to be done by something else than
> IETF last call announcement. Is the best approach is to have the WG
> chairs manually request for this? Note that sometimes there are
> multiple WGLCs. I presume that it would be preferable to have a
> Gen-ART review be done only once at this stage, as otherwise the work
> load would increase. The chairs may have some idea of whether they
> are likely to need another WGLC before they start one.
>
> There may be possibly bigger changes and time lag between the first
> Gen-ART review and the one that checks that the changes are ok.
>
> Some specs may not make it through WGLC, and a review at that stage
> may increase the effort you guys are putting in, by reviewing those
> specs that will fail.
>
> Jari
>
> _______________________________________________ Gen-art mailing list
> [email-masked] https://www.ietf.org/mailman/listinfo/gen-art
>

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 24 May 2013 14:27:21 +1200

> call, even though that would probably produce a small increase in loading.

I think time will tell whether it's small. Getting into the thick of
WG discussion in an area one doesn't know well can be quite a shock
to the system. That's not to say we shouldn't try it, but I would ask
for it to be a time-limited experiment initially.

It seems important to inform WGs explicitly that outside review
is taking place *and* has to be taken seriously (because if not,
the same reviewer will be back at a later stage with the same
questions).

Then there is a question: will Gen-ART re-review at the time of
IETF Last Call as well as when the draft reaches the IESG agenda?

BTW, the concept of early review by Gen-ART is not new, but I think
it was only tried for one or two drafts in the past.

    Brian

> [email-masked]
> https://www.ietf.org/mailman/listinfo/gen-art
>

---

From: Jari Arkko <[email-masked]>
Date: Fri, 24 May 2013 08:59:39 +0300

Joel:

> There is however a complication.  Many WG email lists are "ember only", for good reason.  I do not wish to end up on all those lists.  Which means that the moderator has to let the message through, and then probably remember to whitelist the reviewer.  Ad folk have to keep copying the reviewer explicitly with responses.  I think this can be done, but we should remember such procedural glitches.

Good point. Thanks for bringing that up.

Jari

---

From: Jari Arkko <[email-masked]>
Date: Fri, 24 May 2013 09:11:02 +0300

> for it to be a time-limited experiment initially.

Yes - or scope limited (only some WGs or areas at the beginning).

> questions).'

Good point.

> IETF Last Call as well as when the draft reaches the IESG agenda?

My initial though was that we'd do the same as we do today for drafts that change, i.e., re-review by the same person. The likelihood for draft changes is of course bigger when starting from the WGLC. Another possible work load increase to watch for. I'd be interested in your thoughts on other approaches to deal with re-reviews though.

Jari

---

From: Mary Barnes <[email-masked]>
Date: Fri, 24 May 2013 08:14:59 -0500

I will caveat my comments with the fact that I am no longer taking on
new documents for review. However, having been involved in the process
for 10 years now on the gen-art side and having done the reviews for 3
IETF chairs, and having much experience as a chair and document
editor,  I do have some concerns as laid out below.

Mary.

> Yes - or scope limited (only some WGs or areas at the beginning).
[MB] Absolutely, it needs to have a limited scope initially.  As Brian
said, the concept is not new, it's just that few took advantage of it.
   I would prefer that this stage remain optimal and up to the WG
chairs as to whether it's necessary. Without  the tool support that is
available for the regular LC and Telechat reviews, I don't think we
want to burden Jean too much until we have a very good idea of the
level of effort and until there is tool support (e.g., chairs enter
the docs into a tool that gets the doc into Jean's incoming queue for
assignment). I also believe this increases the burden on the reviewers
significantly more than the burden when we added the LC reviews.  For
folks that haven't been around a while, the original Gen-ART process
only did reviews prior to the telechats.  Adding reviews at LC did
increase the workload for reviewers, however, we did see a somewhat
increase in documents ready at Telechat time.   [/MB]
>> the same reviewer will be back at a later stage with the same
[MB] Exactly.  [/MB]

[MB] My biggest concern is that many documents that go through WGLC in
the WGs in which I am active are NOT ready at that stage.  I also have
seen in cases of Gen-ART reviews, that documents clearly weren't
reviewed by the shepherd or ADs due to the level of editorial issues
and technical (the p2psip reload document is a good example of this).
There were concerns raised in the WG with regards to its readiness but
it was still progressed. Certainly, I hope this is the exception.
However, my guess is that many documents will change quite a bit after
LC.  When I chaired the SIPPING WG, we had multiple stages of review
in the WG where we had assigned reviewers.  We had an initial review
when the doc was first accepted as a WG document, then we had a
pre-WGLC review and then the real WGLC. Many of the documents went
through a 2nd last call. If we could get more WGs to be more
disciplined about ensuring that documents receive adequate review
prior to WGLC, then this idea might work.  At this point, I see very
few WGs that have an effective process in place. One also has to
consider that in some WGs (many that I participate in and write docs
for), there are very often folks that don't review any version of a
document in detail until it goes through WGLC. So, I am personally
quite skeptical that this proposal adds significant value to the
overall process, in particular based on the level of effort required
by both the gen-art reviewers.  I think this is just moving an
existing problem around and asking the wrong set of folks to help
solve the problem. I think a more effective change would be for ADs to
expect WGs to ensure the docs are thoroughly reviewed earlier in the
stage by the WG.

 [/MB]

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 24 May 2013 15:13:56 +0100

FYI, I started the same discussion on the secdir list. [1]

I'll summarise to the IESG when appropriate.

S.

[1] http://www.ietf.org/mail-archive/web/secdir/current/msg03968.html

---

From: "Black, David" <[email-masked]>
Date: Fri, 24 May 2013 12:00:40 -0400

I concur with Brian's concerns - I would strongly suggest that the WG chair(s)
take on two responsibilities for this experiment with Gen-ART reviews at WGLC:

(1) Commit to monitoring/mediating the WG mailing list interaction between the
WG members and the Gen-ART reviewer.  Letting the WG membership loose "no holds
barred" on a Gen-ART reviewer who is not an expert in the protocol being reviewed
may not work out well, e.g., it could generate list reactions along the lines of
"if the Gen-ART reviewer doesn't understand this basic concept, why should we pay
serious attention to anything else in the review?"  I've recently had a behind
the scenes interaction on a technical issue from a Gen-ART review where the
involvement of the responsible WG chair made a significant difference.

(2) Determine whether the draft is mature enough for a Gen-ART review at WGLC.
At a high level, the question to be asked is roughly "if WG LC resulted in no
changes to this draft, would you be comfortable submitting it to your AD with
a publication request?"  If the answer is "no", a Gen-ART review is probably
premature.  A potentially useful mechanism could be to ask the WG chair to
prepare a publication request writeup (Barry's short form should suffice) at
the start of WGLC in order to request a Gen-ART review.  If the WG chair believes
that a writeup is premature, then the Gen-ART review is probably also premature.
For example, if a major purpose of a WGLC is to drive open technical issues
to resolution, Gen-ART review is probably premature, and a writeup would note
the open technical issues that need resolution.

Thanks,
--David

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Brian E Carpenter
> Sent: Thursday, May 23, 2013 10:27 PM
> To: Joel M. Halpern
> Cc: General Area Review Team; The IESG
> Subject: Re: [Gen-art] timing of reviews
>

---

From: Elwyn Davies <[email-masked]>
Date: Fri, 24 May 2013 17:42:23 +0100

The comments from David, Brian, Mary and Joel are broadly in line with
my thinking on this.  One additional point: during the discussion that
Jari's original question provoked I suggested that a written summary of
the design decisions that had led to the chosen protocol design could be
helpful for the 'tail' discussion process.  Although it would make some
extra work for authors and WG chairs, having this information would make
the gen-art (and other) reviewing easier and make interpreting what is
going on during WGLC somewhat easier.

One way of doing this would be to make the summary an appendix to the
proto write up.  Doing this might also serve to bring out any remaining
issues that need to be sorted out during WGLC - and thereby provide
evidence of the draft maturity as highlighted by David. And, perhaps,
lessen the chance (or at least the impact) of attempts to reopen decided
issues.

All this being said, I would certainly be prepared to give it a go on a
trial basis if our secretary can cope with he extra work until the tools
could be extended.

Regards,
Elwyn

---

From: Spencer Dawkins <[email-masked]>
Date: Fri, 24 May 2013 12:17:01 -0500

> said, the concept is not new, it's just that few took advantage of it.

Mary is the best person to comment on the time period when I was a 
Gen-ART reviewer, but I do remember a few working group chairs asking 
for early reviews (which were granted in every case I heard about). I'm 
not sure how many chairs knew they could ask for early reviews, and of 
course, I don't know why they wanted early reviews. But it's fair to say 
that nobody discouraged early reviews, at least not where I could hear that.

To all the reviewers:

Thank all of you for your review work. I dropped off Gen-ART when I 
joined the IAB, but still read your reviews (and am now reading them 
more closely :-)

Spencer

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Sun, 26 May 2013 08:57:11 +0000

I am sharing Mary's concerns - also some of the ones expressed by other.

The principal problem I see with moving the Gen-ART review in parallel with the WGLC rather than IETF LC is related to the level of maturity of documents when they reach WGLC. This is of course variable according to the area and WG, but it is clear that most of the documents are in a less stable status when they reach WGLC than when they reach IETFLC. As a result the Gen-ART reviewers will see their load increase not only as a result of having to review a number of documents that will never exit the WG (I do not know how many these are, can we see statistics?), but also because they will need to follow the full WGLC cycle which may include several revisions, extend over multiple Last Calls, and lead to considerable changes in the documents resulting from other WGLC comments, changes which also need to be reviewed. 

I do believe that there is however a solution to this problem, but it does not come for free either. Let the WGLC take place as it happens right now, but move the Expert Reviews (a.k.a. Directorate Reviews) also within the responsibility of the WG, after the WGLC completes. This will make the WG cycle longer by a few weeks, but will allow for the Directorate comments to be processed by the WG, but on a stable version, which the WG considers as having passed the WGLC. 

Would this work? We may want to hear the wg-chairs opinion, I am concerned about to intensive cross-posting. 

Regards,

Dan

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Mary Barnes
> Sent: Friday, May 24, 2013 4:15 PM
> To: Jari Arkko
> IETF chairs, and having much experience as a chair and document editor,
> I do have some concerns as laid out below.
> 
> Mary.
> 
>    I would prefer that this stage remain optimal and up to the WG chairs
> as to whether it's necessary. Without  the tool support that is
> available for the regular LC and Telechat reviews, I don't think we want
> to burden Jean too much until we have a very good idea of the level of
> effort and until there is tool support (e.g., chairs enter the docs into
> a tool that gets the doc into Jean's incoming queue for assignment). I
> also believe this increases the burden on the reviewers significantly
> more than the burden when we added the LC reviews.  For folks that
> haven't been around a while, the original Gen-ART process only did
> reviews prior to the telechats.  Adding reviews at LC did increase the
> workload for reviewers, however, we did see a somewhat
> >> It seems important to inform WGs explicitly that outside review is
> >> taking place *and* has to be taken seriously (because if not, the
> >> same reviewer will be back at a later stage with the same
> >> questions).'
> >
> > Good point.
> [MB] Exactly.  [/MB]
> 
> reviewed by the shepherd or ADs due to the level of editorial issues and
> technical (the p2psip reload document is a good example of this).
> LC.  When I chaired the SIPPING WG, we had multiple stages of review in
> the WG where we had assigned reviewers.  We had an initial review when
> the doc was first accepted as a WG document, then we had a pre-WGLC
> review and then the real WGLC. Many of the documents went through a 2nd
> last call. If we could get more WGs to be more disciplined about
> ensuring that documents receive adequate review prior to WGLC, then this
> idea might work.  At this point, I see very few WGs that have an
> effective process in place. One also has to consider that in some WGs
> (many that I participate in and write docs for), there are very often
> folks that don't review any version of a document in detail until it
> goes through WGLC. So, I am personally quite skeptical that this
> proposal adds significant value to the overall process, in particular
> based on the level of effort required by both the gen-art reviewers.  I
> think this is just moving an existing problem around and asking the
> wrong set of folks to help solve the problem. I think a more effective
> change would be for ADs to expect WGs to ensure the docs are thoroughly
> reviewed earlier in the stage by the WG.
> 
>  [/MB]
> >
> >> Then there is a question: will Gen-ART re-review at the time of IETF
> >> Last Call as well as when the draft reaches the IESG agenda?
> >
> > My initial though was that we'd do the same as we do today for drafts
> that change, i.e., re-review by the same person. The likelihood for
> draft changes is of course bigger when starting from the WGLC. Another
> possible work load increase to watch for. I'd be interested in your
> thoughts on other approaches to deal with re-reviews though.
> >
> > Jari
> >

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 27 May 2013 07:55:35 +1200

On 26/05/2013 20:57, Romascanu, Dan (Dan) wrote:
...
> I do believe that there is however a solution to this
> problem, but it does not come for free either. Let the WGLC
> take place as it happens right now, but move the Expert
> Reviews (a.k.a. Directorate Reviews) also within the
> responsibility of the WG, after the WGLC completes. 

I think this would be a very good compromise - ask WG chairs
to obtain and respond to the cross-area reviews (including
Gen-ART) just before requesting AD review. This would mean
we reviewers would be looking at reasonably finished work
as we do now, and it should incent the WG to take notice of
the reviews, and relieve the AD to some extent too.

Very good suggestion!

    Brian

---

From: Gonzalo Camarillo <[email-masked]>
Date: Mon, 27 May 2013 10:42:08 +0300

Hi,

> I do believe that there is however a solution to this problem, but it
> does not come for free either. Let the WGLC take place as it happens
> right now, but move the Expert Reviews (a.k.a. Directorate Reviews)
> also within the responsibility of the WG, after the WGLC completes.
> This will make the WG cycle longer by a few weeks, but will allow for
> the Directorate comments to be processed by the WG, but on a stable
> version, which the WG considers as having passed the WGLC.

what Dan is proposing here is already happening to a certain extent in
RAI, in particular in the XRBLOCK WG, which Dan cochairs. The chairs
request Benoit's directorate to perform a review before they send the
documents to me (i.e., before pub req).

Cheers,

Gonzalo

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Mon, 27 May 2013 07:57:49 +0000

Indeed. In the XRBLOCK WG we are actually soliciting both PMDIR and SDP Directorate reviews, and update the documents according to the directorate comments before submitting them to the IESG. So we can say that there is some runing code for this proposal.

Regards,

Dan

> From: Gonzalo Camarillo [mailto:[email-masked]]
> Sent: Monday, May 27, 2013 10:42 AM
> To: Romascanu, Dan (Dan)
> Cc: Mary Barnes; Jari Arkko; General Area Review Team; The IESG

---

From: Jari Arkko <[email-masked]>
Date: Mon, 27 May 2013 11:17:33 +0300

Dan, Brian,

> the reviews, and relieve the AD to some extent too.

I agree. I like this model because it puts the chairs and the WG in charge. Thanks, Dan!

The other potentially helpful technique is to let the WG chairs have some room to decide when to ask for reviews. The chairs typically have some understanding of where their documents are in terms of readiness :-) even when opening a last call. A complex, controversial document is probably better off first complete its round of WGLC(s) before giving it to directorates...

Jari

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Mon, 27 May 2013 09:06:05 +0000

> From: Jari Arkko [mailto:[email-masked]]
> Sent: Monday, May 27, 2013 11:18 AM
> To: Brian E Carpenter
> Cc: Romascanu, Dan (Dan); General Area Review Team; Mary Barnes; The
> IESG
> > I think this would be a very good compromise - ask WG chairs to obtain
> > and respond to the cross-area reviews (including
> > Gen-ART) just before requesting AD review. This would mean we
> > reviewers would be looking at reasonably finished work as we do now,
> > and it should incent the WG to take notice of the reviews, and relieve
> > the AD to some extent too.
> >
> > Very good suggestion!
> 
> The other potentially helpful technique is to let the WG chairs have
> some room to decide when to ask for reviews. The chairs typically have
> some understanding of where their documents are in terms of readiness :-
> ) even when opening a last call. A complex, controversial document is
> probably better off first complete its round of WGLC(s) before giving it
> to directorates...
> 
> Jari

[[DR]] My experience is that WG chairs already feel that they have the room to ask for early directorate reviews for more complex documents - this is at least my experience with the MIB doctors reviews. These are considered however 'early reviews' according to the current process and they are an exception. If we go ahead with this proposal, we make of reviews performed after WGLC but before submission to the IESG the rule. 

Regards,

Dan