From andy at hexten.net Thu Nov 1 03:19:36 2007
From: andy at hexten.net (Andy Armstrong)
Date: Thu, 1 Nov 2007 08:19:36 +0000
Subject: [tap-l] Mad TAP proposal
In-Reply-To: <200710311711.31585.ewilhelm@cpan.org>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<200710311711.31585.ewilhelm@cpan.org>
Message-ID: <A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
On 1 Nov 2007, at 00:11, Eric Wilhelm wrote:
>> But what about a more general mechanism? A TAP directive that means
>> 'schedule these other tests'. So then you'd have a controller test
>> which was the only one directly visible to Test::Harness and that'd
>> decide which other tests to run.
>
> It sounds like it would be re-creating a lot of the same functionality
> needed for declarative extra testing and/or Test::Manifest.
>
> http://scratchcomputing.com/tmp/extra_testing.txt
>
> I would rather it not be a TAP directive. Yes, it abstracts
> the "manifesting" into TAP, but it still requires some code to run to
> determine the manifest, and will therefore be less introspectable.
Yeah - I slept on it. It should be TSP - Test Steering Protocol. If a
test emits TAP it's a test, if it emits TSP it's a controller. TAP
should stay pure.
--
Andy Armstrong, Hexten
From andy at hexten.net Thu Nov 1 03:39:47 2007
From: andy at hexten.net (Andy Armstrong)
Date: Thu, 1 Nov 2007 08:39:47 +0000
Subject: [tap-l] Mad TAP proposal
In-Reply-To: <A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<200710311711.31585.ewilhelm@cpan.org>
<A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
Message-ID: <874F091D-08C2-40E1-A759-2F7C6A5BC69F@hexten.net>
On 1 Nov 2007, at 08:19, Andy Armstrong wrote:
> Yeah - I slept on it. It should be TSP - Test Steering Protocol. If a
> test emits TAP it's a test, if it emits TSP it's a controller. TAP
> should stay pure.
http://testanything.org/wiki/index.php/TSP_Test_Steering_Protocol
--
Andy Armstrong, Hexten
From andy at hexten.net Thu Nov 1 09:26:57 2007
From: andy at hexten.net (Andy Armstrong)
Date: Thu, 1 Nov 2007 14:26:57 +0000
Subject: [tap-l] Mad TAP proposal
In-Reply-To: <4729DED2.5050408@plusthree.com>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<200710311711.31585.ewilhelm@cpan.org>
<A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
<4729DED2.5050408@plusthree.com>
Message-ID: <1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net>
On 1 Nov 2007, at 14:12, Michael Peters wrote:
> Andy Armstrong wrote:
>
>> Yeah - I slept on it. It should be TSP - Test Steering Protocol. If a
>> test emits TAP it's a test, if it emits TSP it's a controller. TAP
>> should stay pure.
>
> Sounds to me like it should just be a custom test harness. If the
> user needs to
> dynamically figure out which tests to run, then why not create a
> custom harness
> which will do what they need. This way they can have control over
> not only which
> files to run, but which routines (if they're using Test::Class),
> etc, etc.
> Writing your own harness is trivial now. I'd say writing something
> to emit a new
> protocol is no easier than writing a script that picks the test
> files/subs to run.
Yeah, but from the user's PoV this is pretty easy:
# t/controller.t
use Test::Steering;
include ('xt/frob') if frob_avail();
include ('xt/slow') if all_the_time_in_the_world();
And it's not too hard to support in the harness.
--
Andy Armstrong, Hexten
From mpeters at plusthree.com Thu Nov 1 09:32:13 2007
From: mpeters at plusthree.com (Michael Peters)
Date: Thu, 01 Nov 2007 10:32:13 -0400
Subject: [tap-l] Mad TAP proposal
In-Reply-To: <1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<200710311711.31585.ewilhelm@cpan.org>
<A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
<4729DED2.5050408@plusthree.com>
<1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net>
Message-ID: <4729E36D.9020707@plusthree.com>
Andy Armstrong wrote:
> Yeah, but from the user's PoV this is pretty easy:
>
> # t/controller.t
> use Test::Steering;
>
> include ('xt/frob') if frob_avail();
> include ('xt/slow') if all_the_time_in_the_world();
But if they just do their own harness it becomes:
push('xt/frob', @tests)if frob_avail();
push('xt/slow) if all_the_time_in_the_world();
Test::Harness::runtests(@tests);
--
Michael Peters
Developer
Plus Three, LP
From pagaltzis at gmx.de Thu Nov 1 09:31:32 2007
From: pagaltzis at gmx.de (A. Pagaltzis)
Date: Thu, 1 Nov 2007 15:31:32 +0100
Subject: [tap-l] Mad TAP proposal
In-Reply-To: <4729DED2.5050408@plusthree.com>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<200710311711.31585.ewilhelm@cpan.org>
<A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
<4729DED2.5050408@plusthree.com>
Message-ID: <20071101143132.GC2710@klangraum>
* Michael Peters <mpeters at plusthree.com> [2007-11-01 15:15]:
> Sounds to me like it should just be a custom test harness. If
> the user needs to dynamically figure out which tests to run,
> then why not create a custom harness which will do what they
> need.
I was going to say the same. Then it occured to me that if you
want to make such a harness reusable and you want to make the
test plans introspectable, then such a harness needs to parse
some sort of config. Which is very nearly what TSP is.
I just don?t see the ?protocol? part. I?d rather think of it in
terms of the Test Plan Configuration format, and that this be
something specific to TAP::Harness rather than documented widely
as something common to TAP implementations. It veers into
operational questions that I don?t think are applicable to every
scenario where TAP is in use. Because?
> This way they can have control over not only which files to
> run, but which routines (if they're using Test::Class), etc,
> etc.
? TAP encompasses scenarios where the notion of files and test
routines doesn?t even compute.
> Writing your own harness is trivial now. I'd say writing
> something to emit a new protocol is no easier than writing a
> script that picks the test files/subs to run.
Agree. Putting this sort of functionality in the default harness
*is* a good idea so people don?t need to reinvent a common wheel,
and so you can use a declarative language to define this sort of
thing. But just make this a config for TAP::Harness.
Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>
From mpeters at plusthree.com Thu Nov 1 09:12:34 2007
From: mpeters at plusthree.com (Michael Peters)
Date: Thu, 01 Nov 2007 10:12:34 -0400
Subject: [tap-l] Mad TAP proposal
In-Reply-To: <A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<200710311711.31585.ewilhelm@cpan.org>
<A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
Message-ID: <4729DED2.5050408@plusthree.com>
Andy Armstrong wrote:
> Yeah - I slept on it. It should be TSP - Test Steering Protocol. If a
> test emits TAP it's a test, if it emits TSP it's a controller. TAP
> should stay pure.
Sounds to me like it should just be a custom test harness. If the user needs to
dynamically figure out which tests to run, then why not create a custom harness
which will do what they need. This way they can have control over not only which
files to run, but which routines (if they're using Test::Class), etc, etc.
Writing your own harness is trivial now. I'd say writing something to emit a new
protocol is no easier than writing a script that picks the test files/subs to run.
--
Michael Peters
Developer
Plus Three, LP
From andy at hexten.net Thu Nov 1 09:43:59 2007
From: andy at hexten.net (Andy Armstrong)
Date: Thu, 1 Nov 2007 14:43:59 +0000
Subject: [tap-l] Mad TAP proposal
In-Reply-To: <20071101143132.GC2710@klangraum>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<200710311711.31585.ewilhelm@cpan.org>
<A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
<4729DED2.5050408@plusthree.com> <20071101143132.GC2710@klangraum>
Message-ID: <F0A73DB1-CC9C-4F33-ACAA-02BE7A6B24CB@hexten.net>
On 1 Nov 2007, at 14:31, A. Pagaltzis wrote:
>> This way they can have control over not only which files to
>> run, but which routines (if they're using Test::Class), etc,
>> etc.
>
> ? TAP encompasses scenarios where the notion of files and test
> routines doesn?t even compute.
Yes - agree 100%. Hence it shouldn't be part of TAP.
>> Writing your own harness is trivial now. I'd say writing
>> something to emit a new protocol is no easier than writing a
>> script that picks the test files/subs to run.
>
> Agree. Putting this sort of functionality in the default harness
> *is* a good idea so people don?t need to reinvent a common wheel,
> and so you can use a declarative language to define this sort of
> thing. But just make this a config for TAP::Harness.
The config needs to be dynamic at test time - so it might as well be a
script that runs and outputs a description of which tests to run,
right? So isn't that nearly TSP? :)
--
Andy Armstrong, Hexten
From andy at hexten.net Thu Nov 1 09:50:53 2007
From: andy at hexten.net (Andy Armstrong)
Date: Thu, 1 Nov 2007 14:50:53 +0000
Subject: [tap-l] Mad TAP proposal
In-Reply-To: <4729E36D.9020707@plusthree.com>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<200710311711.31585.ewilhelm@cpan.org>
<A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
<4729DED2.5050408@plusthree.com>
<1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net>
<4729E36D.9020707@plusthree.com>
Message-ID: <4CC9406A-F2E0-4464-A10F-AFEE56FC93DD@hexten.net>
On 1 Nov 2007, at 14:32, Michael Peters wrote:
> But if they just do their own harness it becomes:
>
> push('xt/frob', @tests)if frob_avail();
> push('xt/slow) if all_the_time_in_the_world();
> Test::Harness::runtests(@tests);
Yes, they could. But if you write your own harness you presumably
lose, e.g., the advantages of all the different ways you can run the
tests via prove. And presumably you couldn't use tk-prove or whatever
it's called.
It's mixing test logic into harness logic. Isn't it better for the
tests to be completely self describing and keep the harness generic?
Then your tests will work with any Test::Harness based tool and any
compatible TAP harness.
--
Andy Armstrong, Hexten
From mpeters at plusthree.com Thu Nov 1 10:38:06 2007
From: mpeters at plusthree.com (Michael Peters)
Date: Thu, 01 Nov 2007 11:38:06 -0400
Subject: [tap-l] Mad TAP proposal
In-Reply-To: <4CC9406A-F2E0-4464-A10F-AFEE56FC93DD@hexten.net>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<200710311711.31585.ewilhelm@cpan.org>
<A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
<4729DED2.5050408@plusthree.com>
<1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net>
<4729E36D.9020707@plusthree.com>
<4CC9406A-F2E0-4464-A10F-AFEE56FC93DD@hexten.net>
Message-ID: <4729F2DE.4020300@plusthree.com>
Andy Armstrong wrote:
> On 1 Nov 2007, at 14:32, Michael Peters wrote:
>> But if they just do their own harness it becomes:
>>
>> push('xt/frob', @tests)if frob_avail();
>> push('xt/slow) if all_the_time_in_the_world();
>> Test::Harness::runtests(@tests);
>
>
> Yes, they could. But if you write your own harness you presumably lose,
> e.g., the advantages of all the different ways you can run the tests
> via prove. And presumably you couldn't use tk-prove or whatever it's
> called.
Ok, that's true.
> It's mixing test logic into harness logic. Isn't it better for the tests
> to be completely self describing and keep the harness generic? Then your
> tests will work with any Test::Harness based tool and any compatible TAP
> harness.
Isn't this the problem that SKIP was designed to fix? If it needs to dynamically
determined if a test should run, isn't the test itself the best place to do
that? xt tests shouldn't be run unless explicity done so by the runner of the
tests (a developer) who presumably knows that they are doing.
Instead of "something else" telling the harness which tests to run, why not have
the harness run them all and the individual tests can decide if they don't want
to go through with it or not. If "something else" tells the harness which tests
to run, we loose the information about which tests didn't run and why.
Testing in Perl is really nice and simple and I'm not seeing the benefits or the
problems being solved with this complexity that can't already be solved using
something else.
--
Michael Peters
Developer
Plus Three, LP
From pagaltzis at gmx.de Thu Nov 1 10:44:29 2007
From: pagaltzis at gmx.de (A. Pagaltzis)
Date: Thu, 1 Nov 2007 16:44:29 +0100
Subject: [tap-l] Mad TAP proposal
In-Reply-To: <F0A73DB1-CC9C-4F33-ACAA-02BE7A6B24CB@hexten.net>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<200710311711.31585.ewilhelm@cpan.org>
<A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
<4729DED2.5050408@plusthree.com> <20071101143132.GC2710@klangraum>
<F0A73DB1-CC9C-4F33-ACAA-02BE7A6B24CB@hexten.net>
Message-ID: <20071101154429.GD2710@klangraum>
* Andy Armstrong <andy at hexten.net> [2007-11-01 15:50]:
> The config needs to be dynamic at test time - so it might as
> well be a script that runs and outputs a description of which
> tests to run, right?
But it only needs to be dynamic in a minority of cases. So it
seems to me it should be the other way around ? rather than run
a script to emit the config, have a config that has a directive
to run a script and include its output as part of the config.
That makes it easier for other tools to analyse the test configs.
> So isn't that nearly TSP? :)
In some ways. I just don?t think the harness should look at the
output of tests to check if it?s TSP rather than TAP. The list of
tests to run should be determined up front, and the test scripts
should only ever output TAP.
Mixing them with scripts that output TSP feels like a confusion
of concerns to me.
For the configs there are also a number of issues like dealing
well with multiple possible configurations and resolution of
relative paths (esp. in compound configs).
I think this needs a little bit more design. And since YAML is
making its way into TAP anyway, maybe it should be YAML-based
rather than a completely ad-hoc TAPish format?
Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>
From andy at hexten.net Thu Nov 1 11:20:43 2007
From: andy at hexten.net (Andy Armstrong)
Date: Thu, 1 Nov 2007 16:20:43 +0000
Subject: [tap-l] Mad TAP proposal
In-Reply-To: <4729F2DE.4020300@plusthree.com>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<200710311711.31585.ewilhelm@cpan.org>
<A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
<4729DED2.5050408@plusthree.com>
<1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net>
<4729E36D.9020707@plusthree.com>
<4CC9406A-F2E0-4464-A10F-AFEE56FC93DD@hexten.net>
<4729F2DE.4020300@plusthree.com>
Message-ID: <55D11C32-F98C-4010-A16E-D688B9EEB602@hexten.net>
On 1 Nov 2007, at 15:38, Michael Peters wrote:
> Isn't this the problem that SKIP was designed to fix? If it needs to
> dynamically
> determined if a test should run, isn't the test itself the best
> place to do
> that? xt tests shouldn't be run unless explicity done so by the
> runner of the
> tests (a developer) who presumably knows that they are doing.
The feature request that started all this relates to a large test
suite most of which needs to be skipped if a graphical environment
isn't available - so presumably it'd involve executing hundreds of
tests just to find out that they skipped.
> Instead of "something else" telling the harness which tests to run,
> why not have
> the harness run them all and the individual tests can decide if they
> don't want
> to go through with it or not. If "something else" tells the harness
> which tests
> to run, we loose the information about which tests didn't run and why.
Having to run each test to find out it didn't want to run doesn't
scale well. Having a mechanism to dynamically select subsets of tests
can only encourage people to write more tests I'd have thought.
If we decide on a set of rules for which xt/* tests are executed under
which circumstances it also gives us a language in which to express
those rules - so instead of being hard wired into TAP::Harness we'll
have a little TSP document that could be used with any TAP parser.
I've also seen a discussion of test ordering and the practice of
numbering tests to guarantee a ordering recently; ordering and more
complex dependencies could also be handled by a t/controller.t that
emits TSP.
> Testing in Perl is really nice and simple and I'm not seeing the
> benefits or the
> problems being solved with this complexity that can't already be
> solved using
> something else.
You could presumably have said the same about the idea of TAP based
testing. You can after all just have a bunch of programs that pass or
fail and run them from the shell.
I'm being slightly facetious there :)
I agree that parsimony should guide our decisions but I'm not finding
it hard to think of use cases for TSP.
--
Andy Armstrong, Hexten
From andy at hexten.net Thu Nov 1 11:33:58 2007
From: andy at hexten.net (Andy Armstrong)
Date: Thu, 1 Nov 2007 16:33:58 +0000
Subject: [tap-l] Mad TAP proposal
In-Reply-To: <20071101154429.GD2710@klangraum>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<200710311711.31585.ewilhelm@cpan.org>
<A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
<4729DED2.5050408@plusthree.com> <20071101143132.GC2710@klangraum>
<F0A73DB1-CC9C-4F33-ACAA-02BE7A6B24CB@hexten.net>
<20071101154429.GD2710@klangraum>
Message-ID: <544AD30E-C612-41F2-8DCA-D6C95F996A9F@hexten.net>
On 1 Nov 2007, at 15:44, A. Pagaltzis wrote:
> * Andy Armstrong <andy at hexten.net> [2007-11-01 15:50]:
>> The config needs to be dynamic at test time - so it might as
>> well be a script that runs and outputs a description of which
>> tests to run, right?
>
> But it only needs to be dynamic in a minority of cases. So it
> seems to me it should be the other way around ? rather than run
> a script to emit the config, have a config that has a directive
> to run a script and include its output as part of the config.
Well if it doesn't need to be dynamic it doesn't need to exist at all
- you just have the tests you want to run.
> That makes it easier for other tools to analyse the test configs.
>
>> So isn't that nearly TSP? :)
>
> In some ways. I just don?t think the harness should look at the
> output of tests to check if it?s TSP rather than TAP. The list of
> tests to run should be determined up front
Well that's desirable certainly - but isn't it also valuable to be
able to dynamically include include or exclude large numbers of tests
when you need to? And to do so using a well defined protocol?
> , and the test scripts
> should only ever output TAP.
Yeah, I have sympathy with that. I really don't want to complicate
things gratuitously.
> Mixing them with scripts that output TSP feels like a confusion
> of concerns to me.
I can see that too. In that sense the existing "Bail Out" directive
feels like an anomaly too.
> For the configs there are also a number of issues like dealing
> well with multiple possible configurations and resolution of
> relative paths (esp. in compound configs).
Yup - not impossible though. You can use HTML-like semantics -
relative paths are resolved relative to the config, abs paths relative
to some well defined project root.
> I think this needs a little bit more design. And since YAML is
> making its way into TAP anyway, maybe it should be YAML-based
> rather than a completely ad-hoc TAPish format?
Oh sure - that'd be fine. The choice of representation is somewhat
orthogonal to whether it's a good idea in the first place though.
--
Andy Armstrong, Hexten
From geoff at modperlcookbook.org Thu Nov 1 11:53:45 2007
From: geoff at modperlcookbook.org (Geoffrey Young)
Date: Thu, 01 Nov 2007 12:53:45 -0400
Subject: [tap-l] Mad TAP proposal
In-Reply-To: <55D11C32-F98C-4010-A16E-D688B9EEB602@hexten.net>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<200710311711.31585.ewilhelm@cpan.org>
<A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
<4729DED2.5050408@plusthree.com>
<1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net>
<4729E36D.9020707@plusthree.com>
<4CC9406A-F2E0-4464-A10F-AFEE56FC93DD@hexten.net>
<4729F2DE.4020300@plusthree.com>
<55D11C32-F98C-4010-A16E-D688B9EEB602@hexten.net>
Message-ID: <472A0499.5020805@modperlcookbook.org>
Andy Armstrong wrote:
> On 1 Nov 2007, at 15:38, Michael Peters wrote:
>> Isn't this the problem that SKIP was designed to fix? If it needs to
>> dynamically
>> determined if a test should run, isn't the test itself the best place
>> to do
>> that? xt tests shouldn't be run unless explicity done so by the runner
>> of the
>> tests (a developer) who presumably knows that they are doing.
>
> The feature request that started all this relates to a large test suite
> most of which needs to be skipped if a graphical environment isn't
> available - so presumably it'd involve executing hundreds of tests just
> to find out that they skipped.
fwiw, Apache-Test respects an all.t file, which looks something like this:
use Apache::Test;
# skip all tests unless mod_perl and mod_apreq are available
plan tests => 1, need_module('mod_perl.c', 'mod_apreq2.c');
ok 1;
the test is silently executed by our harness. if the plan succeeds and
the test run, A-T will run all tests in the current directory. if the
plan fails, the skip message is printed and all tests in the directory
are never executed.
it sits next to the tests as
t/foo/all.t
t/foo/01do.t
t/foo/02done.t
or whatnot.
anyway, its not the best way to handle things necessarily (I didn't
write it, and it's over 6 years old now :) but it it does come in handy
every once in a while, so the idea in general is probably sound.
--Geoff
From andy at hexten.net Thu Nov 1 12:09:07 2007
From: andy at hexten.net (Andy Armstrong)
Date: Thu, 1 Nov 2007 17:09:07 +0000
Subject: [tap-l] Mad TAP proposal
In-Reply-To: <472A0499.5020805@modperlcookbook.org>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<200710311711.31585.ewilhelm@cpan.org>
<A58BA2BF-664D-4D33-9F5D-075BAC807A78@hexten.net>
<4729DED2.5050408@plusthree.com>
<1EDC7601-D18A-4434-84F0-982141307DD8@hexten.net>
<4729E36D.9020707@plusthree.com>
<4CC9406A-F2E0-4464-A10F-AFEE56FC93DD@hexten.net>
<4729F2DE.4020300@plusthree.com>
<55D11C32-F98C-4010-A16E-D688B9EEB602@hexten.net>
<472A0499.5020805@modperlcookbook.org>
Message-ID: <7C66B913-171F-45DF-8A0C-52FE1715474A@hexten.net>
On 1 Nov 2007, at 16:53, Geoffrey Young wrote:
> anyway, its not the best way to handle things necessarily (I didn't
> write it, and it's over 6 years old now :) but it it does come in
> handy
> every once in a while, so the idea in general is probably sound.
So that's another use case for TSP right? :)
--
Andy Armstrong, Hexten
From scratchcomputing at gmail.com Thu Nov 1 12:52:20 2007
From: scratchcomputing at gmail.com (Eric Wilhelm)
Date: Thu, 1 Nov 2007 10:52:20 -0700
Subject: [tap-l] dynamic test selection
In-Reply-To: <544AD30E-C612-41F2-8DCA-D6C95F996A9F@hexten.net>
References: <638DA795-0F88-4051-8BA3-79A32F3F9508@hexten.net>
<20071101154429.GD2710@klangraum>
<544AD30E-C612-41F2-8DCA-D6C95F996A9F@hexten.net>
Message-ID: <200711011052.21011.ewilhelm@cpan.org>
# from Andy Armstrong
# on Thursday 01 November 2007 09:33:
>On 1 Nov 2007, at 15:44, A. Pagaltzis wrote:
>> * Andy Armstrong <andy at hexten.net> [2007-11-01 15:50]:
>>> The config needs to be dynamic at test time - so it might as
>>> well be a script...
>> But it only needs to be dynamic in a minority of cases. So it
>> seems to me it should be the other way around ? rather than run
>> a script to emit the config, have a config that has a directive
>> to run a script and include its output as part of the config.
This would be preferable for introspectability, yes.
>Well if it doesn't need to be dynamic it doesn't need to exist at all
>- you just have the tests you want to run.
But rather than say "not dynamic", let's say "not arbitrary". The
config is always dynamic -- e.g. a canned method require_gui() would
need to check whatever cross-platform checks and also maybe consult the
tester's "allowables" settings.
The distinction between arbitrary and dynamic is that arbitrary code
hinders introspection while dynamic can mean that it simply refers to a
pre-determined check routine (e.g. the Apache Ant 'exec' directive
introduces "arbitrary" into what is otherwise a 100% declarative
manifest.)
>> That makes it easier for other tools to analyse the test configs.
Introspection. Not only can we decide whether or not to run the gui
tests, but we know which ones require the gui. If you just source
`controller.pl` (arbitrary code) and get a valid list for the current
environment, you have no way to tell which of the selections are
affected by which bits of the environment. To determine the gui tests,
you have to turn-off the $DISPLAY (won't work on windows, and possibly
mac) run `controller.pl` again and then compare the new output. Note
the difficulty of handling "I have a gui, but I don't want the tests
messing with it" on platforms like windows.
With a declarative scheme, we have a tree of data and we can simply ask
which selections require_gui(). Those tests can then be run as their
own group, or even simply examined for some other interesting
information.
And even in the standard use-case, if the tester has opted to not have
her gui interrupted by the tests, they don't run. The declarative
scheme makes it easy to "do the right thing" because author education
is reduced to simply: "set require_gui for the tests that need a gui"
(as opposed to e.g. "check $DISPLAY and then check the preferences
file, etc.") It's a lot harder to get it wrong if it involves code
that you didn't write.
>>> So isn't that nearly TSP? :)
>> ... don?t think the harness should look at the
>> output of tests to check if it?s TSP rather than TAP. The list of
>> tests to run should be determined up front
>
>Well that's desirable certainly - but isn't it also valuable to be
>able to dynamically include include or exclude large numbers of tests
>when you need to? And to do so using a well defined protocol?
Possibly, but it shouldn't be part of the t/*.t files and I think we
would do well to make it declarative rather than a protocol emitted by
arbitrary code.
>> , and the test scripts
>> should only ever output TAP.
+1
>> ... YAML ...
>Oh sure - that'd be fine. The choice of representation is somewhat
>orthogonal to whether it's a good idea in the first place though.
I think we've decided that "selectable tests" are a good idea WRT "extra
testing." Declarative requirements are probably just as useful WRT
normal tests. Note though, that Martin mentions "xt/" in the bug
report ;-)
--Eric
--
To a database person, every nail looks like a thumb.
--Jamie Zawinski
---------------------------------------------------
http://scratchcomputing.com
---------------------------------------------------
From mmoore at clusterresources.com Wed Nov 14 15:53:42 2007
From: mmoore at clusterresources.com (Michael Moore)
Date: Wed, 14 Nov 2007 13:53:42 -0700
Subject: [tap-l] What's the best way to do logging?
Message-ID: <473B6056.4060608@clusterresources.com>
Hello,
I'm glad to have found a TAP mailing list! If Perl specific questions
aren't appropriate for this list, please let me know and forgive me.
I'm working on an automated test suite for some of our software using
Perl, I had been using Test::Harness and Test::Harness::Straps but saw
just yesterday that Test::Harness::Straps won't be developed anymore.
Since we want to use these tests for some time, I'd like to move to
something that's not discontinued. Could someone here tell me if
TAP::Harness and TAP::Parser is what I should be looking at?
I need to do fairly extensive logging. Our tests will normally be run as
nightly regression tests, and we want to be able to compare results from
different nights.
On a test by test basis I am currently logging all of the Test::More
functions. I log the test comment and the result. I also log the many
external programs that are called, the STDOUT, STDERR, and return
status. There are several hundred batch tests, which currently use
Test::Harness's execute_tests. I log which test was called from the
batch test, and what the result for each was.
I've added the logging functionality by overriding the functions,
grabbing the results and logging them before returning them to the
calling function.
I've done this by writing a module which I include instead of
Test::More, Test::Harness and Test::Harness::Straps.
Is TAP::Harness and TAP::Parser what I should be looking at? Or is there
an easier way to do logging?
Thank you,
Michael Moore
From andy at hexten.net Wed Nov 14 16:22:02 2007
From: andy at hexten.net (Andy Armstrong)
Date: Wed, 14 Nov 2007 21:22:02 +0000
Subject: [tap-l] What's the best way to do logging?
In-Reply-To: <473B6056.4060608@clusterresources.com>
References: <473B6056.4060608@clusterresources.com>
Message-ID: <77531E48-D8CF-4418-B5BF-836B63992659@hexten.net>
On 14 Nov 2007, at 20:53, Michael Moore wrote:
> I'm glad to have found a TAP mailing list! If Perl specific questions
> aren't appropriate for this list, please let me know and forgive me.
>
>
> I'm working on an automated test suite for some of our software using
> Perl, I had been using Test::Harness and Test::Harness::Straps but
> saw
> just yesterday that Test::Harness::Straps won't be developed anymore.
>
> Since we want to use these tests for some time, I'd like to move to
> something that's not discontinued. Could someone here tell me if
> TAP::Harness and TAP::Parser is what I should be looking at?
Yes. By design TAP::* replaces the existing Straps mechanism. If it
won't do what you want it to do we need to log at that design.
> I need to do fairly extensive logging. Our tests will normally be
> run as
> nightly regression tests, and we want to be able to compare results
> from
> different nights.
>
> On a test by test basis I am currently logging all of the Test::More
> functions. I log the test comment and the result. I also log the many
> external programs that are called, the STDOUT, STDERR, and return
> status. There are several hundred batch tests, which currently use
> Test::Harness's execute_tests. I log which test was called from the
> batch test, and what the result for each was.
>
> I've added the logging functionality by overriding the functions,
> grabbing the results and logging them before returning them to the
> calling function.
>
> I've done this by writing a module which I include instead of
> Test::More, Test::Harness and Test::Harness::Straps.
>
> Is TAP::Harness and TAP::Parser what I should be looking at? Or is
> there
> an easier way to do logging?
I think TAP::Harness is the way forwards. Michael Peters wrote
TAP::Harness::Archive[1] which is designed to capture everything from
a test run. It's used to upload test results to Smolder[2] - which may
also be of interest to you.
Unfortunately the latest release of Test::Harness has broken
T::H::Archive. There's a fixed version in the Test::Harness subversion
repo - I guess I need to give Michael a poke. In the mean time you can
get the latest version here:
http://svn.hexten.net/tapx/TAP-Harness-Archive/trunk/
May I suggest that you start by reading the documentation for
TAP::Harness::Archive and see if that might do what you want?
[1] http://search.cpan.org/~wonko/TAP-Harness-Archive/
[2] http://sourceforge.net/projects/smolder
From mmoore at clusterresources.com Thu Nov 15 12:24:47 2007
From: mmoore at clusterresources.com (Michael Moore)
Date: Thu, 15 Nov 2007 10:24:47 -0700
Subject: [tap-l] What's the best way to do logging?
In-Reply-To: <77531E48-D8CF-4418-B5BF-836B63992659@hexten.net>
References: <473B6056.4060608@clusterresources.com>
<77531E48-D8CF-4418-B5BF-836B63992659@hexten.net>
Message-ID: <473C80DF.2010409@clusterresources.com>
> Andy Armstrong wrote:
> On 14 Nov 2007, at 20:53, Michael Moore wrote:
>> Since we want to use these tests for some time, I'd like to move to
>> something that's not discontinued. Could someone here tell me if
>> TAP::Harness and TAP::Parser is what I should be looking at?
>
> Yes. By design TAP::* replaces the existing Straps mechanism. If it
> won't do what you want it to do we need to log at that design.
Great. As I've been looking into more, it looks like it'll be even
better than Test::Harness. In particular, it looks like it's easier to
get the results of tests out of TAP::Parser.
>> Is TAP::Harness and TAP::Parser what I should be looking at? Or is
>> there
>> an easier way to do logging?
>
> I think TAP::Harness is the way forwards. Michael Peters wrote
> TAP::Harness::Archive[1] which is designed to capture everything from
> a test run. It's used to upload test results to Smolder[2] - which may
> also be of interest to you.
>
> Unfortunately the latest release of Test::Harness has broken
> T::H::Archive. There's a fixed version in the Test::Harness subversion
> repo - I guess I need to give Michael a poke. In the mean time you can
> get the latest version here:
>
> http://svn.hexten.net/tapx/TAP-Harness-Archive/trunk/
>
> May I suggest that you start by reading the documentation for
> TAP::Harness::Archive and see if that might do what you want?
>
> [1] http://search.cpan.org/~wonko/TAP-Harness-Archive/
> [2] http://sourceforge.net/projects/smolder
Thanks I hadn't see TAP::Harness::Archive. It doesn't do exactly what I
need, but it was very useful as a reference. With is as an example, I
was able to get a sample test working with TAP::Harness.
Looks like it's going to do what we need. TAP::Harness here we come.
Thanks again,
Michael Moore
From mmoore at clusterresources.com Thu Nov 15 12:27:10 2007
From: mmoore at clusterresources.com (Michael Moore)
Date: Thu, 15 Nov 2007 10:27:10 -0700
Subject: [tap-l] What's the best way to do logging?
Message-ID: <473C816E.6000500@clusterresources.com>
> Andy Armstrong wrote:
> On 14 Nov 2007, at 20:53, Michael Moore wrote:
>> Since we want to use these tests for some time, I'd like to move to
>> something that's not discontinued. Could someone here tell me if
>> TAP::Harness and TAP::Parser is what I should be looking at?
>
> Yes. By design TAP::* replaces the existing Straps mechanism. If it
> won't do what you want it to do we need to log at that design.
Great. As I've been looking into more, it looks like it'll be even
better than Test::Harness. In particular, it looks like it's easier to
get the results of tests out of TAP::Parser.
>> Is TAP::Harness and TAP::Parser what I should be looking at? Or is
>> there
>> an easier way to do logging?
>
> I think TAP::Harness is the way forwards. Michael Peters wrote
> TAP::Harness::Archive[1] which is designed to capture everything from
> a test run. It's used to upload test results to Smolder[2] - which may
> also be of interest to you.
>
> Unfortunately the latest release of Test::Harness has broken
> T::H::Archive. There's a fixed version in the Test::Harness subversion
> repo - I guess I need to give Michael a poke. In the mean time you can
> get the latest version here:
>
> http://svn.hexten.net/tapx/TAP-Harness-Archive/trunk/
>
> May I suggest that you start by reading the documentation for
> TAP::Harness::Archive and see if that might do what you want?
>
> [1] http://search.cpan.org/~wonko/TAP-Harness-Archive/
> [2] http://sourceforge.net/projects/smolder
Thanks I hadn't see TAP::Harness::Archive. It doesn't do exactly what I
need, but it was very useful as a reference. With is as an example, I
was able to get a sample test working with TAP::Harness.
Looks like it's going to do what we need. TAP::Harness here we come.
Thanks again,
Michael Moore
From andy at hexten.net Thu Nov 15 12:30:36 2007
From: andy at hexten.net (Andy Armstrong)
Date: Thu, 15 Nov 2007 17:30:36 +0000
Subject: [tap-l] What's the best way to do logging?
In-Reply-To: <473C80DF.2010409@clusterresources.com>
References: <473B6056.4060608@clusterresources.com>
<77531E48-D8CF-4418-B5BF-836B63992659@hexten.net>
<473C80DF.2010409@clusterresources.com>
Message-ID: <2F4BE95B-48B0-4A19-9303-C78A07E5D933@hexten.net>
On 15 Nov 2007, at 17:24, Michael Moore wrote:
> Great. As I've been looking into more, it looks like it'll be even
> better than Test::Harness. In particular, it looks like it's easier to
> get the results of tests out of TAP::Parser.
Should be :)
>> [1] http://search.cpan.org/~wonko/TAP-Harness-Archive/
>> [2] http://sourceforge.net/projects/smolder
>
> Thanks I hadn't see TAP::Harness::Archive. It doesn't do exactly
> what I
> need, but it was very useful as a reference. With is as an example, I
> was able to get a sample test working with TAP::Harness.
>
> Looks like it's going to do what we need. TAP::Harness here we come.
Splendid! :)
--
Andy Armstrong, Hexten
From mmoore at clusterresources.com Mon Nov 19 16:43:25 2007
From: mmoore at clusterresources.com (Michael Moore)
Date: Mon, 19 Nov 2007 14:43:25 -0700
Subject: [tap-l] Nested tests with TAP::Harness and verbose output
Message-ID: <4742037D.9050708@clusterresources.com>
I've set up our batch tests to use TAP::Harness's runtests to run
sub-tests. We have many cases where BatchTestA calls BatchTestB which
calls TestC.
If I create a TAP::Harness object in both BatchTestA and BatchTestB, it
works fine (all good so far!).
I wanted to get a little more information out of our tests, so I passed
in a hash with the parameter "verbosity => 1" to each of the
TAP::Harness objects.
I found that If any TAP::Harness object other than the top level test
was set to verbose, it would break the TAP output. Is this expected
behavior?
Here is the resulting output:
=== Start output ===
./LevelA.bat
LevelB........
LevelC......
ok 1 - The test in LevelC
1..1
ok 2
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.00 usr 0.01 sys + 0.00 cusr
0.02 csys = 0.03 CPU)
Result: PASS
ok 1 - The test in LevelB
1..1
All 1 subtests passed
Test Summary Report
-------------------
LevelB.bat (Wstat: 0 Tests: 3 Failed: 2)
Failed test number(s): 1-2
Parse errors: Plan (1..1) must be at the beginning or end of the TAP
output
Tests out of sequence. Found (1) but expected (3)
More than one plan found in TAP output
Bad plan. You planned 1 tests but ran 3.
Files=1, Tests=3, 0 wallclock secs ( 0.01 usr 0.00 sys + 0.00 cusr
0.10 csys = 0.11 CPU)
Result: FAIL
ok 1 - The test in LevelA
1..1
=== End output ===
I'm not sure if attachments are OK on this list, so I'll just put the
three tests I used to narrow down the behavior here inline. If you save
these three tests and run LevelA.bat, you should see the described
behavior. If you comment out the line "verbosity => 1" in LevelB.bat,
the problem will go away.
-----LevelA.bat-----
#!/usr/bin/perl
use strict;
use warnings;
use TAP::Harness;
use Test::More;
plan('no_plan');
my %tapHarnessArgs = (
verbosity => 1,
);
my $batch = TAP::Harness->new(\%tapHarnessArgs);
$batch->runtests("LevelB.bat");
ok(1==1,"The test in LevelA");
-----LevelB.bat-----
#!/usr/bin/perl
use strict;
use warnings;
use TAP::Harness;
use Test::More;
plan('no_plan');
my %tapHarnessArgs = (
verbosity => 1,
);
my $batch = TAP::Harness->new(\%tapHarnessArgs);
$batch->runtests("LevelC.t");
ok(2==2,"The test in LevelB");
-----LevelC.t-----
#!/usr/bin/perl
use strict;
use warnings;
use Test::More;
plan('no_plan');
ok(3==3,"The test in LevelC");
Many Thanks,
Michael Moore
From mmoore at clusterresources.com Mon Nov 19 18:46:19 2007
From: mmoore at clusterresources.com (Michael Moore)
Date: Mon, 19 Nov 2007 16:46:19 -0700
Subject: [tap-l] Nested tests with TAP::Harness and verbose output]
Message-ID: <4742204B.7090502@clusterresources.com>
I tried to send this earlier, but I think it bounced, so here it is again.
I've set up our batch tests to use TAP::Harness's runtests to run
sub-tests. We have many cases where BatchTestA calls BatchTestB which
calls TestC.
If I create a TAP::Harness object in both BatchTestA and BatchTestB, it
works fine (all good so far!).
I wanted to get a little more information out of our tests, so I passed
in a hash with the parameter "verbosity => 1" to each of the
TAP::Harness objects.
I found that If any TAP::Harness object other than the top level test
was set to verbose, it would break the TAP output. Is this expected
behavior?
Here is the resulting output:
=== Start output ===
./LevelA.bat
LevelB........
LevelC......
ok 1 - The test in LevelC
1..1
ok 2
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.00 usr 0.01 sys + 0.00 cusr
0.02 csys = 0.03 CPU)
Result: PASS
ok 1 - The test in LevelB
1..1
All 1 subtests passed
Test Summary Report
-------------------
LevelB.bat (Wstat: 0 Tests: 3 Failed: 2)
Failed test number(s): 1-2
Parse errors: Plan (1..1) must be at the beginning or end of the TAP
output
Tests out of sequence. Found (1) but expected (3)
More than one plan found in TAP output
Bad plan. You planned 1 tests but ran 3.
Files=1, Tests=3, 0 wallclock secs ( 0.01 usr 0.00 sys + 0.00 cusr
0.10 csys = 0.11 CPU)
Result: FAIL
ok 1 - The test in LevelA
1..1
=== End output ===
I'm not sure if attachments are OK on this list, so I'll just put the
three tests I used to narrow down the behavior here inline. If you save
these three tests and run LevelA.bat, you should see the described
behavior. If you comment out the line "verbosity => 1" in LevelB.bat,
the problem will go away.
-----LevelA.bat-----
#!/usr/bin/perl
use strict;
use warnings;
use TAP::Harness;
use Test::More;
plan('no_plan');
my %tapHarnessArgs = (
verbosity => 1,
);
my $batch = TAP::Harness->new(\%tapHarnessArgs);
$batch->runtests("LevelB.bat");
ok(1==1,"The test in LevelA");
-----LevelB.bat-----
#!/usr/bin/perl
use strict;
use warnings;
use TAP::Harness;
use Test::More;
plan('no_plan');
my %tapHarnessArgs = (
verbosity => 1,
);
my $batch = TAP::Harness->new(\%tapHarnessArgs);
$batch->runtests("LevelC.t");
ok(2==2,"The test in LevelB");
-----LevelC.t-----
#!/usr/bin/perl
use strict;
use warnings;
use Test::More;
plan('no_plan');
ok(3==3,"The test in LevelC");
Many Thanks,
Michael Moore
From andy at hexten.net Tue Nov 20 04:26:31 2007
From: andy at hexten.net (Andy Armstrong)
Date: Tue, 20 Nov 2007 09:26:31 +0000
Subject: [tap-l] Nested tests with TAP::Harness and verbose output
In-Reply-To: <4742037D.9050708@clusterresources.com>
References: <4742037D.9050708@clusterresources.com>
Message-ID: <0F4D1395-D48C-4446-937E-408F8880789B@hexten.net>
On 19 Nov 2007, at 21:43, Michael Moore wrote:
> I've set up our batch tests to use TAP::Harness's runtests to run
> sub-tests. We have many cases where BatchTestA calls BatchTestB which
> calls TestC.
>
> If I create a TAP::Harness object in both BatchTestA and BatchTestB,
> it
> works fine (all good so far!).
>
> I wanted to get a little more information out of our tests, so I
> passed
> in a hash with the parameter "verbosity => 1" to each of the
> TAP::Harness objects.
>
> I found that If any TAP::Harness object other than the top level test
> was set to verbose, it would break the TAP output. Is this expected
> behavior?
Yeah, I think so. It looks as if you're feeding the console output
from the nested harness into the containing harness as TAP.
A test that runs under TAP::Harness should output syntactically clean
TAP.
You could have a look at Test::Steering to discover how to run nested
tests.
--
Andy Armstrong, Hexten
From adrianh at quietstars.com Tue Nov 20 11:41:26 2007
From: adrianh at quietstars.com (Adrian Howard)
Date: Tue, 20 Nov 2007 16:41:26 +0000
Subject: [tap-l] Deferred Plans
In-Reply-To: <373AE811-37BA-4571-ADB1-BF6D62F76C01@hexten.net>
References: <729237.83345.qm@web60811.mail.yahoo.com>
<373AE811-37BA-4571-ADB1-BF6D62F76C01@hexten.net>
Message-ID: <25E45D81-571E-4408-A345-4F4109FD53AD@quietstars.com>
Taken to tap-l from perl-qa...
On 19 Nov 2007, at 13:06, Andy Armstrong wrote:
[snip]
> I think we really need to reach a decision on
>
> http://testanything.org/wiki/index.php/Test_Groups versus
> http://testanything.org/wiki/index.php/Test_Blocks
Having just glanced at these again it's occurred that if you change
the "end" keyword in the block proposal to an "ok" / "not ok" (based
on treating the block of tests as a successful/unsuccessful test run)
you get:
1..4
begin 1 Object creation
1..2
ok 1 Object created OK
ok 2 Object isa Flunge::Twizzler
ok 1 Object creation
ok 2 Clone OK
begin 3 Methods
1..4
ok 1 has twizzle method
ok 2 has burnish method
ok 3 has spangle method
not ok 4 has frob method
not ok 3 Methods
ok 4 Resources released
which (I think) still reads nicely, and has the advantage of being
100% compatible with older parsers.
Makes more work for whatever is generating the TAP of course.
Just a thought...
Adrian
From andy at hexten.net Tue Nov 20 11:54:52 2007
From: andy at hexten.net (Andy Armstrong)
Date: Tue, 20 Nov 2007 16:54:52 +0000
Subject: [tap-l] Deferred Plans
In-Reply-To: <25E45D81-571E-4408-A345-4F4109FD53AD@quietstars.com>
References: <729237.83345.qm@web60811.mail.yahoo.com>
<373AE811-37BA-4571-ADB1-BF6D62F76C01@hexten.net>
<25E45D81-571E-4408-A345-4F4109FD53AD@quietstars.com>
Message-ID: <BED590D6-BC3F-4981-8F1C-310B24DB5C4A@hexten.net>
On 20 Nov 2007, at 16:41, Adrian Howard wrote:
> Having just glanced at these again it's occurred that if you change
> the "end" keyword in the block proposal to an "ok" / "not ok" (based
> on treating the block of tests as a successful/unsuccessful test run)
> you get:
>
> 1..4
> begin 1 Object creation
> 1..2
> ok 1 Object created OK
> ok 2 Object isa Flunge::Twizzler
> ok 1 Object creation
> ok 2 Clone OK
> begin 3 Methods
> 1..4
> ok 1 has twizzle method
> ok 2 has burnish method
> ok 3 has spangle method
> not ok 4 has frob method
> not ok 3 Methods
> ok 4 Resources released
>
> which (I think) still reads nicely, and has the advantage of being
> 100% compatible with older parsers.
>
> Makes more work for whatever is generating the TAP of course.
>
> Just a thought...
Oooh. I like that very much.
It means that if you're combining TAP you have to parse it to generate
the summary result and also that the indentation becomes significant -
which wasn't my original intention - but I'm OK with both of those
things.
--
Andy Armstrong, Hexten
From andy at hexten.net Tue Nov 20 19:35:01 2007
From: andy at hexten.net (Andy Armstrong)
Date: Wed, 21 Nov 2007 00:35:01 +0000
Subject: [tap-l] SKIP_ALL tests should not get hidden
In-Reply-To: <4743702C.90308@pobox.com>
References: <166662.31668.qm@web60815.mail.yahoo.com>
<155AA7B6-809C-4339-B56A-FADB09E21412@hexten.net>
<4743702C.90308@pobox.com>
Message-ID: <DA9D3F36-7868-429B-A46B-3238E29E3797@hexten.net>
On 20 Nov 2007, at 23:39, Michael G Schwern wrote:
>> Do we like that?
>
> Test::Harness 2 put it on it's own line mostly to avoid wrapping off
> the right
> side of the screen. I still lean in that direction.
Hmm. I'm kind of hooked on the new behaviour now. It puts a summary
column right where I can find it.
>>> However, TAP::Parser doesn't seem to have a &skip_all method
>>> (returning boolean or explanation? What if no skip reason given?).
>>
>> TAP::Parser->skip_all returns a true value if all tests are skipped.
>> It either returns the skip reason or '(no reason given)' as above.
>
> I don't like that it hard wires the default string, that's mixing up
> display
> with the API. It leaves developers having to write weird things
> like "if(
> $skip eq '(no reason given)' ) { ...then put in your own
> default... }".
>
> This is a situation where "0 but true" becomes handy. Or in this
> case, '' but
> true. It might make sense, in the case where there is no reason
> given, to
> return a boolean and string overloaded object which returns false
> for bool and
> the empty string as a string.
Agreed. I'll take a look at it - unless you want to.
--
Andy Armstrong, Hexten
From schwern at pobox.com Tue Nov 20 18:39:24 2007
From: schwern at pobox.com (Michael G Schwern)
Date: Tue, 20 Nov 2007 15:39:24 -0800
Subject: [tap-l] SKIP_ALL tests should not get hidden
In-Reply-To: <155AA7B6-809C-4339-B56A-FADB09E21412@hexten.net>
References: <166662.31668.qm@web60815.mail.yahoo.com>
<155AA7B6-809C-4339-B56A-FADB09E21412@hexten.net>
Message-ID: <4743702C.90308@pobox.com>
Andy Armstrong wrote:
> I've added logic that produces output like this:
>
> 13:54] andy $ prove t/sample-tests/skipall t/sample-tests/
> skipall_nomsg t/sample-tests/simple
> t/sample-tests/skipall..........skipped: rope
> t/sample-tests/skipall_nomsg....skipped: (no reason given)
> t/sample-tests/simple...........ok
> All tests successful.
> Files=3, Tests=5, 0 wallclock secs ( 0.01 usr 0.01 sys + 0.01 cusr
> 0.00 csys = 0.03 CPU)
> Result: PASS
>
> Do we like that?
Test::Harness 2 put it on it's own line mostly to avoid wrapping off the right
side of the screen. I still lean in that direction.
>> However, TAP::Parser doesn't seem to have a &skip_all method
>> (returning boolean or explanation? What if no skip reason given?).
>
> TAP::Parser->skip_all returns a true value if all tests are skipped.
> It either returns the skip reason or '(no reason given)' as above.
I don't like that it hard wires the default string, that's mixing up display
with the API. It leaves developers having to write weird things like "if(
$skip eq '(no reason given)' ) { ...then put in your own default... }".
This is a situation where "0 but true" becomes handy. Or in this case, '' but
true. It might make sense, in the case where there is no reason given, to
return a boolean and string overloaded object which returns false for bool and
the empty string as a string.
It occurs to me that an "X but true" CPAN module would be handy.
--
Don't try the paranormal until you know what's normal.
-- "Lords and Ladies" by Terry Prachett