PlUnit extension

Request For Comments
#1

Hi,

I have been using PlUnit for years to grade students’ projects.
On feature that I always wanted is to (1) run a test battery and (2) check the results afterward. Alas, as with many unit test libraries in several languages, PLUnit is a “my report is a bunch of printouts” beast.

I finally bit the bullet and delved deeply in PlUnit’s guts and wrenched out the necessary information. Now I can write

testing_run :-
    writeln('>>> Starting testing run.'),
    run_tests_and_report(report(Passed,
                                Failed,
                                FailedAssertions,
                                Blocked,
                                STO)),
    writeln('>>> Finished testing run.'),
    testing_run_report(Passed, Failed, FailedAssertions, Blocked, STO).

Forget the testing_run_report that just does some computation and other stuff, the new predicate is run_tests_and_report/1, which is a counterpart to run_tests/0. As you can see it unifies a report/5 term with the nice information that PlUnit does compute internally. The predicate always succeeds (modulo catastrophic errors), unlike run_tests, which may fails if some test fails; this is key to allow post processing of the overall test results by the user.

Now I can do post processing.

I am attaching a file with the relevant code. I am not well versed in SWI structured comments or conventions, so it will need some polishing. Of course, I’d lobby for inclusion of these entry points in PlUnit.

I can generate a pull request, once the code has been vetted by someone more expert… You-know-who-you-are :smile: :smiling_face_with_three_hearts:

It has been fun.

All the best

Marco

plunit_reporting.pl (5.2 KB)

2 Likes
#2

The overall idea is that if you want anything else than the standard report you use message_hook/3 to display the information as you like it (done by several people to create output that is compatible with various test environments). You can of course also use this hook to assert the info into the database and fetch it after the call to run_tests/0,1

That should work with just a little code. The exact information you are looking for is a little hard to assemble. That can be fixed using an additional print_message(silent, plunit(Term))) in report/0 in plunit.pl. I’m happy to accept a PR with improvements to the print_message/2 calls as long as they do not break compatibility.

1 Like
#3

Hi Jan

I read the manual entry for message_hook/3 and print_message/2. I am not sure they would fit the bill as they would, AFAIU, require the parsing of the message. Note that I do not want only a different printed report. I want access to the actual numbers. And I assembled the information I needed, and, I believe, all the information that is there, by providing a different predicate with a different semantics, in addition to those that are already in the package. The implementation was straightforward once I understood the logic behind run_tests/0.

All the best

Marco

PS. Next, much trickier, add a timeout option and wrap the tests in a catch(call_with_time_limit(...)...).

#4

Hi Marco,

The nice thing is that it does not require parsing the messages. The message infrastructure passes an abstract data term to print_message/2 that can be captured in message_hook/3. I’ve pushed a couple of enhancements to library(plunit) to provide a (silent) summary message and also make the ., !, etc. use the message infrastructure. Now we can do:

:- thread_local
    plunit_summary/1.

user:message_hook(plunit(summary(Dict)), _, _) :-
    asserta(plunit_summary(Dict)).
user:message_hook(plunit(_), _, _).

test_results(TestSet, Summary) :-
    setup_call_cleanup(
        true,
        ignore((   TestSet == all
               ->  run_tests
               ;   run_tests(TestSet)
              )),
        retract(plunit_summary(Summary))).

Example run:

?- test_results(all, R).
R = plunit{blocked:0, failed:0, failed_assertions:0, passed:16, sto:0}.

The fact that you need to act on callbacks to deal with global status is of course not that clean. You can however get what you are after as well as just about any other level of detail.

I don’t think it is particularly hard, but you should make it available as a PR on plunit :slight_smile: Hacking it from outside will get tricky. You probably want to be able to set this both globally, at the unit level and at the test level.

#5

As maybe an example of the sort of trick Jan is referring to, using the message hook to intercept errors and do things with it, here’s an example from lsp_server of using user:message_hook/3 to get the errors messages & programmatically manipulate them.

#6

Dear Jan, dear James,

thank you for your replies.

Let me say a couple of things. I was unaware of the “message hooks” facility. I am not a power user of SWI. I do appreciate its flexibility, alas, I believe it has shortcomings WRT my simpler and more direct solution to my (and I believe others’) problem.

First of all, it is not mentioned in the PlUnit documentation. Such a facility should be pointed to if it were to be used for such extensions. However, it seems a rather convoluted way to get information out of a library.

Second, the solution based on message hooks would not change (again, AFAIU) the semantics of the run_tests predicate which would fail if some tests did.

My solution is more direct and adds functionality in, IMHO, clearer way.

Having said so, now I know one more thing.

All the best

Marco

1 Like
#7

You show an interesting way to modify the functionality of an existing module by creating a new module that inherits from the original. I’m not so sure this is a good idea though as you are in the end relying on internal interfaces of the original and my understanding of modules is that it provides a public interface and anyone should be able to change anything inside a module without breaking other code as long as the interface is respected.

Now the trick using message_hook/3 is in the end nothing else than the common trick of hooking to allow users to extend Prolog libraries. Normally you do that using multifile predicates (which are also part of the public interface). Using message hooks is a bit of a grey area. Most libraries that use the message system do not expect anyone really hooking into them and might change the message terms without notification. Strictly speaking this is probably wrong. One of the intends of the message system has always been to create multi-lingual applications. That breaks if the developers of a library change the message terms. In any case, the message terms of plunit are intercepted by various packages to make the library emit feedback that conforms to some standard (a case of multi linguality :slight_smile: ) and these terms will thus not be changed lightly.

The clean solution is probably to give plunit a double interface, one that aims at use from a program and one that aims at interactive usage that is layered on top of the first. The program API would need to be quite extensive though, so possibly the message based approach is a practical compromise.

But, after all this is a free world :slight_smile:

#8

Hi Jan

Thanks for your message.
I used the module system to avoid messing up the internals on PlUnit. Nothing I would advocate.

What I would lobby for is to take what I wrote, clean it up (or tell me how) and include it (and document it) in PlUnit pretty much as is.

All the best

Marco