214 lines
8.2 KiB
Markdown
214 lines
8.2 KiB
Markdown
|
<a id="top"></a>
|
||
|
|
# Reporters
|
||
|
|
|
||
|
|
Reporters are a customization point for most of Catch2's output, e.g.
|
||
|
|
formatting and writing out [assertions (whether passing or failing),
|
||
|
|
sections, test cases, benchmarks, and so on](reporter-events.md#top).
|
||
|
|
|
||
|
|
Catch2 comes with a bunch of reporters by default (currently 9), and
|
||
|
|
you can also write your own reporter. Because multiple reporters can
|
||
|
|
be active at the same time, your own reporters do not even have to handle
|
||
|
|
all reporter event, just the ones you are interested in, e.g. benchmarks.
|
||
|
|
|
||
|
|
|
||
|
|
## Using different reporters
|
||
|
|
|
||
|
|
You can see which reporters are available by running the test binary
|
||
|
|
with `--list-reporters`. You can then pick one of them with the [`-r`,
|
||
|
|
`--reporter` option](command-line.md#choosing-a-reporter-to-use), followed
|
||
|
|
by the name of the desired reporter, like so:
|
||
|
|
|
||
|
|
```
|
||
|
|
--reporter xml
|
||
|
|
```
|
||
|
|
|
||
|
|
You can also select multiple reporters to be used at the same time.
|
||
|
|
In that case you should read the [section on using multiple
|
||
|
|
reporters](#multiple-reporters) to avoid any surprises from doing so.
|
||
|
|
|
||
|
|
|
||
|
|
<a id="multiple-reporters"></a>
|
||
|
|
## Using multiple reporters
|
||
|
|
|
||
|
|
> Support for having multiple parallel reporters was [introduced](https://github.com/catchorg/Catch2/pull/2183) in Catch2 3.0.1
|
||
|
|
|
||
|
|
Catch2 supports using multiple reporters at the same time while having
|
||
|
|
them write into different destinations. The two main uses of this are
|
||
|
|
|
||
|
|
* having both human-friendly and machine-parseable (e.g. in JUnit format)
|
||
|
|
output from one run of binary
|
||
|
|
* having "partial" reporters that are highly specialized, e.g. having one
|
||
|
|
reporter that writes out benchmark results as markdown tables and does
|
||
|
|
nothing else, while also having standard testing output separately
|
||
|
|
|
||
|
|
Specifying multiple reporter looks like this:
|
||
|
|
```
|
||
|
|
--reporter JUnit::out=result-junit.xml --reporter console::out=-::colour-mode=ansi
|
||
|
|
```
|
||
|
|
|
||
|
|
This tells Catch2 to use two reporters, `JUnit` reporter that writes
|
||
|
|
its machine-readable XML output to file `result-junit.xml`, and the
|
||
|
|
`console` reporter that writes its user-friendly output to stdout and
|
||
|
|
uses ANSI colour codes for colouring the output.
|
||
|
|
|
||
|
|
Using multiple reporters (or one reporter and one-or-more [event
|
||
|
|
listeners](event-listeners.md#top)) can have surprisingly complex semantics
|
||
|
|
when using customization points provided to reporters by Catch2, namely
|
||
|
|
capturing stdout/stderr from test cases.
|
||
|
|
|
||
|
|
As long as at least one reporter (or listener) asks Catch2 to capture
|
||
|
|
stdout/stderr, captured stdout and stderr will be available to all
|
||
|
|
reporters and listeners.
|
||
|
|
|
||
|
|
Because this might be surprising to the users, if at least one active
|
||
|
|
_reporter_ is non-capturing, then Catch2 tries to roughly emulate
|
||
|
|
non-capturing behaviour by printing out the captured stdout/stderr
|
||
|
|
just before `testCasePartialEnded` event is sent out to the active
|
||
|
|
reporters and listeners. This means that stdout/stderr is no longer
|
||
|
|
printed out from tests as it is being written, but instead it is written
|
||
|
|
out in batch after each runthrough of a test case is finished.
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
## Writing your own reporter
|
||
|
|
|
||
|
|
You can also write your own custom reporter and tell Catch2 to use it.
|
||
|
|
When writing your reporter, you have two options:
|
||
|
|
|
||
|
|
* Derive from `Catch::ReporterBase`. When doing this, you will have
|
||
|
|
to provide handling for all [reporter events](reporter-events.md#top).
|
||
|
|
* Derive from one of the provided [utility reporter bases in
|
||
|
|
Catch2](#utility-reporter-bases).
|
||
|
|
|
||
|
|
Generally we recommend doing the latter, as it is less work.
|
||
|
|
|
||
|
|
Apart from overriding handling of the individual reporter events, reporters
|
||
|
|
have access to some extra customization points, described below.
|
||
|
|
|
||
|
|
|
||
|
|
### Utility reporter bases
|
||
|
|
|
||
|
|
Catch2 currently provides two utility reporter bases:
|
||
|
|
|
||
|
|
* `Catch::StreamingReporterBase`
|
||
|
|
* `Catch::CumulativeReporterBase`
|
||
|
|
|
||
|
|
`StreamingReporterBase` is useful for reporters that can format and write
|
||
|
|
out the events as they come in. It provides (usually empty) implementation
|
||
|
|
for all reporter events, and if you let it handle the relevant events,
|
||
|
|
it also handles storing information about active test run and test case.
|
||
|
|
|
||
|
|
`CumulativeReporterBase` is a base for reporters that need to see the whole
|
||
|
|
test run, before they can start writing the output, such as the JUnit
|
||
|
|
and SonarQube reporters. This post-facto approach requires the assertions
|
||
|
|
to be stringified when it is finished, so that the assertion can be written
|
||
|
|
out later. Because the stringification can be expensive, and not all
|
||
|
|
cumulative reporters need the assertions, this base provides customization
|
||
|
|
point to change whether the assertions are saved or not, separate for
|
||
|
|
passing and failing assertions.
|
||
|
|
|
||
|
|
|
||
|
|
_Generally we recommend that if you override a member function from either
|
||
|
|
of the bases, you call into the base's implementation first. This is not
|
||
|
|
necessarily in all cases, but it is safer and easier._
|
||
|
|
|
||
|
|
|
||
|
|
Writing your own reporter then looks like this:
|
||
|
|
|
||
|
|
```cpp
|
||
|
|
#include <catch2/reporters/catch_reporter_streaming_base.hpp>
|
||
|
|
#include <catch2/catch_test_case_info.hpp>
|
||
|
|
#include <catch2/reporters/catch_reporter_registrars.hpp>
|
||
|
|
|
||
|
|
#include <iostream>
|
||
|
|
|
||
|
|
class PartialReporter : public Catch::StreamingReporterBase {
|
||
|
|
public:
|
||
|
|
using StreamingReporterBase::StreamingReporterBase;
|
||
|
|
|
||
|
|
static std::string getDescription() {
|
||
|
|
return "Reporter for testing TestCasePartialStarting/Ended events";
|
||
|
|
}
|
||
|
|
|
||
|
|
void testCasePartialStarting(Catch::TestCaseInfo const& testInfo,
|
||
|
|
uint64_t partNumber) override {
|
||
|
|
std::cout << "TestCaseStartingPartial: " << testInfo.name << '#' << partNumber << '\n';
|
||
|
|
}
|
||
|
|
|
||
|
|
void testCasePartialEnded(Catch::TestCaseStats const& testCaseStats,
|
||
|
|
uint64_t partNumber) override {
|
||
|
|
std::cout << "TestCasePartialEnded: " << testCaseStats.testInfo->name << '#' << partNumber << '\n';
|
||
|
|
}
|
||
|
|
};
|
||
|
|
|
||
|
|
|
||
|
|
CATCH_REGISTER_REPORTER("partial", PartialReporter)
|
||
|
|
```
|
||
|
|
|
||
|
|
This create a simple reporter that responds to `testCasePartial*` events,
|
||
|
|
and calls itself "partial" reporter, so it can be invoked with
|
||
|
|
`--reporter partial` command line flag.
|
||
|
|
|
||
|
|
|
||
|
|
### `ReporterPreferences`
|
||
|
|
|
||
|
|
Each reporter instance contains instance of `ReporterPreferences`, a type
|
||
|
|
that holds flags for the behaviour of Catch2 when this reporter run.
|
||
|
|
Currently there are two customization options:
|
||
|
|
|
||
|
|
* `shouldRedirectStdOut` - whether the reporter wants to handle
|
||
|
|
writes to stdout/stderr from user code, or not. This is useful for
|
||
|
|
reporters that output machine-parseable output, e.g. the JUnit
|
||
|
|
reporter, or the XML reporter.
|
||
|
|
* `shouldReportAllAssertions` - whether the reporter wants to handle
|
||
|
|
`assertionEnded` events for passing assertions as well as failing
|
||
|
|
assertions. Usually reporters do not report successful assertions
|
||
|
|
and don't need them for their output, but sometimes the desired output
|
||
|
|
format includes passing assertions even without the `-s` flag.
|
||
|
|
|
||
|
|
|
||
|
|
### Per-reporter configuration
|
||
|
|
|
||
|
|
> Per-reporter configuration was introduced in Catch2 3.0.1
|
||
|
|
|
||
|
|
Catch2 supports some configuration to happen per reporter. The configuration
|
||
|
|
options fall into one of two categories:
|
||
|
|
|
||
|
|
* Catch2-recognized options
|
||
|
|
* Reporter-specific options
|
||
|
|
|
||
|
|
The former is a small set of universal options that Catch2 handles for
|
||
|
|
the reporters, e.g. output file or console colour mode. The latter are
|
||
|
|
options that the reporters have to handle themselves, but the keys and
|
||
|
|
values can be arbitrary strings, as long as they don't contain `::`. This
|
||
|
|
allows writing reporters that can be significantly customized at runtime.
|
||
|
|
|
||
|
|
Reporter-specific options always have to be prefixed with "X" (large
|
||
|
|
letter X).
|
||
|
|
|
||
|
|
|
||
|
|
### Other expected functionality of a reporter
|
||
|
|
|
||
|
|
When writing a custom reporter, there are few more things that you should
|
||
|
|
keep in mind. These are not important for correctness, but they are
|
||
|
|
important for the reporter to work _nicely_.
|
||
|
|
|
||
|
|
* Catch2 provides a simple verbosity option for users. There are three
|
||
|
|
verbosity levels, "quiet", "normal", and "high", and if it makes sense
|
||
|
|
for reporter's output format, it should respond to these by changing
|
||
|
|
what, and how much, it writes out.
|
||
|
|
|
||
|
|
* Catch2 operates with an rng-seed. Knowing what seed a test run had
|
||
|
|
is important if you want to replicate it, so your reporter should
|
||
|
|
report the rng-seed, if at all possible given the target output format.
|
||
|
|
|
||
|
|
* Catch2 also operates with test filters, or test specs. If a filter
|
||
|
|
is present, you should also report the filter, if at all possible given
|
||
|
|
the target output format.
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
[Home](Readme.md#top)
|