Commits: 1
Improve the report format, esp. condensed output
Now multiple skipped steps are also condensed. There is a message explaining what it means.
If there is only one step to display, it will not be condensed (it would look silly and bring almost no saving in output length).
In the spec I elaborate on the reasons for condensed output and few other aspects.
index 920caa7..a22c28e 100644
--- a/spec/basic-usage.md
+++ b/spec/basic-usage.md
@@ -141,73 +141,83 @@ Whe a directory is given as the last argument, load all documents inside (recurs
= * The output will contain `arithmetic scenario` block
=
= ```text
- ✓ Arithmetic
- tagged: math
- source: samples/basic.md:22-34
+ ✓ Arithmetic
+ tagged: math
+ source: samples/basic.md:22-34
=
- ⊞ ⊞ ⊞
+ ⊞ ⊞ ⊞
+ ⬑ all 3 steps successful.
= ```
=
- Notice it's similar to the output of `tbb show`, but now contains unicode symbols to indicate the results of each step. Each completed scenario has a check mark `✓`. The number of steps is indicated by the `⊞` sigils.
+ Notice it's similar to the output of `tbb show`, but now contains unicode symbols to indicate the results of each step. Each completed scenario has a check mark `✓`. The number of steps is indicated by the `⊞` sigils and a message on the following line. This is called "condensed output".
+
+ TODO: Create a separate "condensed output" suite that focuses on this aspect.
=
= * The output will contain `text scenario` block
=
= ```text
- x Text
- tagged: strings work-in-progress
- source: samples/basic.md:35-91
+ x Text
+ tagged: strings work-in-progress
+ source: samples/basic.md:35-91
=
=
- ⊞ The word blocks has 6 characters
- arguments: ["blocks", "6"]
- source: samples/basic.md:45-45
+ ⊞ The word blocks has 6 characters
+ arguments: ["blocks", "6"]
+ source: samples/basic.md:45-45
=
- ⊞ There are 3 properties in the following JSON
- arguments: ["3"], code blocks: 1
- source: samples/basic.md:46-57
+ ⊞ There are 3 properties in the following JSON
+ arguments: ["3"], code blocks: 1
+ source: samples/basic.md:46-57
=
- ⊞ There are 3 rs in the word strawberry
- arguments: ["3", "r", "strawberry"]
- source: samples/basic.md:58-58
+ ⊞ There are 3 rs in the word strawberry
+ arguments: ["3", "r", "strawberry"]
+ source: samples/basic.md:58-58
=
- ⊞ The following table maps words to their lengths
- code blocks: 1, tables: 1
- source: samples/basic.md:59-84
+ ⊞ The following table maps words to their lengths
+ code blocks: 1, tables: 1
+ source: samples/basic.md:59-84
=
- ⊞ The reverse of abc is cba
- arguments: ["abc", "cba"]
- source: samples/basic.md:85-85
+ ⊞ The reverse of abc is cba
+ arguments: ["abc", "cba"]
+ source: samples/basic.md:85-85
=
- ⊠ The reverse of CIA is KGB
- arguments: ["CIA", "KGB"]
- source: samples/basic.md:86-89
+ ⊠ The reverse of CIA is KGB
+ arguments: ["CIA", "KGB"]
+ source: samples/basic.md:86-89
=
- 'KGB' != 'AIC'
+ 'KGB' != 'AIC'
=
- - KGB
- + AIC
+ - KGB
+ + AIC
=
- □ There are 2 os in the word boost
- arguments: ["2", "o", "boost"]
- source: samples/basic.md:90-91
+ □ There are 2 os in the word boost
+ arguments: ["2", "o", "boost"]
+ source: samples/basic.md:90-91
= ```
=
= If some steps fail, the output will be expanded. Successful steps have a squared plus `⊞` sigil . The failing step is markd with a squared times symbol `⊠`. Once a step fails, the subsequent steps in a scenario are not exercised. In the report they are marked with a white square symbol `□`.
+
+ To aid in debugging, each step is now expanded, showing it's source position and arguments.
=
= * The output will contain `geometry scenario` block
=
= ```text
- ? Geometry
- tagged: math
- source: samples/basic.md:92-99
+ ? Geometry
+ tagged: math
+ source: samples/basic.md:92-99
=
- There are no steps to execute in this scenario.
+ There are no steps to execute in this scenario.
= ```
+
+ Scenarios without any steps defined (only prose) are marked with the `?` sigil and a note. It's ok to have scenarios like that - often we start with a general thought about a feature, and we want to write it down before specifying precise steps to verify it. But eventually it should be covered. This output should remind us to do so.
=
= * The standard error will contain `\[.+ ERROR +tbb\] Step failed: Basic BDD suite ❯ Text ❯ The reverse of CIA is KGB`
=
+ At the bottom there is a list of errors. Because the reports tend to be long, it's easy to miss a failing step. This should help.
+
= * The exit code should be `1`
=
+
=## Running without a subcommand
=
=Running `tbb` without a subcommand will print the help message (to stderr) and exit with code 2.index 2696fb8..00f409e 100644
--- a/spec/failing-interpreters.md
+++ b/spec/failing-interpreters.md
@@ -6,17 +6,16 @@ interpreter: python -m spec.self-check
=
=Sometimes the failure is not in any particular step, but in a whole scenario, e.g. when an interpreter misbehaves. This suite describes what should happen when the interpreter process misbehaves in various ways.
=
-
=## Some won't start
=
=If some interpreters' commands are invalid (e.g. refers to a non-existing program), but others run fine, then `tbb` should report it as an error and exit with error status code.
=
= * Run the program with `evaluate samples/some-invalid/` command line arguments
-
+
= There are two documents in this directory: `valid.md` and `invalid.md`.
-
+
= * The output will contain `the valid suite header` block
-
+
= ```text
= A little suite that could
= tagged: basic passing
@@ -24,8 +23,23 @@ If some interpreters' commands are invalid (e.g. refers to a non-existing progra
= source: samples/some-invalid/valid.md
= ```
=
+ Some suites failing should not prevent others from being evaluated.
+
+ * The output will contain `the successful scenario` block
+
+ ``` text
+ ✓ Easy scenario
+ source: samples/some-invalid/valid.md:10-14
+
+ ⊞ Add 2 and 2 to get 4
+ arguments: ["2", "2", "4"]
+ source: samples/some-invalid/valid.md:12-14
+ ```
+
+ Notice that the only step of this scenario is not condensed. Condensing it wouldn't make much sense, since it's only a single step.
+
= * The output will contain `the invalid suite header` block
-
+
= ```text
= Suite 1 from the invalid document
= tagged: not-implemented seriously-underbaked
@@ -34,25 +48,32 @@ If some interpreters' commands are invalid (e.g. refers to a non-existing progra
= ```
=
= * The output will contain `the failing scenario` block
-
+
= ```text
- x Hopeless scenario
- tagged: even more tags very-important work-in-progress
- source: samples/some-invalid/invalid.md:17-29
+ x Hopeless scenario
+ tagged: even more tags very-important work-in-progress
+ source: samples/some-invalid/invalid.md:17-29
+
+ can't read from the interpreter process
+
+ * running scenario: Hopeless scenario
+ * awaiting for the interpreter
+ * can't read from the interpreter process
+
+ □ □
+ ⬑ remaining 2 steps were not evaluated.
= ```
-
- Notice the failing sigil in front.
+
+ Notice the failing sigil in front, a reason for failure and that multiple skipped steps are presented in a condensed format. Compare it with the single skipped step in the ./basic-usage.md document. It is not condensed, because there is only one. Here there are multiple, and since they would be executed only after the failing one, they could not have contributed to the failiure. So showing their details wouldn't bring much value.
=
= * The standard error will contain `\[.+ ERROR +tbb\] Scenario failed: Suite 1 from the invalid document ❯ Hopeless scenario`
=
= * The exit code should be `1`
=
-
=## Multiple scenarios failure
=
=When several different scenarios fail, each one should be mentioned in the summary.
=
-
=## The interpreter won't indicate it's ready
=
=What if the process starts, but never sends any messages to output, specifically the `Ready` message? There should be a timeout period (with a default value of 10s, but configurable per scenario). After the timeout, the interpreter process should be terminated and scenario marked as failed. The output should clearly indicate what the problem was.index 357a1ae..93feb2e 100644
--- a/spec/filtering.md
+++ b/spec/filtering.md
@@ -42,8 +42,9 @@ With the `--only scenario:<tag>` only scenarios that have the given tag should b
= * The output will contain `the expected scenario header` block
=
= ```text
- ✓ Interesting scenario
- tagged: interesting
+ ✓ Interesting scenario
+ tagged: interesting
+ source: samples/some-invalid/valid.md:15-22
= ```
=
= * The output will contain `Suite 1 from the invalid document`index 30dcf1a..4cd1f31 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -296,17 +296,22 @@ impl Display for EvaluationReport<'_> {
= ScenarioStatus::Pending => "?".to_string().bold().dimmed(),
= ScenarioStatus::FailedToRun { .. } => "x".to_string().bold().red(),
= };
- writeln!(f, "\n{sigil} {}", scenario.to_string().indent(2).trim())?;
+ writeln!(f, "\n {sigil} {}", scenario.to_string().indent(4).trim())?;
= writeln!(f, "")?;
=
= if let ScenarioStatus::FailedToRun { error } = status {
= writeln!(
= f,
= "{cause}\n",
- cause = error.root_cause().to_string().red().indent(2)
+ cause = error.root_cause().to_string().indent(4).red().bold()
= )?;
= for cause in error.chain() {
- writeln!(f, " * {}", cause.to_string().indent(2))?
+ writeln!(
+ f,
+ "{indentation}* {}",
+ cause.to_string(),
+ indentation = "".indent(6)
+ )?
= }
= writeln!(f, "\n")?;
= }
@@ -316,42 +321,57 @@ impl Display for EvaluationReport<'_> {
= f,
= "{}",
= "There are no steps to execute in this scenario."
- .indent(2)
+ .indent(4)
= .yellow()
+ .bold()
= )?
- } else if all_ok {
- // Indentation for condensed output
- write!(f, " ")?;
- };
-
- for StepReport { step, status } in steps.iter() {
- let sigil = match status {
- StepStatus::Ok => "⊞".to_string().bold().green(),
- StepStatus::Failed { .. } => "⊠".to_string().bold().red(),
- StepStatus::NotEvaluated => "□".to_string().bold(),
- };
-
- // Condense output if everything is ok
- if all_ok {
- write!(f, "{sigil} ")?;
- continue;
- }
-
- writeln!(
- f,
- "\n{indentation}{sigil} {}",
- step.to_string().indent(4).trim(),
- indentation = "".indent(2),
- )?;
+ } else if all_ok && steps.len() > 1 && f.alternate().not() {
+ // Condense the output when all steps are successful
+ let sigils = "⊞ ".repeat(steps.len()).trim().indent(4).green();
+ let message = format!("⬑ all {count} steps successful.", count = steps.len())
+ .indent(4)
+ .dimmed()
+ .bold();
+ writeln!(f, "{sigils}")?;
+ writeln!(f, "{message}")?;
+ } else {
+ for (index, StepReport { step, status }) in steps.iter().enumerate() {
+ let sigil = match status {
+ StepStatus::Ok => "⊞".to_string().bold().green(),
+ StepStatus::Failed { .. } => "⊠".to_string().bold().red(),
+ StepStatus::NotEvaluated => "□".to_string().bold(),
+ };
+
+ let remaining_count = steps.len() - index;
+ if *status == StepStatus::NotEvaluated && remaining_count > 1 {
+ // Condense the output when all steps are successful
+ let sigils = "□ ".repeat(remaining_count).trim().indent(4);
+ let message =
+ format!("⬑ remaining {remaining_count} steps were not evaluated.")
+ .indent(4)
+ .dimmed()
+ .bold();
+ writeln!(f, "{sigils}")?;
+ writeln!(f, "{message}")?;
+ break;
+ }
=
- if let StepStatus::Failed { reason, hint } = status {
- writeln!(f, "\n{}\n", reason.indent(4).red())?;
- if let Some(hint) = hint {
- writeln!(f, "{}", hint.as_str().indent(4))?;
+ writeln!(
+ f,
+ "\n{indentation}{sigil} {}",
+ step.to_string().indent(6).trim(),
+ indentation = "".indent(4),
+ )?;
+
+ if let StepStatus::Failed { reason, hint } = status {
+ writeln!(f, "\n{}\n", reason.indent(6).red())?;
+ if let Some(hint) = hint {
+ writeln!(f, "{}", hint.as_str().indent(6))?;
+ }
= }
= }
- }
- writeln!(f, "")?;
+ writeln!(f, "")?;
+ };
= }
= writeln!(f, "")?;
= }