Week 49 of 2025
Development log of Tad Better Behavior
12 items
- Split the tagging and failing scenarios from the basic-usage suite
- Update the readme, stub the source info spec
- Write steps to verify source information passing
- Implement the source information passing
- Spec: Rename list sub-command to show
- Rename the list command to show, change display format
- Refactor spec display: delegate to suite, scenario, step
- Make the evaluation report format similar to show
- Condense output when all steps are ok
- Bump minor version to 0.9
- Update the roadmap
- Improve the report format, esp. condensed output
Split the tagging and failing scenarios from the basic-usage suite
On by
index 411d5d7..fb1caf7 100644
--- a/spec/basic-usage.md
+++ b/spec/basic-usage.md
@@ -190,22 +190,3 @@ Notice it's similar to the output of `tbb list`, but now contains unicode symbol
=## Running without a subcommand
=
=Running `tbb` without a subcommand will print the help message (to stderr) and exit with code 2.
-
-
-## A whole scenario failure
-
-Sometimes the failure is not in any particular step, but in a whole scenario, e.g. when an interpreter misbehaves. The status code and the summary should reflect it.
-
-
-## Multiple scenarios failure
-
-When several different scenarios fail, each one should be mentioned in the summary.
-
-
-## Tags are always in alphabetical order
-
-Consider moving it to a new Tags suite.
-
-## Tagging is idempotent.
-
-Tags are a set. Specifying the same tag more than once doesn't have any effect.index 274adcc..c0c4dec 100644
--- a/spec/failing-interpreters.md
+++ b/spec/failing-interpreters.md
@@ -4,7 +4,7 @@ interpreter: python -m spec.self-check
=
=# Failing interpreters
=
-This suite describes what should happen when the interpreter process misbehaves in various ways.
+Sometimes the failure is not in any particular step, but in a whole scenario, e.g. when an interpreter misbehaves. This suite describes what should happen when the interpreter process misbehaves in various ways.
=
=
=## Some won't start
@@ -41,3 +41,21 @@ If some interpreters' commands are invalid (e.g. refers to a non-existing progra
= * The standard error will contain `\[.+ ERROR +tbb\] Scenario failed: Suite 1 from the invalid document ❯ Hopeless scenario`
=
= * The exit code should be `1`
+
+
+## Multiple scenarios failure
+
+When several different scenarios fail, each one should be mentioned in the summary.
+
+
+## The interpreter won't indicate it's ready
+
+What if the process starts, but never sends any messages to output, specifically the `Ready` message? There should be a timeout period (with a default value of 10s, but configurable per scenario). After the timeout, the interpreter process should be terminated and scenario marked as failed. The output should clearly indicate what the problem was.
+
+This is currently **not implemented**.
+
+## The step is never resolved
+
+Similarly, if a single step is never reported as success of failure, the interpreter process should be terminated, and **the step** marked as failed.
+
+This is currently **not implemented**.new file mode 100644
index 0000000..efe375a
--- /dev/null
+++ b/spec/tagging.md
@@ -0,0 +1,39 @@
+---
+interpreter: "python -m spec.self-check"
+---
+
+# Tagging
+
+Suites and scenarios can be tagged. This is useful for [filtering](./filtering.md "Filtering spec").
+
+
+## Setting tags for a suite via a front-matter
+
+A front-matter of a spec document can have `tags` property with an array of tags. They will be applied to suites originating from this document.
+
+
+## Setting tags for a suite via `tbb` code blocks
+
+With a code block like this:
+
+ ``` yaml tbb
+ tags: [ "tag-1", "tag-2" ]
+ ```
+
+A code block must be placed before any scenario heading (h2).
+
+When multiple blocks are present, the tags should be merged.
+
+## Setting tags for a scenario via `tbb` code blocks
+
+Similar to a suite, scenario tags can be applied via (and only via) a `tbb` code block. Those blocks are placed after the given scenario heading. Other than that, same rules apply.
+
+
+## Tags are always printed in alphabetical order
+
+When printing a spec or a report, the tags should always be shown in alphabetical order.
+
+
+## Tagging is idempotent.
+
+Tags are a set. Specifying the same tag more than once doesn't have any effect.Update the readme, stub the source info spec
On by
index 0db1bf3..8d5f4a0 100644
--- a/README.md
+++ b/README.md
@@ -7,6 +7,7 @@ A BDD test runner inspired by Gauge, but better.
= * No magic
= * Flexibility
= * Cross-platform
+ * Batteries included, but replaceable
=
=
=## No magic
@@ -32,6 +33,11 @@ and even the original markdown fragment itself.
=Support for Linux, BSD, OS X and Web (WASM).
=
=
+## Batteries included, but replaceable
+
+The program comes with a versatile interpreter (not yet implemented) capable of web-automation via webdriver, an http client and command execution capabilities. It's also easy to implement your own interpreter. All it takes is reading JSON from `stdin` and writing JSON to `stdout`.
+
+
=## Vocabulary
=
=<!-- TODO: Reformat vocabulary as a definition list? -->new file mode 100644
index 0000000..19f98e2
--- /dev/null
+++ b/spec/source-information.md
@@ -0,0 +1,25 @@
+---
+interpreter: "python -m spec.self-check"
+---
+
+# Passing source information
+
+When executing a step, the interpreter will get it's source file path and lines range, like this:
+
+``` json
+{
+ "variant": "...",
+ "arguments": [],
+ "tables": [],
+ "source": {
+ "document_path": "./important-aspect.md",
+ "first_line": 23,
+ "last_line": 45
+ },
+ ...
+}
+```
+
+This is to fulfill the "batteries included, but replaceable" principle. With this information, an interpreter can extract any additional information from a step.
+
+Write steps to verify source information passing
On by
index 9b1af63..ce81686 100644
--- a/spec/self-check.py
+++ b/spec/self-check.py
@@ -179,4 +179,16 @@ def step_implementation_13(row: int, column: int, table_name: str, expected_valu
= table = state[table_name]
= tester.assertEqual(table[row][column], expected_value)
=
+@step("This step is defined between lines {0} and {1} in {2}")
+def step_implementation_14(first_line: int, last_line: int, document_path: str, **kwargs):
+ tester.assertEqual(
+ {
+ "first_line": first_line,
+ "last_line": last_line,
+ "document_path": document_path
+ },
+ kwargs["source"]
+ )
+
+
=tbb.ready()index 19f98e2..76b8358 100644
--- a/spec/source-information.md
+++ b/spec/source-information.md
@@ -22,4 +22,11 @@ When executing a step, the interpreter will get it's source file path and lines
=
=This is to fulfill the "batteries included, but replaceable" principle. With this information, an interpreter can extract any additional information from a step.
=
+## Source information will be passed to interpreter
=
+ * This step is defined between lines `27` and `30` in `spec/source-information.md`
+
+ Every step is passed with `source` property, conatining `first_line`, `last_line` and `document_path`
+
+ * This step is defined between lines `31` and `31` in `spec/source-information.md`
+ * This step is defined between lines `32` and `32` in `spec/source-information.md`Implement the source information passing
On by
index b37e0a9..2a1b8e0 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -77,16 +77,16 @@ fn list(input: PathBuf) -> Result<(), Error> {
=
= for path in glob(&pattern)? {
= let path = path.context(format!("resolving {pattern}"))?;
- let md = std::fs::read_to_string(&path)
+ let md = std::fs::read_to_string(&path.clone())
= .context(format!("reading a file at {}", path.display()))?;
- spec.load_document(&md)
+ spec.load_document(relative_path(&path), &md)
= .context(format!("loading a document from {}", path.display()))?;
= }
= } else if input.is_file() {
= log::debug!("The {} is a file. Reading...", input.display());
= let md = std::fs::read_to_string(&input)
= .context(format!("reading a file at {}", input.display()))?;
- spec.load_document(&md)
+ spec.load_document(relative_path(&input), &md)
= .context(format!("loading a document from {}", input.display()))?;
= } else {
= bail!("The {} is neither a file nor directory", input.display());
@@ -119,14 +119,14 @@ fn evaluate(
= let path = path.context(format!("resolving {pattern}"))?;
= let md = std::fs::read_to_string(&path)
= .context(format!("reading a file at {}", path.display()))?;
- spec.load_document(&md)
+ spec.load_document(relative_path(&path), &md)
= .context(format!("loading a document from {}", path.display()))?;
= }
= } else if input.is_file() {
= log::debug!("The {} is a file. Reading...", input.display());
= let md = std::fs::read_to_string(&input)
= .context(format!("reading a file at {}", input.display()))?;
- spec.load_document(&md)
+ spec.load_document(relative_path(&input), &md)
= .context(format!("loading a document from {}", input.display()))?;
= } else {
= bail!("The {} is neither a file nor directory", input.display());
@@ -204,3 +204,16 @@ fn evaluate(
= }
= }
=}
+
+fn relative_path(path: &PathBuf) -> PathBuf {
+ let document_path: PathBuf = std::env::current_dir()
+ .context("Getting current working directory")
+ .and_then(|cwd| {
+ log::info!("Stripping prefix {}", cwd.display());
+ path.strip_prefix(cwd)
+ .context("Resolving a relative path to a document")
+ })
+ .map(PathBuf::from)
+ .unwrap_or(PathBuf::from("unknown"));
+ document_path
+}index 885e143..4ec607b 100644
--- a/src/spec.rs
+++ b/src/spec.rs
@@ -6,6 +6,7 @@ use serde::{Deserialize, Serialize};
=use std::collections::BTreeSet;
=use std::fmt::Display;
=use std::ops::Not;
+use std::path::PathBuf;
=
=/// Spec is a collection of suites that together describe a system
=///
@@ -30,8 +31,9 @@ pub struct Spec {
=pub struct Suite {
= pub title: String,
= pub tags: BTreeSet<String>,
- pub interpreter: String,
+ pub interpreter: String, // TODO: Interpreter should be a property of a scenario?
= pub scenarios: Vec<Scenario>,
+ pub source: SourceInfromation,
=}
=
=/// Scenario is set of steps.
@@ -45,6 +47,7 @@ pub struct Scenario {
= pub title: String,
= pub tags: BTreeSet<String>,
= pub steps: Vec<Step>,
+ pub source: SourceInfromation,
=}
=
=#[derive(Debug, Serialize, Deserialize, Clone)]
@@ -58,6 +61,9 @@ pub struct Step {
= /// Values of the arguments
= pub arguments: Vec<String>,
=
+ /// Source information
+ pub source: SourceInfromation,
+
= /// List of code blocks
= pub code_blocks: Vec<CodeBlock>,
=
@@ -68,6 +74,13 @@ pub struct Step {
= pub lists: Vec<List>,
=}
=
+#[derive(Debug, Serialize, Deserialize, Clone)]
+pub struct SourceInfromation {
+ pub document_path: PathBuf,
+ pub first_line: usize,
+ pub last_line: usize,
+}
+
=#[derive(Debug, Serialize, Deserialize, Clone)]
=pub struct CodeBlock {
= value: String,
@@ -88,9 +101,10 @@ pub struct ListItem {
=
=impl Spec {
= /// Load suites from a markdown document
- pub fn load_document(&mut self, md: &str) -> anyhow::Result<()> {
+ pub fn load_document(&mut self, document_path: PathBuf, md: &str) -> anyhow::Result<()> {
= // TODO: Support loading multiple suits from a single document (demarcated by h1)
- let suite = Suite::from_markdown(md).context("loading a markdown document")?;
+ let suite =
+ Suite::from_markdown(document_path, md).context("loading a markdown document")?;
=
= self.suites.push(suite);
= Ok(())
@@ -258,7 +272,7 @@ pub struct Metadata {
=}
=
=impl Suite {
- pub fn from_markdown(md: &str) -> anyhow::Result<Self> {
+ pub fn from_markdown(document_path: PathBuf, md: &str) -> anyhow::Result<Self> {
= let mdast = markdown::to_mdast(
= md,
= &markdown::ParseOptions {
@@ -275,14 +289,6 @@ impl Suite {
= })?;
= log::debug!("Markdown parsed:\n\n{:#?}", mdast);
=
- Self::try_from(mdast).context("extracting a suite from a markdown document")
- }
-}
-
-impl TryFrom<markdown::mdast::Node> for Suite {
- type Error = anyhow::Error;
-
- fn try_from(mdast: markdown::mdast::Node) -> anyhow::Result<Self> {
= // Find the YAML front-matter and extract the interpreter field
= let children = mdast.children().context("the markdown document is empty")?;
= let first = children.first().context("the markdown document is empty")?;
@@ -304,8 +310,8 @@ impl TryFrom<markdown::mdast::Node> for Suite {
= serde_yaml::from_str(yaml).context("Failed to parse front-matter as YAML")?;
=
= // Find h1 and use it as title (
- // TODO: make sure there's only one and it's before any h2
- let title = children
+
+ let title_node = children
= .iter()
= .filter(|element| {
= if let markdown::mdast::Node::Heading(heading) = element
@@ -325,8 +331,22 @@ impl TryFrom<markdown::mdast::Node> for Suite {
= # Some important aspect of the spec
=
= This will be used as a title of a suite.
- "#})?
- .to_string();
+ "#})?;
+
+ let source = SourceInfromation {
+ document_path: document_path.clone(),
+ first_line: title_node
+ .position()
+ .context("Getting the source line on which the suite starts")?
+ .start
+ .line,
+ last_line: mdast
+ .position()
+ .context("Getting the last source line of the document")?
+ .end
+ .line,
+ };
+ let title = title_node.to_string();
=
= // Find any yaml meta code block after the h1 but before any h2
=
@@ -335,15 +355,32 @@ impl TryFrom<markdown::mdast::Node> for Suite {
= // Extract scenarios and steps
= // Split into sections, each starting at h2
= // Convert each section into a scenario
- let mut scenarios = Vec::new();
+ let mut scenarios: Vec<Scenario> = Vec::new();
= for node in children.iter() {
= match node {
= markdown::mdast::Node::Heading(heading) => {
= if heading.depth == 2 {
+ let line = node
+ .position()
+ .context("Getting the source line where the scenario starts")?
+ .start
+ .line;
+
+ // Mark the end of previous scenario, if any
+ if let Some(previous_scenario) = scenarios.last_mut() {
+ previous_scenario.source.last_line = line - 1;
+ }
+
= scenarios.push(Scenario {
= title: node.to_string(),
= tags: BTreeSet::default(),
= steps: [].into(),
+ source: SourceInfromation {
+ document_path: document_path.clone(),
+ first_line: line,
+ // Assume the scenario goes until the end of document, until proven otherwise
+ last_line: source.last_line,
+ },
= });
= }
= }
@@ -359,7 +396,7 @@ impl TryFrom<markdown::mdast::Node> for Suite {
=
= for item in items {
= // First child of a list item should always be a paragraph
- let step = Step::try_from(item)?;
+ let step = Step::from_markdown(document_path.clone(), item)?;
=
= scenario.steps.push(step);
= }
@@ -397,14 +434,32 @@ impl TryFrom<markdown::mdast::Node> for Suite {
= tags: suite_tags,
= scenarios,
= interpreter: frontmatter.interpreter,
+ source,
= })
= }
=}
=
-impl TryFrom<&markdown::mdast::ListItem> for Step {
- type Error = anyhow::Error;
+impl Step {
+ fn from_markdown(
+ document_path: PathBuf,
+ item: &markdown::mdast::ListItem,
+ ) -> Result<Self, anyhow::Error> {
+ let source = SourceInfromation {
+ document_path,
+ first_line: item
+ .clone()
+ .position
+ .context("Getting first line of the step list item")?
+ .start
+ .line,
+ last_line: item
+ .clone()
+ .position
+ .context("Getting last line of the step list item")?
+ .end
+ .line,
+ };
=
- fn try_from(item: &markdown::mdast::ListItem) -> Result<Self, Self::Error> {
= let headline = item
= .children
= .first()
@@ -463,6 +518,7 @@ impl TryFrom<&markdown::mdast::ListItem> for Step {
= description: headline.to_string(),
= variant: variant_headline.to_string(),
= arguments,
+ source,
= code_blocks,
= tables,
= lists,Spec: Rename list sub-command to show
On by
Change the output format.
index c117e8b..7762c58 100644
--- a/roadmap.md
+++ b/roadmap.md
@@ -15,6 +15,9 @@ We use this document mostly to keep track of our progress and a scratchpad for i
= - [ ] Cucumber
= - [ ] Playwright
=- [ ] Demo
+- [ ] Show and report output formats
+ - [ ] JSON
+ - [ ] Markdown
=- [x] Proof of concept
= - [x] Interpretter in a different language (Python)
= - [x] Report why steps fail
@@ -22,7 +25,7 @@ We use this document mostly to keep track of our progress and a scratchpad for i
= - [x] Proof of concept
= - [ ] Comprehensive
= - [x] Help
- - [x] List
+ - [x] Show
= - [x] Evaluate
= - [ ] Tagging
= - [x] Filteringindex fb1caf7..d93816d 100644
--- a/spec/basic-usage.md
+++ b/spec/basic-usage.md
@@ -24,7 +24,7 @@ interpreter: "python -m spec.self-check"
=
= ``` text
= Commands:
- list Print the suites, scenarios and steps of the specification
+ show Print the suites, scenarios and steps of the specification
= evaluate Evaluate the specification
= help Print this message or the help of the given subcommand(s)
= ```
@@ -38,7 +38,7 @@ interpreter: "python -m spec.self-check"
= * The exit code should be `0`
=
=
-## Listing a spec from the `./spec/` directory
+## Showing a spec from the `./spec/` directory
=
=This is the default behavior when a path is not given. We should prepare the samples to replicate this directory structure like this
=
@@ -56,19 +56,21 @@ samples/python/
=We would also need a step like `Change working directory to ...` to execute before running the program.
=
=
-## Listing a spec from a different directory
+## Showing a spec from a different directory
=
=Whe a directory is given as the last argument, load all documents inside (recursively).
=
=
-## Listing suites and scenarios from a single document
+## Showing suites and scenarios from a single document
=
- * Run the program with `list samples/basic.md` command line arguments
+ * Run the program with `show samples/basic.md` command line arguments
= * The output will contain `the suite` block
=
= ``` text
- Basic BDD suite (python -m samples.basic)
+ Basic BDD suite
= tagged: basic not-implemented sample tutorial
+ interpreter: python -m samples.basic
+ source: samples/basic.md
= ```
=
= * The output will contain `the Aritmetic scenario` block
@@ -76,10 +78,17 @@ Whe a directory is given as the last argument, load all documents inside (recurs
= ``` text
= * Arithmetic
= tagged: math
-
- 00. Add 7 and 5 to get 12 ["7", "5", "12"]
- 01. Divide 10 by 4 to get 2.5 ["10", "4", "2.5"]
- 02. Subtract 7 from 5 to get -2 ["7", "5", "-2"]
+ source: samples/basic.md:22-34
+
+ 00. Add 7 and 5 to get 12
+ arguments: ["7", "5", "12"]
+ source: samples/basic.md:30-30
+ 01. Divide 10 by 4 to get 2.5
+ arguments: ["10", "4", "2.5"]
+ source: samples/basic.md:31-31
+ 02. Subtract 7 from 5 to get -2
+ arguments: ["7", "5", "-2"]
+ source: samples/basic.md:32-34
= ```
=
= * The output will contain `the Text scenario` block
@@ -87,14 +96,29 @@ Whe a directory is given as the last argument, load all documents inside (recurs
= ``` text
= * Text
= tagged: strings work-in-progress
-
- 00. The word blocks has 6 characters ["blocks", "6"]
- 01. There are 3 properties in the following JSON ["3"]
- 02. There are 3 rs in the word strawberry ["3", "r", "strawberry"]
- 03. The following table maps words to their lengths []
- 04. The reverse of abc is cba ["abc", "cba"]
- 05. The reverse of CIA is KGB ["CIA", "KGB"]
- 06. There are 2 os in the word boost ["2", "o", "boost"]
+ source: samples/basic.md:35-91
+
+ 00. The word blocks has 6 characters
+ arguments: ["blocks", "6"]
+ source: samples/basic.md:45-45
+ 01. There are 3 properties in the following JSON
+ arguments: ["3"], code blocks: 1
+ source: samples/basic.md:46-57
+ 02. There are 3 rs in the word strawberry
+ arguments: ["3", "r", "strawberry"]
+ source: samples/basic.md:58-58
+ 03. The following table maps words to their lengths
+ code blocks: 1, tables: 1
+ source: samples/basic.md:59-84
+ 04. The reverse of abc is cba
+ arguments: ["abc", "cba"]
+ source: samples/basic.md:85-85
+ 05. The reverse of CIA is KGB
+ arguments: ["CIA", "KGB"]
+ source: samples/basic.md:86-89
+ 06. There are 2 os in the word boost
+ arguments: ["2", "o", "boost"]
+ source: samples/basic.md:90-91
= ```
=
= * The exit code should be `0`
@@ -130,7 +154,7 @@ A complete sample output is like this:
= □ There are 2 os in the word boost ["2", "o", "boost"]
=```
=
-Notice it's similar to the output of `tbb list`, but now contains unicode symbols to indicate the results of each step. Each completed scenario has a check mark `✓`. Successful steps have a squared plus `⊞` . The failing step is markd with a squared times symbol `⊠`. Once a step fails, the subsequent steps in a scenario are not exercised. In the report they are marked with a white square symbol `□`.
+Notice it's similar to the output of `tbb show`, but now contains unicode symbols to indicate the results of each step. Each completed scenario has a check mark `✓`. Successful steps have a squared plus `⊞` . The failing step is markd with a squared times symbol `⊠`. Once a step fails, the subsequent steps in a scenario are not exercised. In the report they are marked with a white square symbol `□`.
=
=
= * Run the program with `evaluate samples/basic.md` command line argumentsRename the list command to show, change display format
On by
As specified.
index 2a1b8e0..c398fe0 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -25,7 +25,7 @@ struct Cli {
=#[derive(Subcommand)]
=enum Command {
= /// Print the suites, scenarios and steps of the specification
- List {
+ Show {
= /// A directory or a markdown file with the spec to list
= #[arg(value_name = "SPEC PATH", default_value = "./spec/")]
= input: PathBuf,
@@ -53,7 +53,7 @@ fn main() -> Result<(), Error> {
= env_logger::init_from_env(log_env);
=
= match cli.command {
- Command::List { input } => list(input),
+ Command::Show { input } => show(input),
= Command::Evaluate {
= input,
= only,
@@ -62,7 +62,7 @@ fn main() -> Result<(), Error> {
= }
=}
=
-fn list(input: PathBuf) -> Result<(), Error> {
+fn show(input: PathBuf) -> Result<(), Error> {
= log::debug!("Reading the specification from {}", input.display());
=
= let input = input.canonicalize()?;index 4ec607b..84a262e 100644
--- a/src/spec.rs
+++ b/src/spec.rs
@@ -3,7 +3,7 @@ use anyhow::{Context, anyhow, bail};
=use colored::Colorize;
=use indoc::indoc;
=use serde::{Deserialize, Serialize};
-use std::collections::BTreeSet;
+use std::collections::{BTreeMap, BTreeSet};
=use std::fmt::Display;
=use std::ops::Not;
=use std::path::PathBuf;
@@ -167,12 +167,7 @@ pub struct Table(Vec<Vec<String>>);
=impl Display for Spec {
= fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
= for suite in self.suites.iter() {
- writeln!(
- f,
- "\n{title} ({interpreter})",
- title = suite.title.bold().underline(),
- interpreter = suite.interpreter.dimmed()
- )?;
+ writeln!(f, "\n{title}", title = suite.title.bold().underline(),)?;
=
= if suite.tags.is_empty().not() {
= let tags = suite
@@ -184,6 +179,21 @@ impl Display for Spec {
= writeln!(f, "tagged: {tags}")?;
= }
=
+ let interpreter = format!(
+ "interpreter: {interpreter}",
+ interpreter = suite.interpreter
+ )
+ .dimmed();
+ writeln!(f, "{interpreter}")?;
+
+ // Source of the suite
+ let source = format!(
+ "source: {source}",
+ source = suite.source.document_path.display()
+ )
+ .dimmed();
+ writeln!(f, "{source}")?;
+
= for scenario in suite.scenarios.iter() {
= writeln!(
= f,
@@ -191,6 +201,7 @@ impl Display for Spec {
= title = scenario.title.bold(),
= indentation = "".indent(2)
= )?;
+
= if scenario.tags.is_empty().not() {
= let tags = scenario
= .tags
@@ -200,15 +211,60 @@ impl Display for Spec {
= .join(" ");
= writeln!(f, "{indentation}tagged: {tags}", indentation = "".indent(4))?;
= }
+
+ // Source of the scenario
+ let source = format!(
+ "source: {document_path}:{first_line}-{last_line}",
+ document_path = scenario.source.document_path.display(),
+ first_line = scenario.source.first_line,
+ last_line = scenario.source.last_line,
+ )
+ .indent(4)
+ .dimmed();
+ writeln!(f, "{source}")?;
= writeln!(f, "")?;
=
= for (index, step) in scenario.steps.iter().enumerate() {
= writeln!(
= f,
- " {index:02}. {description} {arguments}",
+ " {index:02}. {description}",
= description = step.description,
- arguments = format!("{:?}", step.arguments).dimmed()
= )?;
+
+ // Arguments, code blocks, lists and tables
+ let mut data_objects = BTreeMap::from([
+ ("code blocks", step.code_blocks.len()),
+ ("lists", step.lists.len()),
+ ("tables", step.tables.len()),
+ ])
+ .iter()
+ .filter_map(|(kind, count)| {
+ if *count == 0 {
+ None
+ } else {
+ Some(format!("{kind}: {count}"))
+ }
+ })
+ .collect::<Vec<String>>();
+
+ if step.arguments.is_empty().not() {
+ data_objects.insert(0, format!("arguments: {:?}", step.arguments));
+ };
+
+ if data_objects.is_empty().not() {
+ writeln!(f, "{}", data_objects.join(", ").indent(8).dimmed())?;
+ }
+
+ // Source of the step
+ let source = format!(
+ "source: {document_path}:{first_line}-{last_line}",
+ document_path = step.source.document_path.display(),
+ first_line = step.source.first_line,
+ last_line = step.source.last_line,
+ )
+ .indent(8)
+ .dimmed();
+ writeln!(f, "{source}")?;
= }
= }
= writeln!(f, "")?;Refactor spec display: delegate to suite, scenario, step
On by
I want to re-use this logic in report display.
index c398fe0..d674ae7 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -209,7 +209,6 @@ fn relative_path(path: &PathBuf) -> PathBuf {
= let document_path: PathBuf = std::env::current_dir()
= .context("Getting current working directory")
= .and_then(|cwd| {
- log::info!("Stripping prefix {}", cwd.display());
= path.strip_prefix(cwd)
= .context("Resolving a relative path to a document")
= })index 84a262e..955e225 100644
--- a/src/spec.rs
+++ b/src/spec.rs
@@ -167,104 +167,14 @@ pub struct Table(Vec<Vec<String>>);
=impl Display for Spec {
= fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
= for suite in self.suites.iter() {
- writeln!(f, "\n{title}", title = suite.title.bold().underline(),)?;
-
- if suite.tags.is_empty().not() {
- let tags = suite
- .tags
- .iter()
- .map(|tag| tag.underline().to_string())
- .collect::<Vec<String>>()
- .join(" ");
- writeln!(f, "tagged: {tags}")?;
- }
-
- let interpreter = format!(
- "interpreter: {interpreter}",
- interpreter = suite.interpreter
- )
- .dimmed();
- writeln!(f, "{interpreter}")?;
-
- // Source of the suite
- let source = format!(
- "source: {source}",
- source = suite.source.document_path.display()
- )
- .dimmed();
- writeln!(f, "{source}")?;
+ writeln!(f, "\n{suite}")?;
=
= for scenario in suite.scenarios.iter() {
- writeln!(
- f,
- "\n{indentation}* {title}",
- title = scenario.title.bold(),
- indentation = "".indent(2)
- )?;
-
- if scenario.tags.is_empty().not() {
- let tags = scenario
- .tags
- .iter()
- .map(|tag| tag.underline().to_string())
- .collect::<Vec<String>>()
- .join(" ");
- writeln!(f, "{indentation}tagged: {tags}", indentation = "".indent(4))?;
- }
-
- // Source of the scenario
- let source = format!(
- "source: {document_path}:{first_line}-{last_line}",
- document_path = scenario.source.document_path.display(),
- first_line = scenario.source.first_line,
- last_line = scenario.source.last_line,
- )
- .indent(4)
- .dimmed();
- writeln!(f, "{source}")?;
+ writeln!(f, "\n * {}", scenario.to_string().indent(4).trim())?;
= writeln!(f, "")?;
=
= for (index, step) in scenario.steps.iter().enumerate() {
- writeln!(
- f,
- " {index:02}. {description}",
- description = step.description,
- )?;
-
- // Arguments, code blocks, lists and tables
- let mut data_objects = BTreeMap::from([
- ("code blocks", step.code_blocks.len()),
- ("lists", step.lists.len()),
- ("tables", step.tables.len()),
- ])
- .iter()
- .filter_map(|(kind, count)| {
- if *count == 0 {
- None
- } else {
- Some(format!("{kind}: {count}"))
- }
- })
- .collect::<Vec<String>>();
-
- if step.arguments.is_empty().not() {
- data_objects.insert(0, format!("arguments: {:?}", step.arguments));
- };
-
- if data_objects.is_empty().not() {
- writeln!(f, "{}", data_objects.join(", ").indent(8).dimmed())?;
- }
-
- // Source of the step
- let source = format!(
- "source: {document_path}:{first_line}-{last_line}",
- document_path = step.source.document_path.display(),
- first_line = step.source.first_line,
- last_line = step.source.last_line,
- )
- .indent(8)
- .dimmed();
- writeln!(f, "{source}")?;
+ writeln!(f, " {index:02}. {}", step.to_string().indent(8).trim())?;
= }
= }
= writeln!(f, "")?;
@@ -495,6 +405,60 @@ impl Suite {
= }
=}
=
+impl Display for Suite {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ writeln!(f, "{title}", title = self.title.bold().underline(),)?;
+
+ if self.tags.is_empty().not() {
+ let tags = self
+ .tags
+ .iter()
+ .map(|tag| tag.underline().to_string())
+ .collect::<Vec<String>>()
+ .join(" ");
+ writeln!(f, "tagged: {tags}")?;
+ }
+
+ let interpreter =
+ format!("interpreter: {interpreter}", interpreter = self.interpreter).dimmed();
+ writeln!(f, "{interpreter}")?;
+
+ // Source of the suite
+ let source = format!(
+ "source: {source}",
+ source = self.source.document_path.display()
+ )
+ .dimmed();
+ writeln!(f, "{source}")
+ }
+}
+
+impl Display for Scenario {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ writeln!(f, "{title}", title = self.title.bold(),)?;
+
+ if self.tags.is_empty().not() {
+ let tags = self
+ .tags
+ .iter()
+ .map(|tag| tag.underline().to_string())
+ .collect::<Vec<String>>()
+ .join(" ");
+ writeln!(f, "tagged: {}", tags)?;
+ }
+
+ // Source of the scenario
+ let source = format!(
+ "source: {document_path}:{first_line}-{last_line}",
+ document_path = self.source.document_path.display(),
+ first_line = self.source.first_line,
+ last_line = self.source.last_line,
+ )
+ .dimmed();
+ writeln!(f, "{source}")
+ }
+}
+
=impl Step {
= fn from_markdown(
= document_path: PathBuf,
@@ -581,3 +545,43 @@ impl Step {
= })
= }
=}
+
+impl Display for Step {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ writeln!(f, "{description}", description = self.description,)?;
+
+ // Arguments, code blocks, lists and tables
+ let mut data_objects = BTreeMap::from([
+ ("code blocks", self.code_blocks.len()),
+ ("lists", self.lists.len()),
+ ("tables", self.tables.len()),
+ ])
+ .iter()
+ .filter_map(|(kind, count)| {
+ if *count == 0 {
+ None
+ } else {
+ Some(format!("{kind}: {count}"))
+ }
+ })
+ .collect::<Vec<String>>();
+
+ if self.arguments.is_empty().not() {
+ data_objects.insert(0, format!("arguments: {:?}", self.arguments));
+ };
+
+ if data_objects.is_empty().not() {
+ writeln!(f, "{}", data_objects.join(", ").dimmed())?;
+ }
+
+ // Source of the step
+ let source = format!(
+ "source: {document_path}:{first_line}-{last_line}",
+ document_path = self.source.document_path.display(),
+ first_line = self.source.first_line,
+ last_line = self.source.last_line,
+ )
+ .dimmed();
+ writeln!(f, "{source}")
+ }
+}Make the evaluation report format similar to show
On by
index d93816d..08973b8 100644
--- a/spec/basic-usage.md
+++ b/spec/basic-usage.md
@@ -125,38 +125,6 @@ Whe a directory is given as the last argument, load all documents inside (recurs
=
=## Evaluating a spec from a single document
=
-A complete sample output is like this:
-
-``` text
-
-✓ Arithmetic
- tagged: math
-
- ⊞ Add 7 and 5 to get 12 ["7", "5", "12"]
- ⊞ Divide 10 by 4 to get 2.5 ["10", "4", "2.5"]
- ⊞ Subtract 7 from 5 to get -2 ["7", "5", "-2"]
-
-✓ Text
- tagged: strings work-in-progress
-
- ⊞ The word blocks has 6 characters ["blocks", "6"]
- ⊞ There are 3 properties in the following JSON ["3"]
- ⊞ There are 3 rs in the word strawberry ["3", "r", "strawberry"]
- ⊞ The following table maps words to their lengths []
- ⊞ The reverse of abc is cba ["abc", "cba"]
- ⊠ The reverse of CIA is KGB ["CIA", "KGB"]
-
- 'KGB' != 'AIC'
- - KGB
- + AIC
-
-
- □ There are 2 os in the word boost ["2", "o", "boost"]
-```
-
-Notice it's similar to the output of `tbb show`, but now contains unicode symbols to indicate the results of each step. Each completed scenario has a check mark `✓`. Successful steps have a squared plus `⊞` . The failing step is markd with a squared times symbol `⊠`. Once a step fails, the subsequent steps in a scenario are not exercised. In the report they are marked with a white square symbol `□`.
-
-
= * Run the program with `evaluate samples/basic.md` command line arguments
=
= The `basic.md` suit is intentionally wrong. It should be reflected in the status code.
@@ -164,8 +132,10 @@ Notice it's similar to the output of `tbb show`, but now contains unicode symbol
= * The output will contain `the suite header` block
=
= ```text
- Basic BDD suite (python -m samples.basic)
+ Basic BDD suite
= tagged: basic not-implemented sample tutorial
+ interpreter: python -m samples.basic
+ source: samples/basic.md
= ```
=
= * The output will contain `arithmetic scenario` block
@@ -173,36 +143,65 @@ Notice it's similar to the output of `tbb show`, but now contains unicode symbol
= ```text
= ✓ Arithmetic
= tagged: math
-
- ⊞ Add 7 and 5 to get 12 ["7", "5", "12"]
- ⊞ Divide 10 by 4 to get 2.5 ["10", "4", "2.5"]
- ⊞ Subtract 7 from 5 to get -2 ["7", "5", "-2"]
+ source: samples/basic.md:22-34
+
+ ⊞ Add 7 and 5 to get 12
+ arguments: ["7", "5", "12"]
+ source: samples/basic.md:30-30
+ ⊞ Divide 10 by 4 to get 2.5
+ arguments: ["10", "4", "2.5"]
+ source: samples/basic.md:31-31
+ ⊞ Subtract 7 from 5 to get -2
+ arguments: ["7", "5", "-2"]
+ source: samples/basic.md:32-34
= ```
=
+ Notice it's similar to the output of `tbb show`, but now contains unicode symbols to indicate the results of each step. Each completed scenario has a check mark `✓`.
+
= * The output will contain `text scenario` block
=
= ```text
= ✓ Text
= tagged: strings work-in-progress
-
- ⊞ The word blocks has 6 characters ["blocks", "6"]
- ⊞ There are 3 properties in the following JSON ["3"]
- ⊞ There are 3 rs in the word strawberry ["3", "r", "strawberry"]
- ⊞ The following table maps words to their lengths []
- ⊞ The reverse of abc is cba ["abc", "cba"]
- ⊠ The reverse of CIA is KGB ["CIA", "KGB"]
+ source: samples/basic.md:35-91
+
+ ⊞ The word blocks has 6 characters
+ arguments: ["blocks", "6"]
+ source: samples/basic.md:45-45
+ ⊞ There are 3 properties in the following JSON
+ arguments: ["3"], code blocks: 1
+ source: samples/basic.md:46-57
+ ⊞ There are 3 rs in the word strawberry
+ arguments: ["3", "r", "strawberry"]
+ source: samples/basic.md:58-58
+ ⊞ The following table maps words to their lengths
+ code blocks: 1, tables: 1
+ source: samples/basic.md:59-84
+ ⊞ The reverse of abc is cba
+ arguments: ["abc", "cba"]
+ source: samples/basic.md:85-85
+ ⊠ The reverse of CIA is KGB
+ arguments: ["CIA", "KGB"]
+ source: samples/basic.md:86-89
=
= 'KGB' != 'AIC'
=
= - KGB
= + AIC
+
+ □ There are 2 os in the word boost
+ arguments: ["2", "o", "boost"]
+ source: samples/basic.md:90-91
= ```
+
+ Successful steps have a squared plus `⊞` . The failing step is markd with a squared times symbol `⊠`. Once a step fails, the subsequent steps in a scenario are not exercised. In the report they are marked with a white square symbol `□`.
=
= * The output will contain `geometry scenario` block
=
= ```text
= ? Geometry
= tagged: math
+ source: samples/basic.md:92-99
=
= There are no steps to execute in this scenario.
= ```index c0c4dec..2696fb8 100644
--- a/spec/failing-interpreters.md
+++ b/spec/failing-interpreters.md
@@ -18,15 +18,19 @@ If some interpreters' commands are invalid (e.g. refers to a non-existing progra
= * The output will contain `the valid suite header` block
=
= ```text
- A little suite that could (python -m samples.basic)
+ A little suite that could
= tagged: basic passing
+ interpreter: python -m samples.basic
+ source: samples/some-invalid/valid.md
= ```
=
= * The output will contain `the invalid suite header` block
=
= ```text
- Suite 1 from the invalid document (invalid interpreter)
+ Suite 1 from the invalid document
= tagged: not-implemented seriously-underbaked
+ interpreter: invalid interpreter
+ source: samples/some-invalid/invalid.md
= ```
=
= * The output will contain `the failing scenario` block
@@ -34,6 +38,7 @@ If some interpreters' commands are invalid (e.g. refers to a non-existing progra
= ```text
= x Hopeless scenario
= tagged: even more tags very-important work-in-progress
+ source: samples/some-invalid/invalid.md:17-29
= ```
=
= Notice the failing sigil in front.index e462842..357a1ae 100644
--- a/spec/filtering.md
+++ b/spec/filtering.md
@@ -15,8 +15,10 @@ With the `--only suite:<tag>` only suites that have the given tag should be incl
= * The output will contain `the expected suite header` block
=
= ```text
- A little suite that could (python -m samples.basic)
+ A little suite that could
= tagged: basic passing
+ interpreter: python -m samples.basic
+ source: samples/some-invalid/valid.md
= ```
=
= * The output will not contain `Suite 1 from the invalid document`
@@ -31,8 +33,10 @@ With the `--only scenario:<tag>` only scenarios that have the given tag should b
= * The output will contain `the expected suite header` block
=
= ```text
- A little suite that could (python -m samples.basic)
+ A little suite that could
= tagged: basic passing
+ interpreter: python -m samples.basic
+ source: samples/some-invalid/valid.md
= ```
=
= * The output will contain `the expected scenario header` block
@@ -62,8 +66,10 @@ With the `--exclude suite:<tag>` any suites that have the given tag should be ex
= * The output will contain `the passing suite header` block
=
= ```text
- A little suite that could (python -m samples.basic)
+ A little suite that could
= tagged: basic passing
+ interpreter: python -m samples.basic
+ source: samples/some-invalid/valid.md
= ```
=
= * The output will not contain `Suite 1 from the invalid document`
@@ -77,8 +83,10 @@ With the `--exclude scenario:<tag>` any scenarios that have the given tag should
= * The output will contain `the expected suite header` block
=
= ```text
- A little suite that could (python -m samples.basic)
+ A little suite that could
= tagged: basic passing
+ interpreter: python -m samples.basic
+ source: samples/some-invalid/valid.md
= ```
=
= * The output will contain `the expected scenario header` blockindex 546bbf9..0205116 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -275,22 +275,7 @@ pub enum ScenarioStatus {
=impl Display for EvaluationReport<'_> {
= fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
= for SuiteReport { suite, scenarios } in self.suites.iter() {
- writeln!(
- f,
- "\n{title} ({interpreter})",
- title = suite.title.bold().underline(),
- interpreter = suite.interpreter.dimmed()
- )?;
-
- if suite.tags.is_empty().not() {
- let tags = suite
- .tags
- .iter()
- .map(|tag| tag.underline().to_string())
- .collect::<Vec<String>>()
- .join(" ");
- writeln!(f, "tagged: {tags}",)?;
- }
+ writeln!(f, "\n{suite}")?;
=
= for ScenarioReport {
= scenario,
@@ -303,21 +288,7 @@ impl Display for EvaluationReport<'_> {
= ScenarioStatus::Pending => "?".to_string().bold().dimmed(),
= ScenarioStatus::FailedToRun { .. } => "x".to_string().bold().red(),
= };
- writeln!(
- f,
- "\n{indentation}{sigil} {title}",
- indentation = "".indent(0),
- title = scenario.title.bold()
- )?;
- if scenario.tags.is_empty().not() {
- let tags = scenario
- .tags
- .iter()
- .map(|tag| tag.underline().to_string())
- .collect::<Vec<String>>()
- .join(" ");
- writeln!(f, "{indentation}tagged: {tags}", indentation = "".indent(2))?;
- };
+ writeln!(f, "\n{sigil} {}", scenario.to_string().indent(2).trim())?;
= writeln!(f, "")?;
=
= if let ScenarioStatus::FailedToRun { error } = status {
@@ -350,10 +321,9 @@ impl Display for EvaluationReport<'_> {
= };
= writeln!(
= f,
- "{indentation}{sigil} {description} {arguments}",
+ "{indentation}{sigil} {}",
+ step.to_string().indent(4).trim(),
= indentation = "".indent(2),
- description = step.description,
- arguments = format!("{:?}", step.arguments).dimmed()
= )?;
=
= if let StepStatus::Failed { reason, hint } = status {Condense output when all steps are ok
On by
To make finding errors easier.
Also, when there are no steps in a scenario, print the information in yellow (so it stands out more and nags for improvement).
Also use a failing sigil for scenarios that have a failed step.
index 08973b8..920caa7 100644
--- a/spec/basic-usage.md
+++ b/spec/basic-usage.md
@@ -145,41 +145,39 @@ Whe a directory is given as the last argument, load all documents inside (recurs
= tagged: math
= source: samples/basic.md:22-34
=
- ⊞ Add 7 and 5 to get 12
- arguments: ["7", "5", "12"]
- source: samples/basic.md:30-30
- ⊞ Divide 10 by 4 to get 2.5
- arguments: ["10", "4", "2.5"]
- source: samples/basic.md:31-31
- ⊞ Subtract 7 from 5 to get -2
- arguments: ["7", "5", "-2"]
- source: samples/basic.md:32-34
+ ⊞ ⊞ ⊞
= ```
=
- Notice it's similar to the output of `tbb show`, but now contains unicode symbols to indicate the results of each step. Each completed scenario has a check mark `✓`.
+ Notice it's similar to the output of `tbb show`, but now contains unicode symbols to indicate the results of each step. Each completed scenario has a check mark `✓`. The number of steps is indicated by the `⊞` sigils.
=
= * The output will contain `text scenario` block
=
= ```text
- ✓ Text
+ x Text
= tagged: strings work-in-progress
= source: samples/basic.md:35-91
=
+
= ⊞ The word blocks has 6 characters
= arguments: ["blocks", "6"]
= source: samples/basic.md:45-45
+
= ⊞ There are 3 properties in the following JSON
= arguments: ["3"], code blocks: 1
= source: samples/basic.md:46-57
+
= ⊞ There are 3 rs in the word strawberry
= arguments: ["3", "r", "strawberry"]
= source: samples/basic.md:58-58
+
= ⊞ The following table maps words to their lengths
= code blocks: 1, tables: 1
= source: samples/basic.md:59-84
+
= ⊞ The reverse of abc is cba
= arguments: ["abc", "cba"]
= source: samples/basic.md:85-85
+
= ⊠ The reverse of CIA is KGB
= arguments: ["CIA", "KGB"]
= source: samples/basic.md:86-89
@@ -194,7 +192,7 @@ Whe a directory is given as the last argument, load all documents inside (recurs
= source: samples/basic.md:90-91
= ```
=
- Successful steps have a squared plus `⊞` . The failing step is markd with a squared times symbol `⊠`. Once a step fails, the subsequent steps in a scenario are not exercised. In the report they are marked with a white square symbol `□`.
+ If some steps fail, the output will be expanded. Successful steps have a squared plus `⊞` sigil . The failing step is markd with a squared times symbol `⊠`. Once a step fails, the subsequent steps in a scenario are not exercised. In the report they are marked with a white square symbol `□`.
=
= * The output will contain `geometry scenario` block
=index 0205116..30dcf1a 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -257,6 +257,7 @@ pub struct StepReport<'a> {
= status: StepStatus,
=}
=
+#[derive(Debug, Eq, PartialEq)]
=pub enum StepStatus {
= Ok,
= Failed {
@@ -283,8 +284,15 @@ impl Display for EvaluationReport<'_> {
= steps,
= } in scenarios.iter()
= {
+ let all_ok = steps.iter().all(|step| step.status == StepStatus::Ok);
= let sigil = match status {
- ScenarioStatus::Done => "✓".to_string().bold(), // TODO: Use different icon depending on steps status
+ ScenarioStatus::Done => {
+ if all_ok {
+ "✓".to_string().bold().green()
+ } else {
+ "x".to_string().bold().red()
+ }
+ }
= ScenarioStatus::Pending => "?".to_string().bold().dimmed(),
= ScenarioStatus::FailedToRun { .. } => "x".to_string().bold().red(),
= };
@@ -309,9 +317,12 @@ impl Display for EvaluationReport<'_> {
= "{}",
= "There are no steps to execute in this scenario."
= .indent(2)
- .dimmed()
+ .yellow()
= )?
- }
+ } else if all_ok {
+ // Indentation for condensed output
+ write!(f, " ")?;
+ };
=
= for StepReport { step, status } in steps.iter() {
= let sigil = match status {
@@ -319,9 +330,16 @@ impl Display for EvaluationReport<'_> {
= StepStatus::Failed { .. } => "⊠".to_string().bold().red(),
= StepStatus::NotEvaluated => "□".to_string().bold(),
= };
+
+ // Condense output if everything is ok
+ if all_ok {
+ write!(f, "{sigil} ")?;
+ continue;
+ }
+
= writeln!(
= f,
- "{indentation}{sigil} {}",
+ "\n{indentation}{sigil} {}",
= step.to_string().indent(4).trim(),
= indentation = "".indent(2),
= )?;
@@ -329,10 +347,11 @@ impl Display for EvaluationReport<'_> {
= if let StepStatus::Failed { reason, hint } = status {
= writeln!(f, "\n{}\n", reason.indent(4).red())?;
= if let Some(hint) = hint {
- writeln!(f, "{}\n", hint.as_str().indent(4))?;
+ writeln!(f, "{}", hint.as_str().indent(4))?;
= }
= }
= }
+ writeln!(f, "")?;
= }
= writeln!(f, "")?;
= }Bump minor version to 0.9
On by
To celebrate source information feature, condensed output and renaming "list" to "show".
index 09ff789..0297ef9 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -400,7 +400,7 @@ dependencies = [
=
=[[package]]
=name = "tad-better-behavior"
-version = "0.8.0"
+version = "0.9.0"
=dependencies = [
= "anyhow",
= "clap",index 768e846..e4b9e37 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,6 +1,6 @@
=[package]
=name = "tad-better-behavior"
-version = "0.8.0"
+version = "0.9.0"
=edition = "2024"
=
=[dependencies]Update the roadmap
On by
index 7762c58..d4be474 100644
--- a/roadmap.md
+++ b/roadmap.md
@@ -53,15 +53,15 @@ We use this document mostly to keep track of our progress and a scratchpad for i
= - [ ] More instructive error messages
= - [x] Indent multiline error messages
= - [ ] Collapse ommited steps (`□□□□□□ following 6 steps skipped`)
- - [ ] Collapse filtered out suites
- - [ ] Collapse filtered out scenarios
- - [ ] Mark scenarios without steps
+ - [x] Collapse filtered out suites
+ - [x] Collapse filtered out scenarios
+ - [x] Mark scenarios without steps
=- [ ] Pass more step data to interpreters
= - [x] Code blocks
= - [x] Tables
= - [x] Lists
= - [ ] Definition lists
- - [ ] Original markdown fragment
+ - [x] Original markdown fragment (path and lines range)
= - [ ] Block quotes
=- [x] Nix package (from Flake)
=- [x] Use for evaluating Jewiet's Form to Mail specificationImprove the report format, esp. condensed output
On by
Now multiple skipped steps are also condensed. There is a message explaining what it means.
If there is only one step to display, it will not be condensed (it would look silly and bring almost no saving in output length).
In the spec I elaborate on the reasons for condensed output and few other aspects.
index 920caa7..a22c28e 100644
--- a/spec/basic-usage.md
+++ b/spec/basic-usage.md
@@ -141,73 +141,83 @@ Whe a directory is given as the last argument, load all documents inside (recurs
= * The output will contain `arithmetic scenario` block
=
= ```text
- ✓ Arithmetic
- tagged: math
- source: samples/basic.md:22-34
+ ✓ Arithmetic
+ tagged: math
+ source: samples/basic.md:22-34
=
- ⊞ ⊞ ⊞
+ ⊞ ⊞ ⊞
+ ⬑ all 3 steps successful.
= ```
=
- Notice it's similar to the output of `tbb show`, but now contains unicode symbols to indicate the results of each step. Each completed scenario has a check mark `✓`. The number of steps is indicated by the `⊞` sigils.
+ Notice it's similar to the output of `tbb show`, but now contains unicode symbols to indicate the results of each step. Each completed scenario has a check mark `✓`. The number of steps is indicated by the `⊞` sigils and a message on the following line. This is called "condensed output".
+
+ TODO: Create a separate "condensed output" suite that focuses on this aspect.
=
= * The output will contain `text scenario` block
=
= ```text
- x Text
- tagged: strings work-in-progress
- source: samples/basic.md:35-91
+ x Text
+ tagged: strings work-in-progress
+ source: samples/basic.md:35-91
=
=
- ⊞ The word blocks has 6 characters
- arguments: ["blocks", "6"]
- source: samples/basic.md:45-45
+ ⊞ The word blocks has 6 characters
+ arguments: ["blocks", "6"]
+ source: samples/basic.md:45-45
=
- ⊞ There are 3 properties in the following JSON
- arguments: ["3"], code blocks: 1
- source: samples/basic.md:46-57
+ ⊞ There are 3 properties in the following JSON
+ arguments: ["3"], code blocks: 1
+ source: samples/basic.md:46-57
=
- ⊞ There are 3 rs in the word strawberry
- arguments: ["3", "r", "strawberry"]
- source: samples/basic.md:58-58
+ ⊞ There are 3 rs in the word strawberry
+ arguments: ["3", "r", "strawberry"]
+ source: samples/basic.md:58-58
=
- ⊞ The following table maps words to their lengths
- code blocks: 1, tables: 1
- source: samples/basic.md:59-84
+ ⊞ The following table maps words to their lengths
+ code blocks: 1, tables: 1
+ source: samples/basic.md:59-84
=
- ⊞ The reverse of abc is cba
- arguments: ["abc", "cba"]
- source: samples/basic.md:85-85
+ ⊞ The reverse of abc is cba
+ arguments: ["abc", "cba"]
+ source: samples/basic.md:85-85
=
- ⊠ The reverse of CIA is KGB
- arguments: ["CIA", "KGB"]
- source: samples/basic.md:86-89
+ ⊠ The reverse of CIA is KGB
+ arguments: ["CIA", "KGB"]
+ source: samples/basic.md:86-89
=
- 'KGB' != 'AIC'
+ 'KGB' != 'AIC'
=
- - KGB
- + AIC
+ - KGB
+ + AIC
=
- □ There are 2 os in the word boost
- arguments: ["2", "o", "boost"]
- source: samples/basic.md:90-91
+ □ There are 2 os in the word boost
+ arguments: ["2", "o", "boost"]
+ source: samples/basic.md:90-91
= ```
=
= If some steps fail, the output will be expanded. Successful steps have a squared plus `⊞` sigil . The failing step is markd with a squared times symbol `⊠`. Once a step fails, the subsequent steps in a scenario are not exercised. In the report they are marked with a white square symbol `□`.
+
+ To aid in debugging, each step is now expanded, showing it's source position and arguments.
=
= * The output will contain `geometry scenario` block
=
= ```text
- ? Geometry
- tagged: math
- source: samples/basic.md:92-99
+ ? Geometry
+ tagged: math
+ source: samples/basic.md:92-99
=
- There are no steps to execute in this scenario.
+ There are no steps to execute in this scenario.
= ```
+
+ Scenarios without any steps defined (only prose) are marked with the `?` sigil and a note. It's ok to have scenarios like that - often we start with a general thought about a feature, and we want to write it down before specifying precise steps to verify it. But eventually it should be covered. This output should remind us to do so.
=
= * The standard error will contain `\[.+ ERROR +tbb\] Step failed: Basic BDD suite ❯ Text ❯ The reverse of CIA is KGB`
=
+ At the bottom there is a list of errors. Because the reports tend to be long, it's easy to miss a failing step. This should help.
+
= * The exit code should be `1`
=
+
=## Running without a subcommand
=
=Running `tbb` without a subcommand will print the help message (to stderr) and exit with code 2.index 2696fb8..00f409e 100644
--- a/spec/failing-interpreters.md
+++ b/spec/failing-interpreters.md
@@ -6,17 +6,16 @@ interpreter: python -m spec.self-check
=
=Sometimes the failure is not in any particular step, but in a whole scenario, e.g. when an interpreter misbehaves. This suite describes what should happen when the interpreter process misbehaves in various ways.
=
-
=## Some won't start
=
=If some interpreters' commands are invalid (e.g. refers to a non-existing program), but others run fine, then `tbb` should report it as an error and exit with error status code.
=
= * Run the program with `evaluate samples/some-invalid/` command line arguments
-
+
= There are two documents in this directory: `valid.md` and `invalid.md`.
-
+
= * The output will contain `the valid suite header` block
-
+
= ```text
= A little suite that could
= tagged: basic passing
@@ -24,8 +23,23 @@ If some interpreters' commands are invalid (e.g. refers to a non-existing progra
= source: samples/some-invalid/valid.md
= ```
=
+ Some suites failing should not prevent others from being evaluated.
+
+ * The output will contain `the successful scenario` block
+
+ ``` text
+ ✓ Easy scenario
+ source: samples/some-invalid/valid.md:10-14
+
+ ⊞ Add 2 and 2 to get 4
+ arguments: ["2", "2", "4"]
+ source: samples/some-invalid/valid.md:12-14
+ ```
+
+ Notice that the only step of this scenario is not condensed. Condensing it wouldn't make much sense, since it's only a single step.
+
= * The output will contain `the invalid suite header` block
-
+
= ```text
= Suite 1 from the invalid document
= tagged: not-implemented seriously-underbaked
@@ -34,25 +48,32 @@ If some interpreters' commands are invalid (e.g. refers to a non-existing progra
= ```
=
= * The output will contain `the failing scenario` block
-
+
= ```text
- x Hopeless scenario
- tagged: even more tags very-important work-in-progress
- source: samples/some-invalid/invalid.md:17-29
+ x Hopeless scenario
+ tagged: even more tags very-important work-in-progress
+ source: samples/some-invalid/invalid.md:17-29
+
+ can't read from the interpreter process
+
+ * running scenario: Hopeless scenario
+ * awaiting for the interpreter
+ * can't read from the interpreter process
+
+ □ □
+ ⬑ remaining 2 steps were not evaluated.
= ```
-
- Notice the failing sigil in front.
+
+ Notice the failing sigil in front, a reason for failure and that multiple skipped steps are presented in a condensed format. Compare it with the single skipped step in the ./basic-usage.md document. It is not condensed, because there is only one. Here there are multiple, and since they would be executed only after the failing one, they could not have contributed to the failiure. So showing their details wouldn't bring much value.
=
= * The standard error will contain `\[.+ ERROR +tbb\] Scenario failed: Suite 1 from the invalid document ❯ Hopeless scenario`
=
= * The exit code should be `1`
=
-
=## Multiple scenarios failure
=
=When several different scenarios fail, each one should be mentioned in the summary.
=
-
=## The interpreter won't indicate it's ready
=
=What if the process starts, but never sends any messages to output, specifically the `Ready` message? There should be a timeout period (with a default value of 10s, but configurable per scenario). After the timeout, the interpreter process should be terminated and scenario marked as failed. The output should clearly indicate what the problem was.index 357a1ae..93feb2e 100644
--- a/spec/filtering.md
+++ b/spec/filtering.md
@@ -42,8 +42,9 @@ With the `--only scenario:<tag>` only scenarios that have the given tag should b
= * The output will contain `the expected scenario header` block
=
= ```text
- ✓ Interesting scenario
- tagged: interesting
+ ✓ Interesting scenario
+ tagged: interesting
+ source: samples/some-invalid/valid.md:15-22
= ```
=
= * The output will contain `Suite 1 from the invalid document`index 30dcf1a..4cd1f31 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -296,17 +296,22 @@ impl Display for EvaluationReport<'_> {
= ScenarioStatus::Pending => "?".to_string().bold().dimmed(),
= ScenarioStatus::FailedToRun { .. } => "x".to_string().bold().red(),
= };
- writeln!(f, "\n{sigil} {}", scenario.to_string().indent(2).trim())?;
+ writeln!(f, "\n {sigil} {}", scenario.to_string().indent(4).trim())?;
= writeln!(f, "")?;
=
= if let ScenarioStatus::FailedToRun { error } = status {
= writeln!(
= f,
= "{cause}\n",
- cause = error.root_cause().to_string().red().indent(2)
+ cause = error.root_cause().to_string().indent(4).red().bold()
= )?;
= for cause in error.chain() {
- writeln!(f, " * {}", cause.to_string().indent(2))?
+ writeln!(
+ f,
+ "{indentation}* {}",
+ cause.to_string(),
+ indentation = "".indent(6)
+ )?
= }
= writeln!(f, "\n")?;
= }
@@ -316,42 +321,57 @@ impl Display for EvaluationReport<'_> {
= f,
= "{}",
= "There are no steps to execute in this scenario."
- .indent(2)
+ .indent(4)
= .yellow()
+ .bold()
= )?
- } else if all_ok {
- // Indentation for condensed output
- write!(f, " ")?;
- };
-
- for StepReport { step, status } in steps.iter() {
- let sigil = match status {
- StepStatus::Ok => "⊞".to_string().bold().green(),
- StepStatus::Failed { .. } => "⊠".to_string().bold().red(),
- StepStatus::NotEvaluated => "□".to_string().bold(),
- };
-
- // Condense output if everything is ok
- if all_ok {
- write!(f, "{sigil} ")?;
- continue;
- }
-
- writeln!(
- f,
- "\n{indentation}{sigil} {}",
- step.to_string().indent(4).trim(),
- indentation = "".indent(2),
- )?;
+ } else if all_ok && steps.len() > 1 && f.alternate().not() {
+ // Condense the output when all steps are successful
+ let sigils = "⊞ ".repeat(steps.len()).trim().indent(4).green();
+ let message = format!("⬑ all {count} steps successful.", count = steps.len())
+ .indent(4)
+ .dimmed()
+ .bold();
+ writeln!(f, "{sigils}")?;
+ writeln!(f, "{message}")?;
+ } else {
+ for (index, StepReport { step, status }) in steps.iter().enumerate() {
+ let sigil = match status {
+ StepStatus::Ok => "⊞".to_string().bold().green(),
+ StepStatus::Failed { .. } => "⊠".to_string().bold().red(),
+ StepStatus::NotEvaluated => "□".to_string().bold(),
+ };
+
+ let remaining_count = steps.len() - index;
+ if *status == StepStatus::NotEvaluated && remaining_count > 1 {
+ // Condense the output when all steps are successful
+ let sigils = "□ ".repeat(remaining_count).trim().indent(4);
+ let message =
+ format!("⬑ remaining {remaining_count} steps were not evaluated.")
+ .indent(4)
+ .dimmed()
+ .bold();
+ writeln!(f, "{sigils}")?;
+ writeln!(f, "{message}")?;
+ break;
+ }
=
- if let StepStatus::Failed { reason, hint } = status {
- writeln!(f, "\n{}\n", reason.indent(4).red())?;
- if let Some(hint) = hint {
- writeln!(f, "{}", hint.as_str().indent(4))?;
+ writeln!(
+ f,
+ "\n{indentation}{sigil} {}",
+ step.to_string().indent(6).trim(),
+ indentation = "".indent(4),
+ )?;
+
+ if let StepStatus::Failed { reason, hint } = status {
+ writeln!(f, "\n{}\n", reason.indent(6).red())?;
+ if let Some(hint) = hint {
+ writeln!(f, "{}", hint.as_str().indent(6))?;
+ }
= }
= }
- }
- writeln!(f, "")?;
+ writeln!(f, "")?;
+ };
= }
= writeln!(f, "")?;
= }