Week 18 of 2026
Development log of Tad Better Behavior
16 items
- Make interpreter the field of scenario, not suite
- Replace references in the report with ownership
- Get the delegated scenarios to run
- Make a failure in a delegation propagate
- Write more sample specs for delegation
- Implement parameterization for delegations
- Emit an error on an undefined parameter
- Remove some stale TODO comments
- Tweak the parameters to demonstrate a bug
- Fix parameter overriding
- Make the step fail on a failed delegation
- Make the terminal reports better wrt delegations
- Print short statistics at the bottom of the report
- Update the delegations document
- Align the program and spec
- Don't make steps from numbered or commented items
Make interpreter the field of scenario, not suite
On by
Although interpreter is currently set for the whole suite in the front-matter of the Markdown document, it really belongs to a scenario. In principle there is no reason why different scenarios from a suite couldn't be run with different interpreters. We might allow for that via scenario metadata block.
This change simplifies the invocation of Scenario::run by removing the
interpreter argument, and it paves the way to delegations. The
delegations are really what I'm after and what motivates this change.
It's a sort of breaking change, because the reports are now different, with the interpreter being part of the scenario block (or object in JSON output).
index a5e5eec..cf6b5c7 100644
--- a/spec/basic-usage.md
+++ b/spec/basic-usage.md
@@ -83,7 +83,6 @@ When a directory is given as the last argument, load all documents inside (recur
= ``` text
= Basic BDD suite
= tagged: basic not-implemented sample tutorial
- interpreter: python -m samples.basic
= source: samples/basic.md
= ```
=
@@ -92,6 +91,7 @@ When a directory is given as the last argument, load all documents inside (recur
= ``` text
= * Arithmetic
= tagged: math
+ interpreter: python -m samples.basic
= source: samples/basic.md:22-34
=
= 00. Add 7 and 5 to get 12
@@ -110,6 +110,7 @@ When a directory is given as the last argument, load all documents inside (recur
= ``` text
= * Text
= tagged: strings work-in-progress
+ interpreter: python -m samples.basic
= source: samples/basic.md:35-100
=
= 00. The word blocks has 6 characters
@@ -148,7 +149,6 @@ When a directory is given as the last argument, load all documents inside (recur
= ```text
= Basic BDD suite
= tagged: basic not-implemented sample tutorial
- interpreter: python -m samples.basic
= source: samples/basic.md
= ```
=
@@ -157,6 +157,7 @@ When a directory is given as the last argument, load all documents inside (recur
= ```text
= ✓ Arithmetic
= tagged: math
+ interpreter: python -m samples.basic
= source: samples/basic.md:22-34
=
= ⊞ ⊞ ⊞
@@ -172,6 +173,7 @@ When a directory is given as the last argument, load all documents inside (recur
= ```text
= x Text
= tagged: strings work-in-progress
+ interpreter: python -m samples.basic
= source: samples/basic.md:35-100
= ```
=
@@ -325,6 +327,7 @@ When a directory is given as the last argument, load all documents inside (recur
= ```text
= ? Geometry
= tagged: math
+ interpreter: python -m samples.basic
= source: samples/basic.md:101-108
=
= There are no steps to execute in this scenario.index 00f409e..0fbd6d3 100644
--- a/spec/failing-interpreters.md
+++ b/spec/failing-interpreters.md
@@ -19,7 +19,6 @@ If some interpreters' commands are invalid (e.g. refers to a non-existing progra
= ```text
= A little suite that could
= tagged: basic passing
- interpreter: python -m samples.basic
= source: samples/some-invalid/valid.md
= ```
=
@@ -29,6 +28,7 @@ If some interpreters' commands are invalid (e.g. refers to a non-existing progra
=
= ``` text
= ✓ Easy scenario
+ interpreter: python -m samples.basic
= source: samples/some-invalid/valid.md:10-14
=
= ⊞ Add 2 and 2 to get 4
@@ -43,7 +43,6 @@ If some interpreters' commands are invalid (e.g. refers to a non-existing progra
= ```text
= Suite 1 from the invalid document
= tagged: not-implemented seriously-underbaked
- interpreter: invalid interpreter
= source: samples/some-invalid/invalid.md
= ```
=
@@ -52,6 +51,7 @@ If some interpreters' commands are invalid (e.g. refers to a non-existing progra
= ```text
= x Hopeless scenario
= tagged: even more tags very-important work-in-progress
+ interpreter: invalid interpreter
= source: samples/some-invalid/invalid.md:17-29
=
= can't read from the interpreter processindex 93feb2e..fc28512 100644
--- a/spec/filtering.md
+++ b/spec/filtering.md
@@ -17,7 +17,6 @@ With the `--only suite:<tag>` only suites that have the given tag should be incl
= ```text
= A little suite that could
= tagged: basic passing
- interpreter: python -m samples.basic
= source: samples/some-invalid/valid.md
= ```
=
@@ -35,7 +34,6 @@ With the `--only scenario:<tag>` only scenarios that have the given tag should b
= ```text
= A little suite that could
= tagged: basic passing
- interpreter: python -m samples.basic
= source: samples/some-invalid/valid.md
= ```
=
@@ -44,6 +42,7 @@ With the `--only scenario:<tag>` only scenarios that have the given tag should b
= ```text
= ✓ Interesting scenario
= tagged: interesting
+ interpreter: python -m samples.basic
= source: samples/some-invalid/valid.md:15-22
= ```
=
@@ -69,7 +68,6 @@ With the `--exclude suite:<tag>` any suites that have the given tag should be ex
= ```text
= A little suite that could
= tagged: basic passing
- interpreter: python -m samples.basic
= source: samples/some-invalid/valid.md
= ```
=
@@ -86,7 +84,6 @@ With the `--exclude scenario:<tag>` any scenarios that have the given tag should
= ```text
= A little suite that could
= tagged: basic passing
- interpreter: python -m samples.basic
= source: samples/some-invalid/valid.md
= ```
=index 6189733..fd4a932 100644
--- a/spec/json-output.md
+++ b/spec/json-output.md
@@ -10,8 +10,10 @@ interpreter: "python -m spec.self-check"
= * Parse the output as JSON and store it as `the output`
= * From `the output` get the property `suites` as `the suites`
= * From `the suites` get the element where `title` equals `"Suite 1 from the invalid document"` as `the invalid suite`
- * The property `interpreter` of `the invalid suite` equals to `"invalid interpreter"`
= * The property `tags` of `the invalid suite` is an array that includes the value `"not-implemented"`
+ * From `the invalid suite` get the property `scenarios` as `the scenarios`
+ * From `the scenarios` get the element where `title` equals `"Hopeless scenario"` as `the hopeless scenario`
+ * The property `interpreter` of `the hopeless scenario` equals to `"invalid interpreter"`
= * The exit code should be `0`
=
=
@@ -28,9 +30,10 @@ interpreter: "python -m spec.self-check"
= TODO: Consider if we could get rid of this difference. Basically only have a `Report` struct. Upon loading, it would have all steps in `Pending` state. This way a lot of code could be removed and the structure would become flatter.
=
= * From `the suite report` get the property `suite` as `the suite spec`
- * The property `interpreter` of `the suite spec` equals to `"python -m samples.basic"`
= * From `the suite report` get the property `scenarios` as `the scenarios`
= * From `the scenarios` get the element with the value at `["scenario", "title"]` equal to `"Text"` as `the text scenario report`
+ * From `the text scenario report` get the property `scenario` as `the text scenario`
+ * The property `interpreter` of `the text scenario` equals to `"python -m samples.basic"`
= * The property `status` of `the text scenario report` equals to `"Done"`
=
= The execution is done, although some steps failed!index 4b2deed..24fa596 100644
--- a/spec/nushell-library.md
+++ b/spec/nushell-library.md
@@ -18,7 +18,6 @@ There is a library for implementing interpreters in [Nushell](https://www.nushel
= ```text
= Stateful
= tagged: not-implemented
- interpreter: nu --stdin samples/basic.nu
= source: samples/stateful.md
= ```
=
@@ -26,6 +25,7 @@ There is a library for implementing interpreters in [Nushell](https://www.nushel
=
= ```text
= x Count the Twists
+ interpreter: nu --stdin samples/basic.nu
= source: samples/stateful.md:14-30
= ```
=index c65ba7f..79bd7c2 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -87,7 +87,7 @@ impl<'a> SuiteReport<'a> {
=
= for scenario in self.scenarios.iter_mut() {
= let result = scenario
- .run(&self.suite.interpreter, verbose)
+ .run(verbose)
= .context(format!("running scenario: {}", scenario.scenario.title));
= if let Err(error) = result {
= scenario.status = ScenarioStatus::FailedToRun { error }
@@ -106,15 +106,16 @@ pub struct ScenarioReport<'a> {
=}
=impl<'a> ScenarioReport<'a> {
= // TODO: The ScenarioReport::run method is very effectful. I think it should live in it's own module.
- fn run(&mut self, interpreter: &str, verbose: bool) -> anyhow::Result<()> {
+ fn run(&mut self, verbose: bool) -> anyhow::Result<()> {
= log::debug!(
- "Running scenario '{}' using '{interpreter}' as an interpreter",
- self.scenario.title
+ "Running scenario '{title}' using '{interpreter}' as an interpreter",
+ title = self.scenario.title,
+ interpreter = self.scenario.interpreter
= );
=
= let mut process = Command::new("sh")
= .arg("-c")
- .arg(interpreter)
+ .arg(&self.scenario.interpreter)
= .stdin(Stdio::piped())
= .stdout(Stdio::piped())
= .env(
@@ -122,7 +123,10 @@ impl<'a> ScenarioReport<'a> {
= if verbose { "debug" } else { "info" },
= )
= .spawn()
- .context(format!("spawning interpreter from {interpreter}"))?;
+ .context(format!(
+ "spawning interpreter from {interpreter}",
+ interpreter = self.scenario.interpreter
+ ))?;
=
= let Some(input) = process.stdin.take() else {
= // TODO: Avoid string errors!
@@ -244,7 +248,9 @@ impl<'a> ScenarioReport<'a> {
=
= Interpreter: {interpreter}
= Exit code: {exit_code}"
- "#}))
+ "#,
+ interpreter = self.scenario.interpreter
+ },))
= }
= })
= }index 49432c3..878dd74 100644
--- a/src/spec.rs
+++ b/src/spec.rs
@@ -31,7 +31,6 @@ pub struct Spec {
=pub struct Suite {
= pub title: String,
= pub tags: BTreeSet<String>,
- pub interpreter: String, // TODO: Interpreter should be a property of a scenario?
= pub scenarios: Vec<Scenario>,
= pub source: SourceInfromation,
=}
@@ -48,6 +47,7 @@ pub struct Scenario {
= pub tags: BTreeSet<String>,
= pub steps: Vec<Step>,
= pub source: SourceInfromation,
+ pub interpreter: String,
=}
=
=#[derive(Debug, Serialize, Deserialize, Clone)]
@@ -347,6 +347,7 @@ impl Suite {
= // Assume the scenario goes until the end of document, until proven otherwise
= last_line: source.last_line,
= },
+ interpreter: frontmatter.interpreter.clone(),
= });
= }
= }
@@ -399,7 +400,6 @@ impl Suite {
= title,
= tags: suite_tags,
= scenarios,
- interpreter: frontmatter.interpreter,
= source,
= })
= }
@@ -419,10 +419,6 @@ impl Display for Suite {
= writeln!(f, "tagged: {tags}")?;
= }
=
- let interpreter =
- format!("interpreter: {interpreter}", interpreter = self.interpreter).dimmed();
- writeln!(f, "{interpreter}")?;
-
= // Source of the suite
= let source = format!(
= "source: {source}",
@@ -447,7 +443,10 @@ impl Display for Scenario {
= writeln!(f, "tagged: {}", tags)?;
= }
=
- // Source of the scenario
+ let interpreter =
+ format!("interpreter: {interpreter}", interpreter = self.interpreter).dimmed();
+ writeln!(f, "{interpreter}")?;
+
= let source = format!(
= "source: {document_path}:{first_line}-{last_line}",
= document_path = self.source.document_path.display(),Replace references in the report with ownership
On by
I've spent too many hours fighting the borrow checker and trying to load a delegated scenarios. Let's try to implement it with owned (i.e. cloned) values.
The problem with this is that now every SuiteReport contains a copy of Suite, with its scenarios and steps, but also a vector of ScenarioReport items - each with a copy of a Scenario with its steps. ScenarioReport also contains a vector of StepReport items, each containing a copy of a Step. So in total every step is stored in 3 copies, for no good reason.
If I get it to work I can try to refactor it to use references again.
index 7da77aa..acf281b 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -188,7 +188,7 @@ fn evaluate(
= };
= }
=
- let mut report = EvaluationReport::new(&spec, &options);
+ let mut report = EvaluationReport::new(spec, &options);
= report.evaluate(verbose);
=
= // Print the report on STDOUTindex 79bd7c2..a3ec641 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -27,18 +27,18 @@ pub struct ReportOptions {
=}
=
=#[derive(Debug, Serialize)]
-pub struct EvaluationReport<'a> {
- suites: Vec<SuiteReport<'a>>,
+pub struct EvaluationReport {
+ suites: Vec<SuiteReport>,
=}
=
-impl<'a> EvaluationReport<'a> {
+impl EvaluationReport {
= /// Create a blank report to be evaluated
= ///
= /// We start with a report pending to be evaluated. It holds references to
= /// Spec, Suites, Scenarios and Steps (all in pending state). The evaluator
= /// should walk through the report, and "fill" it. That way, even if it
= /// fails to finish it's job, the report will be there to display.
- pub fn new(spec: &'a Spec, options: &ReportOptions) -> Self {
+ pub fn new(spec: Spec, options: &ReportOptions) -> Self {
= log::debug!("{options:#?}");
=
= Self {
@@ -49,7 +49,7 @@ impl<'a> EvaluationReport<'a> {
= suite.tags.is_superset(&options.only_suites)
= && suite.tags.is_disjoint(&options.exclude_suites)
= })
- .map(|suite| SuiteReport::from_suite(suite, options))
+ .map(|suite| SuiteReport::from_suite(suite.clone(), options))
= .collect(),
= }
= }
@@ -63,13 +63,13 @@ impl<'a> EvaluationReport<'a> {
=}
=
=#[derive(Debug, Serialize)]
-pub struct SuiteReport<'a> {
- suite: &'a Suite,
- scenarios: Vec<ScenarioReport<'a>>,
+pub struct SuiteReport {
+ suite: Suite,
+ scenarios: Vec<ScenarioReport>,
=}
=
-impl<'a> SuiteReport<'a> {
- fn from_suite(suite: &'a Suite, options: &ReportOptions) -> Self {
+impl SuiteReport {
+ fn from_suite(suite: Suite, options: &ReportOptions) -> Self {
= let scenarios = suite
= .scenarios
= .iter()
@@ -77,6 +77,7 @@ impl<'a> SuiteReport<'a> {
= scenario.tags.is_superset(&options.only_scenarios)
= && scenario.tags.is_disjoint(&options.exclude_scenarios)
= })
+ .cloned()
= .map(ScenarioReport::from)
= .collect();
= Self { suite, scenarios }
@@ -99,12 +100,12 @@ impl<'a> SuiteReport<'a> {
=}
=
=#[derive(Debug, Serialize)]
-pub struct ScenarioReport<'a> {
- scenario: &'a Scenario,
+pub struct ScenarioReport {
+ scenario: Scenario,
= status: ScenarioStatus,
- steps: Vec<StepReport<'a>>,
+ steps: Vec<StepReport>,
=}
-impl<'a> ScenarioReport<'a> {
+impl ScenarioReport {
= // TODO: The ScenarioReport::run method is very effectful. I think it should live in it's own module.
= fn run(&mut self, verbose: bool) -> anyhow::Result<()> {
= log::debug!(
@@ -159,25 +160,25 @@ impl<'a> ScenarioReport<'a> {
= };
= }
=
- 'steps_loop: for &mut StepReport {
- step,
- ref mut observations,
- ref mut delegations,
- ref mut status,
- } in self.steps.iter_mut()
- {
- log::debug!("Taking step '{}'", step.description);
+ 'steps_loop: for step in self.steps.iter_mut() {
+ // let StepReport {
+ // step,
+ // mut ref observations,
+ // mut delegations,
+ // mut status,
+ // } = step;
+ log::debug!("Taking step '{}'", step.step.description);
= Self::send(
= &mut writer,
= &ControlMessage::Execute {
- step: step.to_owned(),
+ step: step.step.to_owned(),
= },
= )?;
=
= loop {
= match Self::receive(&mut reader)? {
= InterpreterMessage::Text { content } => {
- observations.push(Observation::Text { content })
+ step.observations.push(Observation::Text { content })
= }
= InterpreterMessage::Snippet {
= language,
@@ -185,7 +186,7 @@ impl<'a> ScenarioReport<'a> {
= content,
= caption,
= } => {
- observations.push(Observation::Snippet {
+ step.observations.push(Observation::Snippet {
= language,
= meta,
= content,
@@ -193,13 +194,13 @@ impl<'a> ScenarioReport<'a> {
= });
= }
= InterpreterMessage::Link { url, label } => {
- observations.push(Observation::Link { url, label });
+ step.observations.push(Observation::Link { url, label });
= }
= InterpreterMessage::Delegate {
= suite,
= scenario,
= parameters,
- } => delegations.push(Delegation {
+ } => step.delegations.push(Delegation {
= suite: PathBuf::from(suite),
= scenario,
= parameters: parameters,
@@ -207,12 +208,12 @@ impl<'a> ScenarioReport<'a> {
= }),
= InterpreterMessage::Success => {
= log::debug!("Step executed successfully: {step:#?}");
- *status = StepStatus::Ok;
+ step.status = StepStatus::Ok;
= break; // Proceed with the next step
= }
= InterpreterMessage::Failure { reason, hint } => {
= log::debug!("Step failed:\n\n {step:#?} \n\n {reason:#?}");
- *status = StepStatus::Failed { reason, hint };
+ step.status = StepStatus::Failed { reason, hint };
=
= // Do not execute subsequent steps.
= //
@@ -297,8 +298,8 @@ impl<'a> ScenarioReport<'a> {
=}
=
=#[derive(Debug, Serialize)]
-pub struct StepReport<'a> {
- step: &'a Step,
+pub struct StepReport {
+ step: Step,
= status: StepStatus,
= observations: Vec<Observation>,
= delegations: Vec<Delegation>,
@@ -398,7 +399,7 @@ where
= serializer.serialize_str(&format!("{error:?}"))
=}
=
-impl Display for EvaluationReport<'_> {
+impl Display for EvaluationReport {
= fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
= for SuiteReport { suite, scenarios } in self.suites.iter() {
= writeln!(f, "\n{suite}")?;
@@ -525,9 +526,14 @@ impl Display for EvaluationReport<'_> {
= }
=}
=
-impl<'a> From<&'a Scenario> for ScenarioReport<'a> {
- fn from(scenario: &'a Scenario) -> Self {
- let steps = scenario.steps.iter().map(StepReport::from).collect();
+impl From<Scenario> for ScenarioReport {
+ fn from(scenario: Scenario) -> Self {
+ let steps = scenario
+ .steps
+ .iter()
+ .cloned()
+ .map(StepReport::from)
+ .collect();
= Self {
= scenario,
= status: ScenarioStatus::Pending,
@@ -536,8 +542,8 @@ impl<'a> From<&'a Scenario> for ScenarioReport<'a> {
= }
=}
=
-impl<'a> From<&'a Step> for StepReport<'a> {
- fn from(step: &'a Step) -> Self {
+impl From<Step> for StepReport {
+ fn from(step: Step) -> Self {
= Self {
= step,
= status: StepStatus::NotEvaluated,
@@ -594,7 +600,7 @@ pub enum EvaluationSummary {
= },
=}
=
-impl<'a> From<EvaluationReport<'a>> for EvaluationSummary {
+impl From<EvaluationReport> for EvaluationSummary {
= fn from(report: EvaluationReport) -> Self {
= let mut failed_scenarios: Vec<(Suite, Scenario, Option<Step>)> = Vec::default();
= let mut anything_evaluated = false;Get the delegated scenarios to run
On by
It works in principle, but:
- There is no support for parameters yet
- Failing step in a delegation doesn't terminate the delegating scenario
- The terminal display is not great
- ... and the display code is even worse spaghetti that before
But I'm very happy I got that far after many hours of trying to satisfy Rust's borrow checker - and ultimately failing.
There is a lot to improve, but tomorrow I'll start from a good place.
index fb31490..3f22c3c 100644
--- a/samples/delegating.py
+++ b/samples/delegating.py
@@ -11,7 +11,7 @@ tester = unittest.TestCase()
=@step("Run the {0} scenario from the {1} suite")
=def step_implementation_01(scenario: str, suite: str, **kwargs):
= tbb.delegate(suite, scenario)
- tbb.delegate(suite, "Hardcoded scenario title")
+ # tbb.delegate("samples/dupa.md", "Hardcoded scenario title")
=
=
=tbb.ready()index 27b881a..c8cc5ac 100644
--- a/samples/delegations.md
+++ b/samples/delegations.md
@@ -9,3 +9,4 @@ This sample and related interpreter demonstrates how to use delegation feature o
=## Simple delegation
=
= * Run the `Arithmetic` scenario from the `samples/basic.md` suite
+ * Run the `Text` scenario from the `samples/basic.md` suiteindex a3ec641..62b48d0 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -200,12 +200,17 @@ impl ScenarioReport {
= suite,
= scenario,
= parameters,
- } => step.delegations.push(Delegation {
- suite: PathBuf::from(suite),
- scenario,
- parameters: parameters,
- status: ScenarioStatus::Pending,
- }),
+ } => {
+ let path = PathBuf::from(suite);
+ let scenario = Suite::try_from(&path)?
+ .get_scenario(&scenario)
+ .context("Scenario not found")?;
+ let scenario = ScenarioReport::from(scenario);
+ step.delegations.push(Delegation {
+ scenario,
+ parameters: parameters,
+ })
+ }
= InterpreterMessage::Success => {
= log::debug!("Step executed successfully: {step:#?}");
= step.status = StepStatus::Ok;
@@ -231,6 +236,12 @@ impl ScenarioReport {
= }
= };
= }
+ for delegation in step.delegations.iter_mut() {
+ // TODO: Handle parameters
+ // TODO: If any step fails, fail the parent step
+ // TODO: Consider infinite loops and stuff
+ delegation.scenario.run(verbose)?;
+ }
= }
=
= drop(writer);
@@ -359,16 +370,76 @@ impl Display for Observation {
=
=#[derive(Debug, Serialize)]
=struct Delegation {
- suite: PathBuf,
- scenario: String,
+ scenario: ScenarioReport,
= parameters: HashMap<String, String>,
- status: ScenarioStatus,
=}
=
=impl Display for Delegation {
= fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ let ScenarioReport {
+ ref scenario,
+ ref status,
+ ref steps,
+ } = self.scenario;
+
= let sigil = "☎";
- writeln!(f, "{sigil} {}", self.scenario)
+
+ let source = format!(
+ "({document_path}:{first_line}-{last_line})",
+ document_path = scenario.source.document_path.display(),
+ first_line = scenario.source.first_line,
+ last_line = scenario.source.last_line,
+ )
+ .dimmed();
+
+ writeln!(
+ f,
+ "\n{indentation}{sigil} {title} {source}",
+ indentation = "".indent(4),
+ title = scenario.title,
+ )?;
+
+ // TODO: DRY with Evaluation report
+ // Or even better refactor the display logic. It's a spaghetti!
+ for (
+ index,
+ StepReport {
+ step,
+ status,
+ observations,
+ delegations,
+ },
+ ) in steps.iter().enumerate()
+ {
+ let sigil = match status {
+ StepStatus::Ok => "⊞".to_string().bold().green(),
+ StepStatus::Failed { .. } => "⊠".to_string().bold().red(),
+ StepStatus::NotEvaluated => "□".to_string().bold(),
+ };
+
+ writeln!(
+ f,
+ "\n{indentation}{sigil} {}",
+ step.to_string().indent(6).trim(),
+ indentation = "".indent(4),
+ )?;
+
+ for delegation in delegations {
+ writeln!(f, "{}", delegation.to_string().indent(0))?;
+ }
+
+ for observation in observations {
+ writeln!(f, "\n{}", observation.to_string().indent(6))?;
+ }
+
+ if let StepStatus::Failed { reason, hint } = status {
+ writeln!(f, "\n{}\n", reason.indent(6).red())?;
+ if let Some(hint) = hint {
+ writeln!(f, "{}", hint.as_str().indent(6))?;
+ }
+ }
+ }
+ Ok(())
= }
=}
=
@@ -475,6 +546,7 @@ impl Display for EvaluationReport {
= StepStatus::NotEvaluated => "□".to_string().bold(),
= };
=
+ // TODO: Take delegations into account
= let remaining_count = steps.len() - index;
= if *status == StepStatus::NotEvaluated && remaining_count > 1 {
= // Condense the output when all steps are successful
@@ -496,13 +568,8 @@ impl Display for EvaluationReport {
= indentation = "".indent(4),
= )?;
=
- if delegations.len() != 0 {
- let heading = format!("\n\nDelegations:\n").indent(6).bold().dimmed();
- writeln!(f, "{heading}")?;
- }
-
= for delegation in delegations {
- writeln!(f, "{}", delegation.to_string().indent(6))?;
+ writeln!(f, "{}", delegation.to_string().indent(0))?;
= }
=
= for observation in observations {index 878dd74..cb29040 100644
--- a/src/spec.rs
+++ b/src/spec.rs
@@ -403,6 +403,13 @@ impl Suite {
= source,
= })
= }
+
+ pub fn get_scenario(&self, scenario: &str) -> Option<Scenario> {
+ self.scenarios
+ .iter()
+ .cloned()
+ .find(|item| item.title == scenario)
+ }
=}
=
=impl Display for Suite {
@@ -584,3 +591,13 @@ impl Display for Step {
= writeln!(f, "{source}")
= }
=}
+
+impl TryFrom<&PathBuf> for Suite {
+ type Error = anyhow::Error;
+
+ fn try_from(document_path: &PathBuf) -> Result<Self, Self::Error> {
+ let md = std::fs::read_to_string(&document_path)
+ .context(format!("reading a file at {}", document_path.display()))?;
+ Self::from_markdown(document_path.to_path_buf(), &md).context("loading a markdown document")
+ }
+}Make a failure in a delegation propagate
On by
I.e. mark the delegating step as failed. Both the delegating and the delegated steps are marked as failed.
This should in principle work recursively, but needs to be tested yet.
index 3f22c3c..20bc71c 100644
--- a/samples/delegating.py
+++ b/samples/delegating.py
@@ -11,7 +11,6 @@ tester = unittest.TestCase()
=@step("Run the {0} scenario from the {1} suite")
=def step_implementation_01(scenario: str, suite: str, **kwargs):
= tbb.delegate(suite, scenario)
- # tbb.delegate("samples/dupa.md", "Hardcoded scenario title")
=
=
=tbb.ready()index c8cc5ac..a5d4eef 100644
--- a/samples/delegations.md
+++ b/samples/delegations.md
@@ -9,4 +9,7 @@ This sample and related interpreter demonstrates how to use delegation feature o
=## Simple delegation
=
= * Run the `Arithmetic` scenario from the `samples/basic.md` suite
- * Run the `Text` scenario from the `samples/basic.md` suite
+
+ * Run the `Count the Twists` scenario from the `samples/stateful.md` suite
+
+ Note that this scenario uses a different interpreter than the previous one.index 62b48d0..ff060c7 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -204,7 +204,7 @@ impl ScenarioReport {
= let path = PathBuf::from(suite);
= let scenario = Suite::try_from(&path)?
= .get_scenario(&scenario)
- .context("Scenario not found")?;
+ .context(format!("Scenario not found: {scenario}"))?;
= let scenario = ScenarioReport::from(scenario);
= step.delegations.push(Delegation {
= scenario,
@@ -241,6 +241,13 @@ impl ScenarioReport {
= // TODO: If any step fails, fail the parent step
= // TODO: Consider infinite loops and stuff
= delegation.scenario.run(verbose)?;
+ if delegation.scenario.is_all_ok().not() {
+ step.status = StepStatus::Failed {
+ reason: "Delegation failed".into(),
+ // TODO: Explain the delegation? Or maybe it should be in the delegation header, next to the telephone?
+ hint: None,
+ }
+ }
= }
= }
=
@@ -267,6 +274,10 @@ impl ScenarioReport {
= })
= }
=
+ fn is_all_ok(&self) -> bool {
+ self.steps.iter().all(|step| step.status == StepStatus::Ok)
+ }
+
= /// Send a message to interpreter
= fn send(
= writer: &mut LineWriter<std::process::ChildStdin>,
@@ -475,16 +486,15 @@ impl Display for EvaluationReport {
= for SuiteReport { suite, scenarios } in self.suites.iter() {
= writeln!(f, "\n{suite}")?;
=
- for ScenarioReport {
+ for sr @ ScenarioReport {
= scenario,
= status,
= steps,
= } in scenarios.iter()
= {
- let all_ok = steps.iter().all(|step| step.status == StepStatus::Ok);
= let sigil = match status {
= ScenarioStatus::Done => {
- if all_ok {
+ if sr.is_all_ok() {
= "✓".to_string().bold().green()
= } else {
= "x".to_string().bold().red()
@@ -520,7 +530,7 @@ impl Display for EvaluationReport {
= .yellow()
= .bold()
= )?
- } else if all_ok && steps.len() > 1 && f.alternate().not() {
+ } else if sr.is_all_ok() && steps.len() > 1 && f.alternate().not() {
= // Condense the output when all steps are successful
= let sigils = "⊞ ".repeat(steps.len()).trim().indent(4).green();
= let message = format!("⬑ all {count} steps successful.", count = steps.len())Write more sample specs for delegation
On by
Just to demonstrate correct handling of different failure cases.
index a5d4eef..2b295a2 100644
--- a/samples/delegations.md
+++ b/samples/delegations.md
@@ -13,3 +13,17 @@ This sample and related interpreter demonstrates how to use delegation feature o
= * Run the `Count the Twists` scenario from the `samples/stateful.md` suite
=
= Note that this scenario uses a different interpreter than the previous one.
+
+
+## Delegation to a non-existing suite
+
+ * Run the `Arithmetic` scenario from the `samples/missing.md` suite
+
+ This step should fail with an appropriate message.
+
+
+## Delegation to an undefined scenario
+
+ * Run the `Missing` scenario from the `samples/basic.md` suite
+
+ This step should fail with an appropriate message.Implement parameterization for delegations
On by
In the Markdown document, a scenario can include parameters in a
metadata block, as demonstrated in the stateful.md.
These parameters map placeholder values, like 2 and 3 to names, like
twists_before_the_nap and twists_after_the_nap. Not all the
arguments need to be mapped, for example nice, sleep and 2:00 are
not parameters and cannot be changed during delegation.
Notice that the placeholder value 2 is used twice in the delegeted
scenario:
- in step one for the number of twists
- in step two for the time to sleep
This demonstrates the ability to re-use the parameter in different positions, but also a possible footgun. It's easy to apply a parameter to more arguments then expected, just because the placeholder value is accidentally the same. I need to find a good solution to mitigate the ambiguity. Maybe a step-scoped parameters?
Applying parameters is demonstrated in the samples/delegations.md
document and samples/delegating.py interpreter.
Internally the parameteres work by mutating the steps in
ScenarioReport. There is a new method for this:
Scenario::with_parametersIt mutates self.steps. Again, the cloning of Spec parts in the
EvaluationReport (introduced a few commits ago) shows it's ugly head.
The ScenarioReport contains two copies of steps: one in scenario field
and another in steps: Vec<StepReport>, where each report contains a
copy of Step. This makes the code error prone. I hope to improve it
some time soon.
One questionable decision is that what is called parameters in
Markdown metadata block is re-named to placeholders in the Scenario
Rust struct. This might be confusing. The term "placeholder" is more
accurate, but I think for the spec authors the term "parameter" might be
more intuitive. I'm debating it and might change the naming later.
Minor chages:
-
I switched from
HashMaptoBTreeMapin parameters, to preserve the order in which they are given. It's not that important, but looks better in reports when a user sees the params in the same order as they are specified in Markdown. -
The observations are now printed before delegations
This was na obvious mistake, as observations from the delegating step are made before delegations are executed.
index 20bc71c..48f9940 100644
--- a/samples/delegating.py
+++ b/samples/delegating.py
@@ -10,7 +10,21 @@ tester = unittest.TestCase()
=
=@step("Run the {0} scenario from the {1} suite")
=def step_implementation_01(scenario: str, suite: str, **kwargs):
- tbb.delegate(suite, scenario)
+ parameters = {}
+
+ # TODO: Be smarter about which table to use.
+ # Maybe a helper in rbb.py to get a table by it's header?
+ # Like `tbb.read_tables(["parameter", "value"], kwargs)`
+ # and that should return a list of lists of { parameter: "a", value: "1" }
+ # A list of lists - one per table , so it's easy to concatenate them if need be.
+ if len(kwargs["tables"]) > 0:
+ table = kwargs["tables"][0]
+
+ for [name, value] in table[1:]:
+ tbb.send_text(f"Setting parameter `{name}` to `{value}`")
+ parameters[name] = value
+
+ tbb.delegate(suite, scenario, **parameters)
=
=
=tbb.ready()index 2b295a2..b117a7d 100644
--- a/samples/delegations.md
+++ b/samples/delegations.md
@@ -12,7 +12,12 @@ This sample and related interpreter demonstrates how to use delegation feature o
=
= * Run the `Count the Twists` scenario from the `samples/stateful.md` suite
=
- Note that this scenario uses a different interpreter than the previous one.
+ | parameter | value |
+ |-----------------------|-------|
+ | twists before the nap | 4 |
+ | twists after the nap | 1 |
+
+ Note that this scenario uses a different interpreter than the previous one. Also, because the parameters are adjusted, it won't fail, like the delegated scenario would with the default parameters. In this delegation the total number of twists doesn't exceed 6, so it's fine.
=
=
=## Delegation to a non-existing suiteindex 50d9872..b2d1faa 100644
--- a/samples/stateful.md
+++ b/samples/stateful.md
@@ -13,6 +13,12 @@ tags: [ not-implemented ] # It's intentionally broken
=
=## Count the Twists
=
+``` yaml tbb
+parameters:
+ twists before the nap: 2
+ twists after the nap: 4
+```
+
= * Do the twist `2` times
=
= That should be easy enough.index ff060c7..c681f07 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -4,7 +4,7 @@ use anyhow::{Context, anyhow};
=use colored::Colorize;
=use indoc::formatdoc;
=use serde::{Deserialize, Serialize, Serializer};
-use std::collections::{BTreeSet, HashMap};
+use std::collections::{BTreeMap, BTreeSet};
=use std::fmt::Display;
=use std::io::{BufRead, BufReader, LineWriter, Write};
=use std::ops::Not;
@@ -106,6 +106,37 @@ pub struct ScenarioReport {
= steps: Vec<StepReport>,
=}
=impl ScenarioReport {
+ fn with_parameters(&mut self, parameters: &ScenarioParameters) -> &mut Self {
+ for (name, placeholder) in self.scenario.placeholders.iter() {
+ if let Some(substitute) = parameters.get(name) {
+ log::info!("Substituting `{}` with `{}`", placeholder, substitute);
+ for report in self.steps.iter_mut() {
+ report.step.arguments = report.step
+ .arguments
+ .iter()
+ .map(|argument| {
+ if argument == placeholder {
+ log::debug!(
+ "Substituting `{name}` from `{argument}` to `{substitute}` in `{}`",
+ report.step.description
+ );
+ substitute
+ } else {
+ argument
+ }
+ })
+ .cloned()
+ .collect();
+
+ log::info!("Step after substitutions: {:#?}", report.step);
+ }
+ }
+ }
+
+ log::info!("Scenario after substitutions: {self:#?}");
+ self
+ }
+
= // TODO: The ScenarioReport::run method is very effectful. I think it should live in it's own module.
= fn run(&mut self, verbose: bool) -> anyhow::Result<()> {
= log::debug!(
@@ -240,7 +271,10 @@ impl ScenarioReport {
= // TODO: Handle parameters
= // TODO: If any step fails, fail the parent step
= // TODO: Consider infinite loops and stuff
- delegation.scenario.run(verbose)?;
+ delegation
+ .scenario
+ .with_parameters(&delegation.parameters)
+ .run(verbose)?;
= if delegation.scenario.is_all_ok().not() {
= step.status = StepStatus::Failed {
= reason: "Delegation failed".into(),
@@ -379,10 +413,12 @@ impl Display for Observation {
= }
=}
=
+type ScenarioParameters = BTreeMap<String, String>;
+
=#[derive(Debug, Serialize)]
=struct Delegation {
= scenario: ScenarioReport,
- parameters: HashMap<String, String>,
+ parameters: ScenarioParameters,
=}
=
=impl Display for Delegation {
@@ -410,6 +446,13 @@ impl Display for Delegation {
= title = scenario.title,
= )?;
=
+ if self.parameters.is_empty().not() {
+ let parameters = format!("Parameters: {:#?}", self.parameters)
+ .indent(4)
+ .dimmed();
+ writeln!(f, "{parameters}")?;
+ }
+
= // TODO: DRY with Evaluation report
= // Or even better refactor the display logic. It's a spaghetti!
= for (
@@ -578,14 +621,14 @@ impl Display for EvaluationReport {
= indentation = "".indent(4),
= )?;
=
- for delegation in delegations {
- writeln!(f, "{}", delegation.to_string().indent(0))?;
- }
-
= for observation in observations {
= writeln!(f, "\n{}", observation.to_string().indent(6))?;
= }
=
+ for delegation in delegations {
+ writeln!(f, "{}", delegation.to_string().indent(0))?;
+ }
+
= if let StepStatus::Failed { reason, hint } = status {
= writeln!(f, "\n{}\n", reason.indent(6).red())?;
= if let Some(hint) = hint {
@@ -653,7 +696,7 @@ pub enum InterpreterMessage {
= Delegate {
= suite: String,
= scenario: String,
- parameters: HashMap<String, String>,
+ parameters: ScenarioParameters,
= },
= Success,
= Failure {index cb29040..19ffb1b 100644
--- a/src/spec.rs
+++ b/src/spec.rs
@@ -45,6 +45,7 @@ pub struct Suite {
=pub struct Scenario {
= pub title: String,
= pub tags: BTreeSet<String>,
+ pub placeholders: BTreeMap<String, String>,
= pub steps: Vec<Step>,
= pub source: SourceInfromation,
= pub interpreter: String,
@@ -235,6 +236,8 @@ pub struct FrontMatter {
=pub struct Metadata {
= #[serde(default)]
= pub tags: BTreeSet<String>,
+ #[serde(default)]
+ pub parameters: BTreeMap<String, String>,
=}
=
=impl Suite {
@@ -340,6 +343,7 @@ impl Suite {
= scenarios.push(Scenario {
= title: node.to_string(),
= tags: BTreeSet::default(),
+ placeholders: BTreeMap::default(),
= steps: [].into(),
= source: SourceInfromation {
= document_path: document_path.clone(),
@@ -387,6 +391,8 @@ impl Suite {
=
= if let Some(scenario) = scenarios.last_mut() {
= scenario.tags.extend(metadata.tags);
+ // In Markdown it's called `parameters`, in Rust - `placeholders`. Is that good?
+ scenario.placeholders.extend(metadata.parameters);
= } else {
= suite_tags.extend(metadata.tags);
= }Emit an error on an undefined parameter
On by
An undefined parameter is one that is supplied by an interpreter during substitution, but has no corresponding placeholder in the delegated scenario. There is a sample demonstrating it.
index b117a7d..406dbef 100644
--- a/samples/delegations.md
+++ b/samples/delegations.md
@@ -32,3 +32,15 @@ This sample and related interpreter demonstrates how to use delegation feature o
= * Run the `Missing` scenario from the `samples/basic.md` suite
=
= This step should fail with an appropriate message.
+
+
+## Delegation with invalid parameters
+
+ * Run the `Count the Twists` scenario from the `samples/stateful.md` suite
+
+ | parameter | value |
+ |----------------------|-------|
+ | twist before the nap | 3 |
+ | twists after the nap | 2 |
+
+ Note that there is a typo in the first parameter. It should be plural "twists". This scenario will fail.index c681f07..2ba3118 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -106,35 +106,42 @@ pub struct ScenarioReport {
= steps: Vec<StepReport>,
=}
=impl ScenarioReport {
- fn with_parameters(&mut self, parameters: &ScenarioParameters) -> &mut Self {
- for (name, placeholder) in self.scenario.placeholders.iter() {
- if let Some(substitute) = parameters.get(name) {
- log::info!("Substituting `{}` with `{}`", placeholder, substitute);
- for report in self.steps.iter_mut() {
- report.step.arguments = report.step
- .arguments
- .iter()
- .map(|argument| {
- if argument == placeholder {
- log::debug!(
- "Substituting `{name}` from `{argument}` to `{substitute}` in `{}`",
- report.step.description
- );
- substitute
- } else {
- argument
- }
- })
- .cloned()
- .collect();
+ fn with_parameters(&mut self, parameters: &ScenarioParameters) -> anyhow::Result<&mut Self> {
+ // FIXME: Subsequent iterations override the previous substitutions
+ let known_parameters = self.scenario.placeholders.keys();
+
+ for (name, substitute) in parameters {
+ let placeholder = self.scenario.placeholders.get(name).context(formatdoc! {"
+ Undefined parameter `{name}`
+
+ valid parameter names are: {known_parameters:?}
+ "})?;
+ log::info!("Substituting `{}` with `{}`", placeholder, substitute);
+ for report in self.steps.iter_mut() {
+ report.step.arguments = report
+ .step
+ .arguments
+ .iter()
+ .map(|argument| {
+ if argument == placeholder {
+ log::debug!(
+ "Substituting `{name}` from `{argument}` to `{substitute}` in `{}`",
+ report.step.description
+ );
+ substitute
+ } else {
+ argument
+ }
+ })
+ .cloned()
+ .collect();
=
- log::info!("Step after substitutions: {:#?}", report.step);
- }
+ log::info!("Step after substitutions: {:#?}", report.step);
= }
= }
=
= log::info!("Scenario after substitutions: {self:#?}");
- self
+ Ok(self)
= }
=
= // TODO: The ScenarioReport::run method is very effectful. I think it should live in it's own module.
@@ -274,6 +281,7 @@ impl ScenarioReport {
= delegation
= .scenario
= .with_parameters(&delegation.parameters)
+ .context("Applying parameters")?
= .run(verbose)?;
= if delegation.scenario.is_all_ok().not() {
= step.status = StepStatus::Failed {Remove some stale TODO comments
On by
index 2ba3118..896e9c5 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -275,8 +275,6 @@ impl ScenarioReport {
= };
= }
= for delegation in step.delegations.iter_mut() {
- // TODO: Handle parameters
- // TODO: If any step fails, fail the parent step
= // TODO: Consider infinite loops and stuff
= delegation
= .scenarioTweak the parameters to demonstrate a bug
On by
The implementation of parameters application is faulty. It involves multiple passes, possibly overwriting previous changes. Setting the parameters like this demonstrates this:
First the twists after the nap value is changed from 4 to 2, which
accidentally matches placeholder value of twists before the nap. So in
the second pass it gets changed to 3, producing invalid results.
index 406dbef..43fb50f 100644
--- a/samples/delegations.md
+++ b/samples/delegations.md
@@ -14,8 +14,8 @@ This sample and related interpreter demonstrates how to use delegation feature o
=
= | parameter | value |
= |-----------------------|-------|
- | twists before the nap | 4 |
- | twists after the nap | 1 |
+ | twists before the nap | 3 |
+ | twists after the nap | 2 |
=
= Note that this scenario uses a different interpreter than the previous one. Also, because the parameters are adjusted, it won't fail, like the delegated scenario would with the default parameters. In this delegation the total number of twists doesn't exceed 6, so it's fine.
=Fix parameter overriding
On by
As described in the previous commit, there was a subtle bug. Now it's fixed, and the code actually got simpler, with less rightward drift.
Also on undefined parameters the delegation will fail with a helpful error message in the report.
index 896e9c5..e35581c 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -108,36 +108,34 @@ pub struct ScenarioReport {
=impl ScenarioReport {
= fn with_parameters(&mut self, parameters: &ScenarioParameters) -> anyhow::Result<&mut Self> {
= // FIXME: Subsequent iterations override the previous substitutions
+
+ // Check if all parameters have corresponding placeholders
= let known_parameters = self.scenario.placeholders.keys();
+ let mut substitutions = BTreeMap::default();
=
= for (name, substitute) in parameters {
- let placeholder = self.scenario.placeholders.get(name).context(formatdoc! {"
- Undefined parameter `{name}`
+ let placeholder = self
+ .scenario
+ .placeholders
+ .get(name)
+ .context(format!("Undefined parameter `{name}`"))
+ .context(format!("valid parameter names are: {known_parameters:?}"))?;
+
+ substitutions.insert(placeholder, substitute);
=
- valid parameter names are: {known_parameters:?}
- "})?;
= log::info!("Substituting `{}` with `{}`", placeholder, substitute);
- for report in self.steps.iter_mut() {
- report.step.arguments = report
- .step
- .arguments
- .iter()
- .map(|argument| {
- if argument == placeholder {
- log::debug!(
- "Substituting `{name}` from `{argument}` to `{substitute}` in `{}`",
- report.step.description
- );
- substitute
- } else {
- argument
- }
- })
- .cloned()
- .collect();
+ }
=
- log::info!("Step after substitutions: {:#?}", report.step);
- }
+ for report in self.steps.iter_mut() {
+ report.step.arguments = report
+ .step
+ .arguments
+ .iter()
+ .map(|argument| substitutions.get(argument).cloned().unwrap_or(argument))
+ .cloned()
+ .collect();
+
+ log::info!("Step after substitutions: {:#?}", report.step);
= }
=
= log::info!("Scenario after substitutions: {self:#?}");Make the step fail on a failed delegation
On by
If a step in a delgation fails, the failure should cascade to the parent step. The delegating scenario should not be further ran.
Also, remove a stale FIXME comment.
Also, lower the noise around parameters substitution.
index 43fb50f..cee9a2e 100644
--- a/samples/delegations.md
+++ b/samples/delegations.md
@@ -20,6 +20,26 @@ This sample and related interpreter demonstrates how to use delegation feature o
= Note that this scenario uses a different interpreter than the previous one. Also, because the parameters are adjusted, it won't fail, like the delegated scenario would with the default parameters. In this delegation the total number of twists doesn't exceed 6, so it's fine.
=
=
+## Delegation to a failing scenario
+
+``` yaml tbb
+tags: [ focus ]
+```
+
+
+ * Run the `Count the Twists` scenario from the `samples/stateful.md` suite
+
+ | parameter | value |
+ |-----------------------|-------|
+ | twists before the nap | 8 |
+ | twists after the nap | 2 |
+
+ Here the scenario will fail again.
+
+ * Run the `Arithmetic` scenario from the `samples/basic.md` suite
+
+ This next step should not be executed. The number of skipped steps should be 3 (two steps from the delegated scenario + this one).
+
=## Delegation to a non-existing suite
=
= * Run the `Arithmetic` scenario from the `samples/missing.md` suite
@@ -31,7 +51,7 @@ This sample and related interpreter demonstrates how to use delegation feature o
=
= * Run the `Missing` scenario from the `samples/basic.md` suite
=
- This step should fail with an appropriate message.
+ This step should also fail with an appropriate message.
=
=
=## Delegation with invalid parametersindex e35581c..1569dbc 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -107,8 +107,6 @@ pub struct ScenarioReport {
=}
=impl ScenarioReport {
= fn with_parameters(&mut self, parameters: &ScenarioParameters) -> anyhow::Result<&mut Self> {
- // FIXME: Subsequent iterations override the previous substitutions
-
= // Check if all parameters have corresponding placeholders
= let known_parameters = self.scenario.placeholders.keys();
= let mut substitutions = BTreeMap::default();
@@ -123,7 +121,7 @@ impl ScenarioReport {
=
= substitutions.insert(placeholder, substitute);
=
- log::info!("Substituting `{}` with `{}`", placeholder, substitute);
+ log::debug!("Substituting `{}` with `{}`", placeholder, substitute);
= }
=
= for report in self.steps.iter_mut() {
@@ -135,10 +133,10 @@ impl ScenarioReport {
= .cloned()
= .collect();
=
- log::info!("Step after substitutions: {:#?}", report.step);
+ log::debug!("Step after substitutions: {:#?}", report.step);
= }
=
- log::info!("Scenario after substitutions: {self:#?}");
+ log::debug!("Scenario after substitutions: {self:#?}");
= Ok(self)
= }
=
@@ -274,6 +272,8 @@ impl ScenarioReport {
= }
= for delegation in step.delegations.iter_mut() {
= // TODO: Consider infinite loops and stuff
+ // TODO: Mark the step as failed when with_parameters() fail
+ // TODO: Mark the step as failed when run() fail
= delegation
= .scenario
= .with_parameters(&delegation.parameters)
@@ -284,7 +284,8 @@ impl ScenarioReport {
= reason: "Delegation failed".into(),
= // TODO: Explain the delegation? Or maybe it should be in the delegation header, next to the telephone?
= hint: None,
- }
+ };
+ break 'steps_loop;
= }
= }
= }
@@ -429,7 +430,7 @@ impl Display for Delegation {
= fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
= let ScenarioReport {
= ref scenario,
- ref status,
+ status: ref _status,
= ref steps,
= } = self.scenario;
=Make the terminal reports better wrt delegations
On by
Always print observations before delegations.
Count delegated steps toward completed or remaining steps (with two helper methods on ScenarioReport struct).
The delegation block won't show the not-evaluated steps, leaving it to the root scenario to count and display them.
Also, drop some unused bindings.
index 1569dbc..a42e76b 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -356,6 +356,29 @@ impl ScenarioReport {
= };
= Ok(buffer)
= }
+
+ fn steps_count(&self) -> usize {
+ self.steps.iter().fold(0, |count, step| {
+ count
+ + step.delegations.iter().fold(1, |count, delegation| {
+ count + delegation.scenario.steps_count()
+ })
+ })
+ }
+
+ fn remaining_steps_count(&self) -> usize {
+ self.steps.iter().fold(0, |count, step| {
+ count
+ + (if step.status == StepStatus::NotEvaluated {
+ 1
+ } else {
+ 0
+ })
+ + step.delegations.iter().fold(0, |count, delegation| {
+ count + delegation.scenario.remaining_steps_count()
+ })
+ })
+ }
=}
=
=#[derive(Debug, Serialize)]
@@ -460,15 +483,12 @@ impl Display for Delegation {
=
= // TODO: DRY with Evaluation report
= // Or even better refactor the display logic. It's a spaghetti!
- for (
- index,
- StepReport {
- step,
- status,
- observations,
- delegations,
- },
- ) in steps.iter().enumerate()
+ for StepReport {
+ step,
+ status,
+ observations,
+ delegations,
+ } in steps.iter()
= {
= let sigil = match status {
= StepStatus::Ok => "⊞".to_string().bold().green(),
@@ -483,19 +503,23 @@ impl Display for Delegation {
= indentation = "".indent(4),
= )?;
=
- for delegation in delegations {
- writeln!(f, "{}", delegation.to_string().indent(0))?;
- }
-
= for observation in observations {
= writeln!(f, "\n{}", observation.to_string().indent(6))?;
= }
=
+ for delegation in delegations {
+ writeln!(f, "{}", delegation.to_string().indent(0))?;
+ }
+
= if let StepStatus::Failed { reason, hint } = status {
= writeln!(f, "\n{}\n", reason.indent(6).red())?;
= if let Some(hint) = hint {
= writeln!(f, "{}", hint.as_str().indent(6))?;
= }
+
+ if !f.alternate() {
+ break;
+ }
= }
= }
= Ok(())
@@ -580,23 +604,21 @@ impl Display for EvaluationReport {
= )?
= } else if sr.is_all_ok() && steps.len() > 1 && f.alternate().not() {
= // Condense the output when all steps are successful
- let sigils = "⊞ ".repeat(steps.len()).trim().indent(4).green();
- let message = format!("⬑ all {count} steps successful.", count = steps.len())
+ let steps_count = sr.steps_count();
+ let sigils = "⊞ ".repeat(steps_count).trim().indent(4).green();
+ let message = format!("⬑ all {steps_count} steps successful.")
= .indent(4)
= .dimmed()
= .bold();
= writeln!(f, "{sigils}")?;
= writeln!(f, "{message}")?;
= } else {
- for (
- index,
- StepReport {
- step,
- status,
- observations,
- delegations,
- },
- ) in steps.iter().enumerate()
+ for StepReport {
+ step,
+ status,
+ observations,
+ delegations,
+ } in steps.iter()
= {
= let sigil = match status {
= StepStatus::Ok => "⊞".to_string().bold().green(),
@@ -605,7 +627,7 @@ impl Display for EvaluationReport {
= };
=
= // TODO: Take delegations into account
- let remaining_count = steps.len() - index;
+ let remaining_count = sr.remaining_steps_count();
= if *status == StepStatus::NotEvaluated && remaining_count > 1 {
= // Condense the output when all steps are successful
= let sigils = "□ ".repeat(remaining_count).trim().indent(4);Print short statistics at the bottom of the report
On by
Also a big, green "All good" message if all is indeed good.
Also, remove the focus tag from delegations scenario. I was using it
for hand-evaluation, but it shouldn't be committed.
index cee9a2e..faab921 100644
--- a/samples/delegations.md
+++ b/samples/delegations.md
@@ -22,9 +22,6 @@ This sample and related interpreter demonstrates how to use delegation feature o
=
=## Delegation to a failing scenario
=
-``` yaml tbb
-tags: [ focus ]
-```
=
=
= * Run the `Count the Twists` scenario from the `samples/stateful.md` suiteindex acf281b..5c1a31b 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -4,6 +4,7 @@ mod spec;
=
=use anyhow::{Context, Error, bail};
=use clap::{Parser, Subcommand, ValueEnum};
+use colored::Colorize;
=use env_logger;
=use glob::glob;
=use log;
@@ -198,7 +199,10 @@ fn evaluate(
= }
=
= match EvaluationSummary::from(report) {
- EvaluationSummary::AllOk => Ok(()),
+ EvaluationSummary::AllOk => {
+ println!("{}", format!("All good!").green().bold());
+ Ok(())
+ }
= EvaluationSummary::NothingToEvaluate => {
= log::error!("Nothing to evaluate");
= process::exit(1)index a42e76b..7498739 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -669,6 +669,20 @@ impl Display for EvaluationReport {
= writeln!(f, "")?;
= }
=
+ writeln!(
+ f,
+ "Evaluated {suites} suite(s), ran {scenarios} scenario(s), and executed {steps} step(s).",
+ suites = self.suites.len(),
+ scenarios = self
+ .suites
+ .iter()
+ .fold(0, |count, suite| count + suite.scenarios.len()),
+ steps = self.suites.iter().fold(0, |count, suite| count
+ + suite
+ .scenarios
+ .iter()
+ .fold(0, |count, scenario| count + scenario.steps_count()))
+ )?;
= Ok(())
= }
=}Update the delegations document
On by
It's still just the prose (no executable steps), but now it reflects
latest design, and to an extend the current implementation. There is the
sample/delegations.md suite that to a large extent covers the spec,
but it's not automatically evaluated yet - only manually with
$ tbb evaluate samples/delegations.mdBefore I get to automate the evaluation I want to fix other parts of the spec that got violated by the work on delegations. And maybe work with the delegations in a real project, to get more confidence in the design.
index 90b13ec..1dba7f2 100644
--- a/spec/delegations-and-fixtures.md
+++ b/spec/delegations-and-fixtures.md
@@ -2,16 +2,20 @@
=interpreter: "python -m spec.self-check"
=---
=
-# Delegation and fixtures
+# Delegation and Fixtures
=
-In response to the "execute step" command, interpreters can request to run one or more scenarios. This is a feature sometimes known as "fixtures", but it has broader application. Any scenario can be re-used this way, but scenarios with `fixture: true` property are not run, unless required by an interpreter.
+Scenarios can be composed.
+
+In response to the "execute step" command, interpreters can request to run one or more scenarios. This is a feature sometimes known as "fixtures", but it has broader application. Any scenario can be reused this way, but scenarios with `fixture: true` property are not run, unless required by an interpreter.
=
=Scenarios can have parameters, defined as follows:
=
=``` markdown
= \``` yaml tbb
= parameters:
- name: Placeholder
+ project name: Project alpha
+ path: alpha/
+ maintainer: Alice
= \```
=```
=
@@ -20,13 +24,12 @@ When an interpreter requests to run another scenario as a fixture, it can pass v
=``` json
={
= "parameters": {
- "name": "Actual value"
+ "project name": "Project beta",
+ "path": "beta/"
= }
=}
=```
=
-When running the requested scenario, TBB will substitute every argument with the value `"Placeholder"` with the `"Actual value"`. Notice that the substitution is done by value, not by name. The names are given to values to create a substitution map. This way we avoid any special syntax for fixtures. Reusable scenarios are written with default (or placeholder) values, and re-used with substitutions. We maintain the natural, example driven manner of writing specs.
-
=A step that would request another scenario would look something like this:
=
=``` markdown
@@ -38,19 +41,24 @@ A step that would request another scenario would look something like this:
= | path | beta/ |
=```
=
+When running the requested scenario, TBB will substitute every argument with the value `"Project alpha"` with `"Project beta"`, and `"alpha/"` with `"beta/"`. Notice that the substitution is done by value, not by name. The names are given to values to create a substitution map. This way we avoid any special syntax for fixtures. Reusable scenarios are written with default (or placeholder) values, and reused with substitutions. We maintain the natural, example driven manner of writing specs. In this case the `maintainer` parameter will retain its original (placeholder) value (Alice).
=
-The delegation request is a terminating message of the step, just like `Ok` and `Failure`. While awaiting a delegation to complete, the delegating interpreter process is kept alive, but no messages should be passed until the next step. Once the delegation is successfully completed, the control process will send the next step and the interpreter will resume. If a delegation fail, the interpreter will be terminated, as if one of it's own steps failed.
+While awaiting a delegation to complete, the delegating interpreter process is kept alive, but no messages should be passed until the next step. This is consistent with the expected behavior of an interpreter - after completing a step it should stay silent until the next `execute` command comes in. Once the delegation is successfully completed, the control process will send the next step and the interpreter will resume. If a delegation fail, the interpreter will be terminated, as if one of its own steps failed.
=
-In the report, the delegating step should be immediately followed by the requested steps, as if they were integral part of a single scenario. The delegating step is marked with the telephone sigil (☎). The block should list all the requested scenarios, with their result sigils, source position and interpreters (so combining parts of the suite block and parts of the scenario block). The delegation block should be appropriately indented, not to be confused with a top-level scenario. But the delegate steps should not be indented to avoid rightward drift. Conceptually they are part of the delegating scenario.
+A step can delegate work to multiple scenarios, by sending multiple messages `Delegate` messages. Delegated scenarios will run only after the step is successfully completed, in the order of requests. If the delegating step itself failed (after sending the delegations), the delegated scenarios should not be run.
=
-The delegate scenarios are executed in a separate process, using the interpreter specified in their suite, just like any other scenarios. This allows to use specialized interpreters to run fixtures, other than the delegating interpreter. Another effect of this is that they don't automatically share any state. If interpreters need to pass any data other than parameters, they need to implement an appropriate mechanism (like pre-arranged file, named pipe, a database, etc.).
+In the terminal report, the delegating step should be immediately followed by the requested steps, as if they were integral part of a single scenario. The "collapsing" system should apply to them, so if a step delegated to two scenarios, with 3 and 5 steps each, it should produce 9 green sigils: one for the delegating step and 8 for the delegates.
=
-A step can delegate work to multiple scenarios. In that case, they should run consecutively, in order of requests.
+If the steps are not collapsed, e.g. there was a failure, then each delegation is marked as with the telephone icon (☎), immediately followed by the title of the step, and source position. After the "telephone line", the delegated steps should be printed the usual way, without any extra indentation (to avoid rightward drift). Conceptually they are part of the delegating scenario.
+
+The delegate scenarios are executed in a separate process, using the interpreter specified in their suite, just like any other scenarios. This allows spec authors to delegate work to a specialized interpreters (i.e. other than the delegating interpreter). Another effect of this is that they don't automatically share any state. If interpreters need to pass any data other than parameters, they need to implement an appropriate mechanism (like prearranged file, named pipe, a database, etc.).
=
=A delegate can request another delegation. This way the process can be recursive. Failure will result in termination of a whole stack of interpreters. The report should explain the situation, including the stack of requests.
=
=> Should there be limits to the recursion?
=
-> What if a requested scenario is missing? Or there is a typo?
+If the delegation requests a scenario from a non-existing suite, the delegating step is immediately terminated with an appropriate error message. The step is considered failed.
+
+Similarly, if the delegation requests a scenario from an existing suite, but the scenario is not defined there. In that case the error message should mention available scenarios in the referenced suite.
=
-> What if the parameters don't match?
+If the parameter name doesn't match any of the parameters defined in the scenario, the step should be terminated as failed, and the error should explain the problem and list the available parameters.Align the program and spec
On by
The implementation passes the tests again.
In a few places the spec had to be updated, because the source positions changed (some lines were added in samples).
The spec/delegations.md is now tagged as not-implemented, because most
of the scenarios fail. Without it, the spec is picked up by the
filtering suite and is evaluated with an expectations to succeed. This
is a recurring problem. Adding a new sample often breaks the filtering.
It's brittle, and I think I'll need to restructure the samples
directory to avoid this kind of thing.
To detect it, I used the relatively recent observations API in the
self-check.py interpreter, namely to print stdout and stdin. While
at it, I also made it observe the complete command that is used to
produce the output. And thanks to this I could remove the noisy warning
that was polluting the output. I really like the observations feature.
Finally, a new error occurred around the backtick escaping (introduced a few days ago). I'm surprised it wasn't causing problems before. Anyway, it's fixed now.
index faab921..e619d1a 100644
--- a/samples/delegations.md
+++ b/samples/delegations.md
@@ -1,5 +1,6 @@
=---
=interpreter: "python -m samples.delegating"
+tags: [not-implemented]
=---
=
=# Let's delegate some workindex 24fa596..c3532e2 100644
--- a/spec/nushell-library.md
+++ b/spec/nushell-library.md
@@ -26,7 +26,7 @@ There is a library for implementing interpreters in [Nushell](https://www.nushel
= ```text
= x Count the Twists
= interpreter: nu --stdin samples/basic.nu
- source: samples/stateful.md:14-30
+ source: samples/stateful.md:14-36
= ```
=
= * The output will contain `an expanded successful step with a snippet observation` block
@@ -36,7 +36,7 @@ There is a library for implementing interpreters in [Nushell](https://www.nushel
= ``` text
= ⊞ Do the twist 2 times
= arguments: ["2"]
- source: samples/stateful.md:16-20
+ source: samples/stateful.md:22-26
=
= State before the step:
=
@@ -64,7 +64,7 @@ There is a library for implementing interpreters in [Nushell](https://www.nushel
= ``` text
= ⊞ Do something nice and then sleep for 2 hours
= arguments: ["nice", "sleep", "2"], code blocks: 1
- source: samples/stateful.md:21-26
+ source: samples/stateful.md:27-32
=
= State update to be merged:
=
@@ -93,7 +93,7 @@ There is a library for implementing interpreters in [Nushell](https://www.nushel
= ``` text
= ⊠ Do the twist 4 times
= arguments: ["4"]
- source: samples/stateful.md:27-29
+ source: samples/stateful.md:33-35
=
= The value returned by the step is not a record; Not updating:
=index 8a417a4..325e6c1 100644
--- a/spec/self-check.py
+++ b/spec/self-check.py
@@ -14,7 +14,6 @@ from lib.tbb import step, log, indent_tail, get_at
=base_command = "tbb"
=if os.environ.get ("CARGO"):
= cargo_command = "cargo run --"
- log.warning(f"Running inside Cargo. Will use `{cargo_command}` instead of `{base_command}`.")
= base_command = cargo_command
=
=tester = unittest.TestCase()
@@ -28,7 +27,13 @@ def step_implementation_00(args: str, **kwargs):
= global completed
=
= command = shlex.split (base_command) + shlex.split (args)
- completed = subprocess.run(command , stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ tbb.send_snippet("shell", str.join(" ", command), caption="Command to execute")
+
+ completed = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+ tbb.send_snippet("text", completed.stdout.decode("utf8"), caption="Standard output")
+ tbb.send_snippet("text", completed.stderr.decode("utf8"), caption="Standard error")
+
=
=@step("The exit code should be {0}")
=def step_implementation_01(expected_code: int, **kwargs):
@@ -91,7 +96,7 @@ def step_implementation_04(label: str, **kwargs):
= # Trim all blank lines and trailing whitespece. Without it the assertions
= # are very brittle. Also un-escape back-ticks in the block. This is useful
= # for nested code-blocks in the spec documents.
- needle = re.sub("\\s+\n", "\n", block).replace("\`", "`")
+ needle = re.sub("\\s+\n", "\n", block).replace("\\`", "`")
= haystack = re.sub("\\s+\n", "\n", output)
= # tester.assertIn gives unreadable output
=index 7498739..65cf0f8 100644
--- a/src/report.rs
+++ b/src/report.rs
@@ -669,20 +669,22 @@ impl Display for EvaluationReport {
= writeln!(f, "")?;
= }
=
- writeln!(
- f,
- "Evaluated {suites} suite(s), ran {scenarios} scenario(s), and executed {steps} step(s).",
- suites = self.suites.len(),
- scenarios = self
- .suites
- .iter()
- .fold(0, |count, suite| count + suite.scenarios.len()),
- steps = self.suites.iter().fold(0, |count, suite| count
- + suite
- .scenarios
+ if self.suites.len() > 0 {
+ writeln!(
+ f,
+ "Evaluated {suites} suite(s), ran {scenarios} scenario(s), and executed {steps} step(s).",
+ suites = self.suites.len(),
+ scenarios = self
+ .suites
= .iter()
- .fold(0, |count, scenario| count + scenario.steps_count()))
- )?;
+ .fold(0, |count, suite| count + suite.scenarios.len()),
+ steps = self.suites.iter().fold(0, |count, suite| count
+ + suite
+ .scenarios
+ .iter()
+ .fold(0, |count, scenario| count + scenario.steps_count()))
+ )?;
+ }
= Ok(())
= }
=}Don't make steps from numbered or commented items
On by
As described in the new spec.
This is important, because sometimes when writing prose (e.g. during a client meeting) we want to use bullet points simply to enumerate something. We need a simple syntax to tell TBB that these bullets are not meant to be interpreted as steps.
Regarding the ordered lists, I'm considering if it should be the other way around: only the numbered items should be interpreted as steps. It looks much more intuitive to have
- Do something
- Do something else
That would be a big breaking change, but maybe it's worth it? Updating all the specs shouldn't be that hard.
index b5b548c..1c5a132 100644
--- a/samples/basic.md
+++ b/samples/basic.md
@@ -40,8 +40,6 @@ tags:
= - work-in-progress
=```
=
-This
-
= * The word `blocks` has `6` characters
= * There are `3` properties in the following JSON
=index cf6b5c7..4e34cc9 100644
--- a/spec/basic-usage.md
+++ b/spec/basic-usage.md
@@ -345,3 +345,46 @@ When a directory is given as the last argument, load all documents inside (recur
=## Running without a subcommand
=
=Running `tbb` without a subcommand will print the help message (to `stderr`) and exit with code 2.
+
+
+## Lists that don't make steps
+
+``` yaml tbb
+tags: [ focus ]
+```
+
+If you want to use a list under the `h2` heading for other purposes than defining a step, you have two options:
+
+ 1. Use ordered lists (starting with a number followed by a period)
+
+ as demonstrated here. This list item won't be interpreted as a step.
+
+ 2. Start the list item with a double slash sequence
+
+ as demonstrated below.
+
+
+Here is a list with three items, but only two of them make steps. The middle one will be ignored.
+
+
+ * Do nothing
+
+ Yes, it's a step :)
+
+ * // This will be ignored
+
+ You can write any content under this bullet point.
+
+ * Do nothing
+
+ again.
+
+
+This also work in a condensed lists, like this:
+
+ * Do nothing
+ * // This will be ignored too
+ * Do nothing
+
+
+Notice that this scenario is self-evaluating. Clever, huh?index 325e6c1..9bcf43b 100644
--- a/spec/self-check.py
+++ b/spec/self-check.py
@@ -246,5 +246,10 @@ def step_implementation_20(from_name: str, json_path: str, json_value: str, name
=
= state[name] = matching[0]
=
+@step("Do nothing")
+def step_implementation_21(**kwargs):
+ return # Literally nothing
+
+
=if __name__ == "__main__":
= tbb.ready()index 19ffb1b..e9f1890 100644
--- a/src/spec.rs
+++ b/src/spec.rs
@@ -356,10 +356,27 @@ impl Suite {
= }
= }
= markdown::mdast::Node::List(list) => {
- if let Some(scenario) = scenarios.last_mut() {
+ if let Some(scenario) = scenarios.last_mut()
+ && list.ordered.not()
+ {
= let items = list.children.iter().filter_map(|child| {
= if let markdown::mdast::Node::ListItem(item) = child {
- Some(item)
+ // List items that start with `//` need to be ignored
+ let is_a_comment: bool = (|| {
+ let Some(markdown::mdast::Node::Paragraph(paragraph)) =
+ item.children.first()
+ else {
+ return false;
+ };
+ let Some(markdown::mdast::Node::Text(text)) =
+ paragraph.children.first()
+ else {
+ return false;
+ };
+
+ text.value.starts_with("//")
+ })();
+ if is_a_comment { None } else { Some(item) }
= } else {
= None
= }