Commits: 7

Refine the spec regarding ignore

The previous aproach with ignore field under the project is not very ergonomic, as it requires every ignored path to have it's own section and a name. In my ~/Projects/ directory I have hundreds of little experiments with one or two commits each and I don't want to give them all names just to avoid excavating them.

So now in the config there is the [ignore] section with paths - an array of strings. Easy to maintain.

index 9a8d531..eb670a5 100644
--- a/devlog-sample.toml
+++ b/devlog-sample.toml
@@ -5,6 +5,7 @@
=# Zola (https://www.getzola.org/documentation/content/taxonomies/).
=projects_cell_path = [ "taxonomies", "projects" ]
=
+# Each project you want to excavate should have it's own section, like this:
=[[projects]]
=name = "Tad Better Behavior"
=path = "~/Projects/tad-better-behavior"
@@ -16,3 +17,10 @@ path = "~/Projects/better-tech-club/bettertech.eu"
=[[projects]]
=name = "DevLog Extracavator"
=path = "."
+
+[ignore]
+# List all the paths to ignore when excavating
+paths = [
+    "~/Projects/manhattan",
+    "~/Projects/for-your-eyes-only",
+]

index 658718a..7a7364f 100644
--- a/spec/BASIC.md
+++ b/spec/BASIC.md
@@ -214,7 +214,17 @@ This seems to be specific to `skip`. Other commands fail as expected.
=
=By default `./devlog.toml` shall be used, but it can be changed with `--config-file` flag. The file must list all the projects for which devlog is to be generated. The name of the project comes from there. See the devlog-sample.toml file.
=
-A project can be flagged to be ignored with the `ignore` key.
+Any path under "ignore.paths" will be ignored, i.e. no entries will be excavated from it and there will be no warning.
+
+``` toml
+[ignore]
+paths = [
+    "~/Projects/manhattan",
+    "~/Projects/for-your-eyes-only",
+]
+```
+
+In the future there might be more things to ignore, like authors, files in patches (globs), etc.
=
-Any project that is not listed, but had activity within the period for which a devlog is being generated should produce a warning with useful instructions.
+Any project that is not listed and not ignored, but had activity within the period for which a devlog is being generated, will produce a warning with useful instructions.
=

Implement the ignore feature

As described in the spec.

index 8a37e16..da4d957 100755
--- a/excavate
+++ b/excavate
@@ -16,6 +16,7 @@ export def main [
=    open $config_file
=    | update projects { update path { path expand } } # Resolve all project paths
=    | upsert frontmatter.projects_cell_path { default { [ extra projects ] } | into cell-path }
+    | upsert ignore.paths { default [] | each { path expand } }
=    | let config
=
=    $until | default (date now) | let until
@@ -48,7 +49,7 @@ export def main [
=
=    # The main pipeline:
=    $projects_dir
-    | project list
+    | project list --ignore $config.ignore.paths
=    | each $extract_log
=    | flatten
=    | update time { |commit| $commit.time | into datetime }
@@ -103,11 +104,15 @@ export def main [
=    }
=}
=
-def "project list" []: path -> list<path> {
+def "project list" [
+    --ignore: list<path> = [] # Paths to ignore
+]: path -> list<path> {
=    $in
+    | path expand
=    | path join "**/.git"
=    | glob $in
=    | path dirname
+    | where { |project_path| not ($ignore | any { |ignored_path| $ignored_path in $project_path }) }
=}
=
=def "project log" [

Export project sub-commands: list and log

I finally figured out how to seamlessly export sub-commands both from a Nushell module and a script.

Excavator can be run in two ways. As a Nushell command:

use ./excavate
excavate --dry-run

and as a stand-alone program (in any other shell, like Bash):

$ ./excavate --dry-run

I want to have a as much feature parity between the two as possible. One thing I was struggling with until today were sub-commands. For a script they need to be prefixed with main, like main project list. For the module, they need to be exported.

I tried something like export def "main project list [...] {}" but then when imported I have to call them like this:

excavate main project list

Not elegant at all! But now, thanks to my yesterday's fruitless research on aliasing ls, I've found a trick. Name the commands in a natural way (without the main prefix) and then alias them with the prefix. Nice!

One more thing to remember is enabling stdin when invoking the nu interpreter. In a script this is best done in the shebang line. Just don't forget to add the -S or --split-string.

NOTE: The portability of this solution (e.g. with BSD and OSX) is questionable. Maybe we need to do something smarted here? I need to investigate it.

index da4d957..ae9964a 100755
--- a/excavate
+++ b/excavate
@@ -1,4 +1,4 @@
-#!/usr/bin/env -S running_as_a_script=true nu
+#!/usr/bin/env -S running_as_a_script=true nu --stdin
=
=use std/log
=
@@ -104,7 +104,8 @@ export def main [
=    }
=}
=
-def "project list" [
+# List all projects under a given path
+export def "project list" [
=    --ignore: list<path> = [] # Paths to ignore
=]: path -> list<path> {
=    $in
@@ -112,10 +113,13 @@ def "project list" [
=    | path join "**/.git"
=    | glob $in
=    | path dirname
-    | where { |project_path| not ($ignore | any { |ignored_path| $ignored_path in $project_path }) }
+    | where { |project_path| not ($ignore | any { path expand | $in in $project_path }) }
=}
=
-def "project log" [
+# Excavate a devlog from a single project
+# 
+# NOTE: This command does not write to any files, only lists commits.
+export def "project log" [
=    since: datetime,
=    until: datetime,
=]: path -> list {
@@ -161,6 +165,8 @@ def "project log" [
=    | insert path $project_path
=}
=
+alias "main project log" = project log
+alias "main project list" = project list
=
=def "format entry" [
=    title: string

Implement the stub projects sub-command

When starting a devlog (like I do now) it's useful to write out markdown document for each project.

Stubbing is driven by the config file, which needs to be normalized before it can be processed. This logic is shared with the main command, so now there is a normalize config helper just for that.

By default stubs are written to the ./projects/ directory, so it shouldn't be checked into version control. The name of this directory seems a bit too generic. I might change it later.

index 74bb444..e91585b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,4 +1,5 @@
=/drafts/
+/projects/
=/devlog.toml
=.direnv
=.devenv

index ae9964a..b4e2059 100755
--- a/excavate
+++ b/excavate
@@ -14,9 +14,7 @@ export def main [
=    # Read the config file, process and set some defaults
=    # TODO: DRY with `main list`
=    open $config_file
-    | update projects { update path { path expand } } # Resolve all project paths
-    | upsert frontmatter.projects_cell_path { default { [ extra projects ] } | into cell-path }
-    | upsert ignore.paths { default [] | each { path expand } }
+    | normalize config
=    | let config
=
=    $until | default (date now) | let until
@@ -104,6 +102,44 @@ export def main [
=    }
=}
=
+export def "stub projects" [
+    --config-file: path = "./devlog.toml"
+] {
+    open $config_file
+    | normalize config
+    | let config
+
+    mkdir $config.projects_stub_dir
+
+    for project in $config.projects {
+        $project.name
+        | str kebab-case
+        | {
+            parent: $config.projects_stub_dir,
+            stem: $in,
+            extension: "md"
+        }
+        | path join
+        | let out_path
+
+
+        {
+            title: $project.name
+        }
+        | to yaml
+        | str trim
+        | [
+            $"---"
+            $in
+            $"---"
+            $""
+            $"# ($project.name)"
+        ]
+        | str join "\n"
+        | save --force $out_path
+    }
+}
+
=# List all projects under a given path
=export def "project list" [
=    --ignore: list<path> = [] # Paths to ignore
@@ -167,6 +203,7 @@ export def "project log" [
=
=alias "main project log" = project log
=alias "main project list" = project list
+alias "main stub projects" = stub projects
=
=def "format entry" [
=    title: string
@@ -214,3 +251,12 @@ def "format commit" [] {
=def "format diff" [] {
=    $"``` diff\n($in | str trim)\n```"
=}
+
+# Standardize some elements of the config so we don't have to worry about it later
+def "normalize config" []: record -> record {
+    $in
+    | update projects { update path { path expand } } # Resolve all project paths
+    | upsert frontmatter.projects_cell_path { default { [ extra projects ] } | into cell-path }
+    | upsert ignore.paths { default [] | each { path expand } }
+    | upsert projects_stub_dir { default "./projects" | path expand }
+}

index 7a7364f..8d136da 100644
--- a/spec/BASIC.md
+++ b/spec/BASIC.md
@@ -228,3 +228,6 @@ In the future there might be more things to ignore, like authors, files in patch
=
=Any project that is not listed and not ignored, but had activity within the period for which a devlog is being generated, will produce a warning with useful instructions.
=
+## Stubbing projects 
+
+Use the `excavate stub projects` command to write a markdown file for each project listed in the configuration file. The output path is controlled by the `projects_stubs_dir` key.

Let the program die on broken projects

It will log each failure, including stderr from git status and then return an error listing each failed project.

index b4e2059..5b1c6b0 100755
--- a/excavate
+++ b/excavate
@@ -19,6 +19,36 @@ export def main [
=
=    $until | default (date now) | let until
=
+    # Check if all listed projects have proper git repositories
+    $config.projects
+    | insert status {
+        cd $in.path
+        $in.path
+        | path join ".git"
+        | git --git-dir $in status --porcelain
+        | complete
+    }
+    | each {
+        if ($in | get status.stdout | is-not-empty) {
+            log warning $"Repository not clean at ($in.name) ($in.path)"
+            $in
+        } else {
+            $in
+        }
+    }
+    | where { $in.status.exit_code != 0 }
+    | let broken_projects
+    | each {
+        log error $"Project ($in.name) at ($in.path) seems broken:\n($in.status.stderr)"
+    }
+
+    if ($broken_projects | is-not-empty) {
+        $broken_projects
+        | each { $"  ($in.name) \(($in.path)\)" }
+        | str join "\n"
+        | error make --unspanned $"Some projects are broken. See errors logged above.\n\n($in)"
+    }
+
=    let commit_day = { |commit|
=        $commit
=        | get time

index 8d136da..39a6a0c 100644
--- a/spec/BASIC.md
+++ b/spec/BASIC.md
@@ -181,6 +181,11 @@ If some work was done on such a project, there should be a warning about it.
=
=If a project is then added to the list, running the command again should not override existing drafts, but create new files for the newly listed projects.
=
+
+## Broken projects
+
+Projects listed in the configuration file, with path not pointing to the **root of a git** repository will result in program terminating early with an error.
+
=## Binary files issue
=
=A non-unicode fragment in the diff breaks my script. For example in a repo with some PDFs checked in I'm getting this:

Let project stubs and devlog go to ./drafts/

By default excavate stub projects will write to drafts/projects (this can be changed in the configuration file) and the main excavate command will write to ./drafts/devlog (this can be changed using the --out-dir cli flag). That way there is only one directory to manage, and also it's more obvious that these are all drafts.

index e91585b..74bb444 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,4 @@
=/drafts/
-/projects/
=/devlog.toml
=.direnv
=.devenv

index 5b1c6b0..8ec75ef 100755
--- a/excavate
+++ b/excavate
@@ -7,7 +7,7 @@ export def main [
=    --until: datetime                       # When did you last committed anything good?
=    --projects-dir: path = "~/Projects",    # Where do you keep your projects?
=    --day-start: string = "04:00"           # Until how late can you code?
-    --out-dir: path = "drafts"              # Where to write excavated posts
+    --out-dir: path = "drafts/devlog"       # Where to write excavated posts
=    --config-file: path = "devlog.toml"     # The configuration file
=    --dry-run                               # Just list the commits to excavate
=] {
@@ -288,5 +288,5 @@ def "normalize config" []: record -> record {
=    | update projects { update path { path expand } } # Resolve all project paths
=    | upsert frontmatter.projects_cell_path { default { [ extra projects ] } | into cell-path }
=    | upsert ignore.paths { default [] | each { path expand } }
-    | upsert projects_stub_dir { default "./projects" | path expand }
+    | upsert projects_stub_dir { default "./drafts/projects" | path expand }
=}

More configuration options

Directory paths for

• projects,
• drafts,
• and project stubs

are now configurable.

Also, the default configuration file path is now a constant (no more repeated strings).

One stale TODO comment removed.

index eb670a5..bcc83b7 100644
--- a/devlog-sample.toml
+++ b/devlog-sample.toml
@@ -1,3 +1,6 @@
+drafts_dir = "content/devlog"
+projects_stubs_dir = "content/works/"
+
=[frontmatter]
=
=# By default projects are listed under extra.projects.  This example shows how

index 8ec75ef..fe9310f 100755
--- a/excavate
+++ b/excavate
@@ -2,22 +2,25 @@
=
=use std/log
=
+const default_config_file = "devlog.toml"
+
=export def main [
-    --since: datetime = 1970-01-01          # How far back should we look?
-    --until: datetime                       # When did you last committed anything good?
-    --projects-dir: path = "~/Projects",    # Where do you keep your projects?
-    --day-start: string = "04:00"           # Until how late can you code?
-    --out-dir: path = "drafts/devlog"       # Where to write excavated posts
-    --config-file: path = "devlog.toml"     # The configuration file
-    --dry-run                               # Just list the commits to excavate
+    --since: datetime = 1970-01-01              # How far back should we look?
+    --until: datetime                           # When did you last committed anything good?
+    --day-start: string = "04:00"               # Until how late can you code?
+    --config-file: path = $default_config_file  # The configuration file
+    --out-dir: path                             # Where to write excavated posts (default: $config.drafts_dir)
+    --projects-dir: path                        # Where do you keep your projects (default: $config.projects_dir)
+    --dry-run                                   # Just list the commits to excavate
=] {
=    # Read the config file, process and set some defaults
-    # TODO: DRY with `main list`
=    open $config_file
=    | normalize config
=    | let config
=
=    $until | default (date now) | let until
+    $out_dir | default $config.drafts_dir | let out_dir
+    $projects_dir | default $config.projects_dir | let projects_dir
=
=    # Check if all listed projects have proper git repositories
=    $config.projects
@@ -133,19 +136,22 @@ export def main [
=}
=
=export def "stub projects" [
-    --config-file: path = "./devlog.toml"
+    --config-file: path = $default_config_file
+    --out-dir: path                             # Where to write the stubs (default: $config.projects_stubs_dir)
=] {
=    open $config_file
=    | normalize config
=    | let config
=
-    mkdir $config.projects_stub_dir
+    $out_dir | default $config.projects_stubs_dir | let out_dir
+
+    mkdir $out_dir
=
=    for project in $config.projects {
=        $project.name
=        | str kebab-case
=        | {
-            parent: $config.projects_stub_dir,
+            parent: $out_dir,
=            stem: $in,
=            extension: "md"
=        }
@@ -288,5 +294,7 @@ def "normalize config" []: record -> record {
=    | update projects { update path { path expand } } # Resolve all project paths
=    | upsert frontmatter.projects_cell_path { default { [ extra projects ] } | into cell-path }
=    | upsert ignore.paths { default [] | each { path expand } }
-    | upsert projects_stub_dir { default "./drafts/projects" | path expand }
+    | upsert projects_dir { default "~/Projects" | path expand }
+    | upsert projects_stubs_dir { default "./drafts/projects" | path expand }
+    | upsert drafts_dir { default "./drafts/devlog" | path expand }
=}

index 39a6a0c..5025af6 100644
--- a/spec/BASIC.md
+++ b/spec/BASIC.md
@@ -181,10 +181,11 @@ If some work was done on such a project, there should be a warning about it.
=
=If a project is then added to the list, running the command again should not override existing drafts, but create new files for the newly listed projects.
=
+Any project that is not listed and not ignored, but had activity within the period for which a devlog is being generated, will produce a warning with useful instructions.
=
=## Broken projects
=
-Projects listed in the configuration file, with path not pointing to the **root of a git** repository will result in program terminating early with an error.
+Projects listed in the configuration file, with path not pointing to the **root of a git** repository will result in program terminating early with an error. All broken projects should be listed in the error output.
=
=## Binary files issue
=
@@ -217,9 +218,14 @@ This seems to be specific to `skip`. Other commands fail as expected.
=
=## Configuration file
=
-By default `./devlog.toml` shall be used, but it can be changed with `--config-file` flag. The file must list all the projects for which devlog is to be generated. The name of the project comes from there. See the devlog-sample.toml file.
+By default `./devlog.toml` shall be used, but it can be changed with `--config-file` flag. The file must list all the projects for which devlog is to be generated. The name of each project comes from there.
+
+In addition it can specify:
=
-Any path under "ignore.paths" will be ignored, i.e. no entries will be excavated from it and there will be no warning.
+  - `drafts_dir`: where devlog entries are written (default: `./drafts/devlog`)
+  - `projects_stubs_dir`: where projects' are written by `excavate stub projects`(defautl: `./drafts/projects`)
+
+It can also include the `ignore` section. Any path under `ignore.paths` will be ignored, i.e. no entries will be excavated from it and there will be no warning.
=
=``` toml
=[ignore]
@@ -231,7 +237,12 @@ paths = [
=
=In the future there might be more things to ignore, like authors, files in patches (globs), etc.
=
-Any project that is not listed and not ignored, but had activity within the period for which a devlog is being generated, will produce a warning with useful instructions.
+A configuration file can also contain `frontmatter` section with configuration related to the entries front-matter.
+
+  - `projects_cell_path`: where to write the name of a project that the entry belongs to
+
+See the `devlog-sample.toml` file.
+
=
=## Stubbing projects 
=