Luckily in the README I found a reference that there is an example in the example branch. I checked it out and after a few minutes figured out how to run generate the site from the example input data.

The plan

I already dealt with similar cases so here is what my plan was:

• Create one or more examples (based on the one in the example branch)
• Generate the results from each example, manually inspect if this is what we expect, and save them in a directory for expected output.
• The tests then would generate each example and then compare the results to the expected output. They should be the same.
• When there is some expected change, either because the input files of an example changed or because the processing changed, the authors should be able to verify the changes in the output and then they should be able to update the expected output easily.
• All this should be in the same branch where the development takes place so the expected output is always aligned with the current version of input and code.
• Each example could show one specific feature of the system or they could be like a tutorial showing how one builds a site with more and more complex features. In any case the examples are used both as documentation and as tests.

I opened an issue describing my idea a bit more briefly. Soon I got a reply from Tina, the author. She also assigned the issue to me.

The implementation

So here is what I did.

• I created a new branch based on the master branch.
• Created a directory called examples and in it create a very simple example in the examples/01-basic/ subdirectory. It had a subdirectory called input with all the necessary files for the most basic example. (And juts to clarify, this is short sentence, but this took me quite some time.)
• Then I ran PYTHONPATH=src python -m sssimp --input examples/01-basic/input example/01-basic/output that generated the output.
• I looked at the output files. They looked good. As much as I can tell.
• Then I added the following two files

examples/sssimp/test_examples.py

import os
import sys
import filecmp
import difflib
import shutil

import pytest


root = os.path.dirname(__file__)
examples = os.listdir(os.path.join(root, 'examples'))

def check_diffs(dcmp):
    print(f'check_diffs {dcmp.left} and {dcmp.right}')
    assert dcmp.left_only == []
    assert dcmp.right_only == []
    success = True
    for filename in dcmp.diff_files:
        success = False
        print(f"File with difference: {filename}")
        with open(os.path.join(dcmp.left, filename)) as fh:
            left_content = fh.readlines()
        with open(os.path.join(dcmp.right, filename)) as fh:
            right_content = fh.readlines()
        for row in difflib.unified_diff(left_content, right_content):
            print(row, end="")
        print()

    #assert dcmp.diff_files == []
    for sub_dcmp in dcmp.subdirs.values():
        success = check_diffs(sub_dcmp) and success
    return success

@pytest.mark.parametrize("example", examples)
def test_examples(example, tmpdir, request):
    print(example)
    os.environ["PYTHONPATH"] = os.path.join(root, 'src')
    outdir = os.path.join(tmpdir, 'out')
    cmd = f"{sys.executable} -m sssimp --input {os.path.join(root, 'examples', example, 'input')} {outdir}"
    print(cmd)
    exit_code = os.system(cmd)
    assert exit_code == 0
    expected_output = os.path.join(root, 'examples', example, 'output')
    save = request.config.getoption("--save")
    if save:
        if os.path.exists(expected_output):
            shutil.rmtree(expected_output)
        shutil.move(outdir, expected_output)
        return

    dcmp = filecmp.dircmp(expected_output, outdir)
    print(expected_output, outdir)
    assert check_diffs(dcmp), "Some files differ. See above using the -s flag of pytest"

examples/sssimp/conftest.py

def pytest_addoption(parser):
    parser.addoption("--save", action='store_true')

Running tests

Now I can run the tests with

pytest -sv

If I make some changes to the input files or to the code then the test will fail and will show me changes in the output. e.g. something like this:

File with difference: index.html
---
+++
@@ -5,6 +5,6 @@
 	<body>
 		<h1>Title</h1>
 		<p>Content:</p>
-<h1>Head 1</h1>
+<h1>Head new</h1>
 	</body>
 </html>

Then, if I think the changes I are as I wanted them I can run

pytest --save

This will re-generate the output and replace the expected output with the new files. I can inspect them again, now the full version of the file. If they are as I really wanted them then I can commit all the new files. If I decide this is not what I wanted I can reset the content of the output directories and go back to my earlier expectation.

Continuous Integration (CI) with GitHub Actions

For the continuous integration I added the following file as .github/workflows/ci.yml

examples/sssimp/.github/workflows/ci.yml

name: Python

on:
  push:
  pull_request:

jobs:
  build_python:

    strategy:
      fail-fast: false
      matrix:
        runner: [ubuntu-latest, macos-latest, windows-latest]
        python-version: ["3.9", "3.10", "3.11.0-rc.2"]

    runs-on: ${{matrix.runner}}
    name: OS ${{matrix.runner}} Python ${{matrix.python-version}}

    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python-version }}

    - name: Install dependencies
      run: |
        pip install -e .
        pip install pytest

    - name: Check Python version
      run: python -V

    - name: Test with pytest
      run: pytest -vs

It runs on 3 different Operating Systems and 3 different version of Python.

After pushing the whole thing to my fork on GitHub I noticed that the tests are failing on both OSX and Windows. On OSX the problem was the order of entries. At first I considered this a bug in the Python module, but then I figured I could use some Jinja instructions to force the list to be in the same order every time.

However in the case of Windows I found that it looks like a real bug. In the href of some links Windows would insert a back-slash \ between the parts as is normal for Windows pathes, but incorrect for URLs. So I left that failure in the CI system.