testing: runtest.py refactor, simplification, new features, easy oiiotool docs

[lgritz]

Refactor of runtest.py:

• Enhance the workhorse run_app() that assembles command lines: add a failureok parameter and handle that logic internally; automatically call oiio_app() on the command name if it's one of the OIIO command line apps, so the caller doesn't need to.

• Rewrite many of the other utilities like info_command, diff_command, maketx_command, rw_command, testtex_command, iconvert, oiiotool, to (a) use the new run_app() underneath and push a lot of the redundant logic into this core function; (b) use Python "f-strings" as a more compact notation for assembling the command strings. This combination greatly simpifies these functions into fairly thin wrappers around run_app().

• New run_commands that can take a string of multiple commands, split on newlines, and concatenate the run_app() result of each. Blank lines or comments (lines whose first non-whitespace charcter is #) are safely ignored.

• Remove the "successmessage" parameter to iconvert -- we only used it in a couple places, and those can use "&& echo ..." instead.

• Compare test text output vs reference in a way that tolerates trailing whitespace changes (helped resolve some tricky Windows corner cases related to the "&& echo" idiom mentioned above.

So what does all this buy us, especially the run_commands()? It can transform a sequence in a testsuite/blah/run.py like this,

command += oiiotool("-pattern constant:color=.25,.5,.75 64x64 3 -d half -o rgb64.exr")
command += oiiotool("-pattern constant:color=.25,.5,.75 64x64 3 -pattern constant:color=42 64x64 1 --chnames Z --siappend -d half -o rgb-z-parts64.exr")

into this:

command += run_commands("""
    oiiotool -pattern constant:color=.25,.5,.75 64x64 3 -d half -o rgb64.exr
    oiiotool -pattern constant:color=.25,.5,.75 64x64 3 -pattern constant:color=42 64x64 1 --chnames Z --siappend -d half -o rgb-z-parts64.exr
    """)

Nice, yeah? Within multi-line strings, we can just have shell commands exactly as how we would type them...

... and also exactly how we'd like them to appear in documentation!

So this allows us to do with command line examples in the oiiotool.md documentation chapter what we have been doing with C++ and Python code examples in the other chapters -- have the code actually execute as part of our testing (so we know they always work), and have the docs incorporate specific lines by reference, without duplication. For example, in a test:

command += run_commands("""
    oiiotool -pattern constant:color=.25,.5,.75 64x64 3 -d half -o rgb64.exr
    # BEGIN-oiiotool-example
    oiiotool -pattern constant:color=.25,.5,.75 64x64 3 -pattern constant:color=42 64x64 1 --chnames Z --siappend -d half -o rgb-z-parts64.exr
    # END-oiiotool-example
    """)

and then in the docs,

```{literalinclude} ../../testsuite/oiiotool/run.py
:language: bash
:start-after: BEGIN-oiiotool-example
:end-before: END-oiiotool-example
```

and it will insert just that one line between the BEGIN/END comment pair. The line right from the test, that actually executes in the testsuite, so it never is incorrect or stale.

[jinhgkim]

Nice! I like this refactor.

[jinhgkim]

nit: I don't think we should pass failureok here since testtex_command doesn't take it as a param? failureok is default to 0, so it doesn't hurt to have it here, but if we wanna be super strict, I think we remove it.

[nrusch]

Out of curiosity, is there any reason to prefer a multi-line string for multiple commands over a sequence of strings? The latter makes it possible to wrap very long commands for readability (if you wanted to), and still allows you to intersperse comments, except they would be actual Python comments.

(also a minor style nit:

Suggested change

	def run_commands(commands : str, silent: bool=False,
	def run_commands(commands: str, silent: bool=False,

[lgritz]

Out of curiosity, is there any reason to prefer a multi-line string for multiple commands over a sequence of strings? The latter makes it possible to wrap very long commands for readability (if you wanted to), and still allows you to intersperse comments, except they would be actual Python comments.

@nrush, I'm not sure what you mean; can you clarify?

[nrush]

@nrusch

[lgritz]

so sorry

[nrusch]

By a "sequence of strings" I mean a sequence-type container like list or tuple.

So, for example, instead of:

command += run_commands(
"""
# Here's a comment on the first command
iconvert short-exif-app1-len4.jpg out.null && echo short-exif-app1-len4-ok
# And here's a note about the second
iconvert short-exif-app1-len5.jpg out.null && echo short-exif-app1-len5-ok
""")

You would do something like:

command += run_commands(
    [
        # Here's a comment on the first command
        "iconvert short-exif-app1-len4.jpg out.null && echo short-exif-app1-len4-ok",
        # And here's a note about the second
        "iconvert short-exif-app1-len5.jpg out.null && echo short-exif-app1-len5-ok",
    ]
)

[lgritz]

@nrusch

The specific goal of this PR was to be able to have a sequence of consecutive lines in the run.py that would be verbatim identical to any of (a) what a user would type into a shell, (b) what would appear in a shell script to invoke that set of commands, (c) what should get included/copied by reference into the documentation showing examples of the commands.

I believe that this is a property of the first example you wrote above (verbatim lines within a """ block), but not the second, which defeats it by needing the extra quoting and commas.

[linsen458-spec]

Nice — this makes the docs-verbatim test blocks possible, and the run_commands form reads much closer to what a user would actually type.

Two small things I noticed while reading:

1. In run_commands, the default failureok: bool=failureok captures the module-level global at function-definition time — which is before run.py is exec'd, so a failureok = True set inside a run.py isn't seen by calls that omit the parameter. Today it works out because the final runtest(command, outputs, failureok=failureok) reads the global again at run time, but it reads like it's picking up the caller's value when it isn't. Maybe default to None and resolve inside the function?
2. run_app does words = cmd.split(maxsplit=1) and then indexes words[0] — an all-whitespace command (possible if someone calls run_app directly, run_commands filters them) would raise IndexError. Cheap to guard.

Other than that, the _rstrip_line trick for cmd.exe's trailing space on echo is a nice touch, and the conversions all look faithful to the originals.

We'll build the rework of #5464 on top of this once it's in — thanks for setting up the template!

[lgritz]

@linsen458-spec Thanks, that's helpful. I'm not enough of a fluent python users to have spotted those two missteps on my part. I will fix.

[grdanny]

is it ever ok to be not one of the known oiio_app_list commands?

[lgritz]

Yes! You can run any command you want. But in that case, we wouldn't want to prepend the relative path to the build we are testing, because something not built by OIIO won't be there.

[grdanny]

This looks cool to me as well! had one small questions there, but other than this looks very cool and might be nice to get in before dev days if someone wants to do the oiiotool-->docs stuff...