diff --git a/CHANGELOG.md b/CHANGELOG.md index 13f7b48..08782dc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,10 +17,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- use `DEFAULT_NAMESPACE` as the environment setting for the default stack name + - allow `-u`/`--unresolved` to select zero or more variables, matching `-r`/`--resolve`, with `-q` for a single value. Put stack names before the option: replace `envstack -u dev` with `envstack dev -u`. ### Added +- document expansion, YAML transformation timing, runtime controls, and current name-collision behavior, with a composed example (#125) + - add NumPy, Pillow, and OpenEXR examples demonstrating accumulated search paths and runtime selection of NumPy builds - add a standalone PySide6 example with a small GUI application - add installation guides and distman manifests aligned with the examples' deployment paths diff --git a/docs/api.md b/docs/api.md index 889513c..20d36fa 100644 --- a/docs/api.md +++ b/docs/api.md @@ -48,7 +48,6 @@ Put stack names before these options: use `envstack dev -u`, not `envstack -u dev`. Arguments after `-u` now select variables, just as they do after `-r`. - ## Resolving a stack into concrete values To evaluate variable references such as `${ROOT}/${ENV}`, resolve the loaded @@ -222,14 +221,7 @@ Restore the previous process environment: ## Configuration variables -The following environment variables affect runtime behavior: - -| Name | Description | -|------|-------------| -| `ALLOW_COMMANDS` | Allow embedded commands | -| `COMMAND_TIMEOUT` | Embedded command timeout in seconds | -| `DEFAULT_NAMESPACE` | Name of the default environment stack (`default`) | -| `ENVPATH` | Colon-separated paths to search for environment files | -| `IGNORE_MISSING` | Ignore missing stack files when resolving environments | -| `INTERACTIVE` | Force shells to run in interactive mode | -| `STACK` | Stores the name of the current environment stack | +See [Runtime context](runtime-context.md) for the complete control-variable table, +when settings are read, and collision behavior. The default-stack environment +setting is `DEFAULT_NAMESPACE`; `ENVPATH` uses the host platform's path delimiter +(`:` on POSIX and `;` on Windows). diff --git a/docs/expansion.md b/docs/expansion.md index 7825198..2ac9e1e 100644 --- a/docs/expansion.md +++ b/docs/expansion.md @@ -1,5 +1,15 @@ # Variable Expansion +Runtime variables provide context. Parameter modifiers select and compose +values. YAML tags transform their representation when values are resolved or +written. These mechanisms work together, but they run at different stages. + +For example, a shell-supplied `ROOT` provides context, `${ROOT:-/shared}` selects +a fallback, and `!base64` marks a stored encoded value for decoding. A tag does +not change variable precedence or select the active stack. + +See [Runtime context](runtime-context.md) for control variables and name collisions. + envstack supports Bash-like variable expansion inside stack values. This is the syntax behind values such as `${ROOT}/${ENV}`, @@ -14,13 +24,18 @@ This is the syntax behind values such as `${ROOT}/${ENV}`, | `${VAR:-default}` | If `VAR` is unset or empty, use `default` without assigning it | | `${VAR:?message}` | Raise an error if `VAR` is unset or empty | +This is the complete supported modifier set. Bash forms such as `${VAR-default}`, +`${VAR:+alternate}`, `${#VAR}`, substring operations, and pattern replacements +are not supported. Use the forms above rather than relying on unsupported +expressions to behave like a shell. + These rules apply both in stack files and when resolving values through the Python API. ## Simple substitution -Direct references are resolved against the current stack and then fall back to -the parent or process environment: +Direct references use the final stack definition. A missing variable can fall +back to the process environment; an explicit empty string does not: ```yaml ROOT: /mnt/pipe @@ -43,8 +58,8 @@ OPTIONAL_SUFFIX: ${SUFFIX} ## Defaults With `:=` -Use `:=` when the fallback should become the variable's effective value for the -rest of the resolution pass: +Use `:=` when subsequent references should be able to reuse a fallback. A +self-assignment stays local to the definition being evaluated: ```yaml LOG_LEVEL: ${LOG_LEVEL:=INFO} @@ -69,9 +84,10 @@ variable itself: PROMPT_COLOR: ${PROMPT_COLOR:-green} ``` -This is helpful when a value should have a display or formatting fallback, but -you do not want that fallback to become the new resolved value of -`PROMPT_COLOR`. +Here `PROMPT_COLOR` itself resolves to `green` when no preceding value exists. +For a fallback that leaves a separate variable unchanged, use +`LABEL: ${PROMPT_COLOR:-green}`: `LABEL` gets `green`, but no assignment is made +to `PROMPT_COLOR`. ## Required Variables With `:?` @@ -96,9 +112,12 @@ If neither `FOO` nor `BAR` is already set, this resolves to: ```bash FOO=/foo/bar -BAR=/foo/bar ``` +The nested `:=` also makes `BAR` available to subsequent references during +resolution. It does not add an absent `BAR` key to the returned environment or +to the process environment. + Nested expressions are often the cleanest way to define a chain of fallbacks without repeating the final literal value. @@ -211,16 +230,105 @@ PYVERSION: $(python -c "import sys; print(f'{sys.version_info[0]}.{sys.version_i PYTHONPATH: ${DEPLOY_ROOT}/lib/python${PYVERSION} ``` -Command substitution is separate from modifier syntax and is only evaluated when -embedded commands are allowed. +Set `ALLOW_COMMANDS=1` before launching envstack or importing it in Python. +Command substitution recognizes a whole value of the form `$(...)`; it is not +shell-style interpolation into arbitrary surrounding text. Successful output is +stripped of leading and trailing whitespace. Command execution starts with the launching process environment and overlays +the resolved default stack. Put the intended interpreter on `PATH` before +starting envstack, and ensure the default stack preserves that selection. + +Loading can resolve values internally to update `ENVPATH`, and dependency lookup +can evaluate a value more than once. Do not rely on a command running exactly +once or only when you request that variable with `-r`. + +## YAML Transformation Tags + +Built-in tags apply to scalar values. A tag in a file describes the stored +representation; it is not a parameter modifier. + +| Tag | Loading and resolution | Writing YAML | +| --- | --- | --- | +| `!base64` | Decode the stored Base64 payload before expanding its contents | Encode a newly constructed `Base64Node`; preserve an unchanged payload loaded from YAML | +| `!aesgcm` | Decrypt with `ENVSTACK_SYMMETRIC_KEY`, then expand | Encrypt a new `AESGCMNode`; preserve an unchanged loaded payload | +| `!fernet` | Decrypt with `ENVSTACK_FERNET_KEY`, then expand | Encrypt a new `FernetNode`; preserve an unchanged loaded payload | +| `!encrypt` | Select AES-GCM, then Fernet, then Base64 according to available keys | Use the same selection for a new `EncryptedNode`; preserve an unchanged loaded payload | +| `!md5` | Return the stored text; resolution does not compute a hash | Hash the node's current text when serializing | + +Base64 is encoding, not encryption. MD5 is a digest, not encryption and not a +password-storage mechanism. See [Secrets and encryption](secrets.md) for keys +and encrypted-file workflows. For `!encrypt`, use a key context compatible with +the one used when writing the payload; the tag does not identify its algorithm. + +For example, writing `MD5Node("hello")` produces +`!md5 '5d41402abc4b2a76b9719d911017c592'`. Loading `!md5 hello` alone does not +hash `hello`. Serializing that loaded MD5 node hashes its current text again; +unlike the encryption nodes, it does not preserve an unchanged loaded payload. + +### Compose Context, Tags, and Modifiers -## Resolution Notes +Save this as `context.env`: + +```yaml +all: + ROOT: ${ROOT:-/shared} + ENV: ${ENV:-prod} + DEPLOY_ROOT: !base64 JHtST09UfS8ke0VOVn0= + LIB: ${DEPLOY_ROOT}/lib/python +``` + +The Base64 payload represents the literal expression `${ROOT}/${ENV}`. With +`ROOT=/studio` and `ENV=dev` in the launching shell: + +```bash +envstack context -r DEPLOY_ROOT LIB +``` + +produces: + +```text +DEPLOY_ROOT=/studio/dev +LIB=/studio/dev/lib/python +``` -- Empty strings are treated the same as unset values for `:=`, `:-`, and `:?`. -- Nested expressions are supported in modifier arguments. -- Cyclical references resolve to an empty string instead of recursing forever. -- Path-like results such as `PATH` and `PYTHONPATH` are normalized and - deduplicated during resolution. +Without those shell values, the defaults produce `/shared/prod` and +`/shared/prod/lib/python`. The tag decodes the expression, the modifiers select +context values, and ordinary references compose the final paths. Use +`envstack context -u DEPLOY_ROOT` to inspect the stored payload instead. + +## Evaluation Order + +1. Read YAML, construct tagged nodes, and process YAML anchors and merge keys. + Load included sources in order and select the requested platform section. + If that section is absent, use `all`; use `<<: *all` in a platform section + when it should inherit an anchored `all` mapping. +2. Assemble the final stack and retain earlier definitions for self-references. + `load_environ()` returns unresolved values, but internally resolves the + environment to update `ENVPATH` between requested stacks. +3. Decode or decrypt top-level tagged values using the available key context. + Process-environment encryption keys take precedence over same-named stack + keys during resolution. +4. Resolve dependencies and modifiers on demand. Ordinary references use final + definitions; self-references fall back through preceding definitions. An + eligible `$(...)` value runs when evaluated if commands are enabled. +5. Normalize and deduplicate path-like results using the selected platform, + then restore supported Python literal types in top-level results. +6. Apply the result only when requested. `resolve_environ()` returns a new + environment; `init()` updates `os.environ` and `sys.path`, while a wrapped + command receives its own composed environment. + +This is dependency resolution, not sequential execution of YAML assignments. +Avoid using key order as a way to control effects. `:=` can retain assignments +for subsequent references, so use explicit variable definitions when several +independent expressions need the same default. + +Lists and dictionaries are traversed for variable expansion. Keep encoded or +encrypted expressions at the top level when composing tags with modifiers: +nested tagged values do not follow the same top-level decode-then-expand path. + +A cycle uses earlier definitions when possible. Without an earlier or shell +value, a plain recursive reference contributes an empty string; default and +required-value operators can instead supply a fallback or raise an error. +YAML `null` is distinct from `""`; it is not an instruction to unset a variable. ## See Also diff --git a/docs/index.md b/docs/index.md index 332c6b1..ac5b7a3 100644 --- a/docs/index.md +++ b/docs/index.md @@ -199,3 +199,8 @@ cat .env | envstack --set -o out.env values, and nested modifiers - [FAQ](faq.md): operational details and gotchas - [API](api.md): Python and CLI reference material + +## Expansion and Runtime Context + +- [Variable expansion](expansion.md): modifiers, YAML tags, and evaluation order +- [Runtime context](runtime-context.md): control variables and name collisions diff --git a/docs/runtime-context.md b/docs/runtime-context.md new file mode 100644 index 0000000..fc2c72c --- /dev/null +++ b/docs/runtime-context.md @@ -0,0 +1,87 @@ +# Runtime Context + +Runtime context is the information available when a stack is loaded or resolved: +the launching process environment, selected stack names, selected platform, +search directories, and explicitly defined stack values. Modifiers select and +compose those values; YAML tags encode, decode, encrypt, or hash representations. +See [Variable expansion](expansion.md) for their syntax and evaluation order. + +## Context and Stack Data + +`ROOT`, `ENV`, `HOME`, and `USER` are ordinary variable names. A stack can define +them, use a self-reference to inherit them, or leave them absent so references +can fall back to the process environment. envstack does not inject a complete +runtime-variable dictionary into every loaded stack. + +Likewise, `PLATFORM` inside a stack is ordinary data. Selecting `--platform +windows` or `load_environ(..., platform="windows")` chooses YAML sections and +path delimiters; it does not create or overwrite a `PLATFORM` variable. Changing +that variable does not change platform selection. + +`STACK` is special: loading supplies the last requested stack name when the +merged stack has no truthy `STACK` value. A nonempty value defined in a stack is +preserved. Treat `STACK` as loader metadata rather than using it for unrelated +application data. + +## Runtime Controls + +Set import-time controls in the launching shell before starting envstack, or +before importing it in Python. A same-named value inside a stack is still data; +it does not reconfigure constants already read by the running process. + +| Name | Purpose and timing | +| --- | --- | +| `ENVPATH` | Search directories, separated by the host's path delimiter. Read when loading; a resolved stack `ENVPATH` can affect subsequent requested stacks. The current directory is also searched. | +| `DEFAULT_NAMESPACE` | Default stack name, read at import time; defaults to `default`. Explicit stack arguments override it. | +| `ALLOW_COMMANDS` | Enable `$(...)`, read at import time; disabled by default. Values `1`, `true`, `True`, and `TRUE` enable it. | +| `COMMAND_TIMEOUT` | Command timeout in seconds, read at import time; defaults to 5, including when the setting is not an integer. | +| `IGNORE_MISSING` | Ignore missing stack files, read at import time; enabled by default. Uses the same true values as `ALLOW_COMMANDS`. The Python loading API can override it. | +| `LOG_LEVEL` | Logging level read at import time; defaults to `INFO`. | +| `INTERACTIVE` | Read from a command wrapper's environment to select interactive command execution. | +| `SHELL` / `ComSpec` | Host shell detection at import time: `SHELL` on POSIX and `ComSpec` on Windows. | +| `ENVSTACK_SHELL` | Override the shell selected when launching an envstack shell session. | +| `ENVSTACK_SYMMETRIC_KEY` | AES-GCM key used during encryption/decryption. | +| `ENVSTACK_FERNET_KEY` | Fernet key used during encryption/decryption. | + +Enabling command substitution in a stack with `ALLOW_COMMANDS: 1` does not +enable it for the process currently loading that file. Exporting that value can +influence a subsequently launched envstack process. + +## Runtime Names and Collisions + +Use application-specific names, such as `MYAPP_CACHE_ROOT`, for application +settings. Keep `ENVSTACK_*` for envstack integration, and avoid repurposing the +runtime control names above. This is naming guidance, not an enforced reserved +namespace: the loader does not reject these keys or emit collision warnings. +There is no separate runtime lookup syntax or implicit `ENVSTACK_*` variable +expansion mechanism. + +The current collision behavior is: + +- Ordinary references prefer stack definitions over shell values. An explicit + empty string also wins. A missing variable can fall back to the shell. +- A self-reference uses preceding stack definitions before the shell. +- Import-time control settings and same-named stack data can differ. For example, + a stack's `LOG_LEVEL` does not automatically reconfigure the current logger. +- For decryption, process-environment encryption keys override matching stack + keys. Do not use key names for unrelated data. +- A nonempty stack-defined `STACK` overrides the loader's default metadata value. + +No new collision rejection or reserved-prefix enforcement is introduced by +this model. Applications should not assume that writing a runtime-looking name +inside a stack changes envstack's own state. + +## Inspect Before Applying + +```bash +envstack myapp -u ROOT ENV DEPLOY_ROOT +envstack myapp -r ROOT ENV DEPLOY_ROOT +``` + +The first command displays stored values; the second displays resolved values. +Both load the stack, so enabled commands may run during internal `ENVPATH` +resolution. Neither command changes the invoking shell's environment. + +For an application process, use `envstack myapp -- command`, or apply a stack +inside Python with `envstack.init("myapp")`. The [Python API](api.md) describes +loading, resolution, activation, and restoration. diff --git a/docs/secrets.md b/docs/secrets.md index e0ee896..44a5d0f 100644 --- a/docs/secrets.md +++ b/docs/secrets.md @@ -30,13 +30,13 @@ decrypt values**. --- -## Supported encryption algorithms +## Supported encryption and encoding -envstack supports the following encryption algorithms: +envstack supports the following encryption and encoding methods: | Algorithm | Key required | Environment variable | |---------|--------------|----------------------| -| Base64 | No | (default) | +| Base64 encoding (not encryption) | No | (default) | | AES-GCM | Yes | `ENVSTACK_SYMMETRIC_KEY` | | Fernet | Yes | `ENVSTACK_FERNET_KEY` | @@ -151,4 +151,7 @@ Values will automatically decrypt during resolution. - Use includes to inject keys at activation time - Combine with external secret managers for higher security needs -envstack encryption is intentionally simple, explicit, and inspectable. \ No newline at end of file +envstack encryption is intentionally simple, explicit, and inspectable. + +For YAML tag syntax and read/write timing, see +[YAML transformation tags](expansion.md#yaml-transformation-tags). diff --git a/lib/envstack/config.py b/lib/envstack/config.py index 5e0c71a..343c578 100644 --- a/lib/envstack/config.py +++ b/lib/envstack/config.py @@ -61,7 +61,7 @@ def detect_shell(): DEBUG = os.getenv("DEBUG") # default namespace -DEFAULT_NAMESPACE = os.getenv("DEFAULT_ENV_STACK", "default") +DEFAULT_NAMESPACE = os.getenv("DEFAULT_NAMESPACE", "default") # allow embedded commands ALLOW_COMMANDS = os.getenv("ALLOW_COMMANDS", "0") in ("1", "true", "True", "TRUE") diff --git a/tests/test_config.py b/tests/test_config.py new file mode 100644 index 0000000..ac224fc --- /dev/null +++ b/tests/test_config.py @@ -0,0 +1,44 @@ +"""Verify import-time settings through fresh CLI and Python API processes.""" + +import json +import os +import subprocess +import sys +from pathlib import Path + +import pytest + + +@pytest.mark.parametrize("namespace", [None, "custom"]) +@pytest.mark.parametrize("explicit", [None, "chosen"]) +def test_default_namespace_at_startup(tmp_path, namespace, explicit): + for name in ("default", "custom", "chosen"): + (tmp_path / (name + ".env")).write_text("all:\n SELECTED: " + name + "\n") + process_env = dict(os.environ) + process_env.pop("DEFAULT_NAMESPACE", None) + if namespace is not None: + process_env["DEFAULT_NAMESPACE"] = namespace + process_env["ENVPATH"] = str(tmp_path) + process_env["PYTHONPATH"] = str(Path(__file__).resolve().parents[1] / "lib") + expected = explicit or namespace or "default" + code = """ +import json +import sys +from envstack import config, load_environ +from envstack.cli import parse_args +explicit = sys.argv[1:] +sys.argv = ["envstack"] + explicit + ["-u", "SELECTED"] +args, command = parse_args() +api = load_environ(*explicit) +cli = load_environ(args.namespace) +print(json.dumps([config.DEFAULT_NAMESPACE, api["SELECTED"], cli["SELECTED"]])) +""" + result = subprocess.run( + [sys.executable, "-c", code] + ([explicit] if explicit else []), + env=process_env, + cwd=str(tmp_path), + capture_output=True, + text=True, + check=True, + ) + assert json.loads(result.stdout) == [namespace or "default", expected, expected]