Document resolution and runtime context

CHANGELOG.md

@@ -17,10 +17,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- use `DEFAULT_NAMESPACE` as the environment setting for the default stack name
+
 - allow `-u`/`--unresolved` to select zero or more variables, matching `-r`/`--resolve`, with `-q` for a single value. Put stack names before the option: replace `envstack -u dev` with `envstack dev -u`.
 
 ### Added
 
+- document expansion, YAML transformation timing, runtime controls, and current name-collision behavior, with a composed example (#125)
+
 - add NumPy, Pillow, and OpenEXR examples demonstrating accumulated search paths and runtime selection of NumPy builds
 - add a standalone PySide6 example with a small GUI application
 - add installation guides and distman manifests aligned with the examples' deployment paths

docs/api.md

@@ -48,7 +48,6 @@ Put stack names before these options: use `envstack dev -u`, not
 `envstack -u dev`. Arguments after `-u` now select variables, just as they do
 after `-r`.
 
-
 ## Resolving a stack into concrete values
 
 To evaluate variable references such as `${ROOT}/${ENV}`, resolve the loaded
@@ -222,14 +221,7 @@ Restore the previous process environment:
 
 ## Configuration variables
 
-The following environment variables affect runtime behavior:
-
-| Name | Description |
-|------|-------------|
-| `ALLOW_COMMANDS` | Allow embedded commands |
-| `COMMAND_TIMEOUT` | Embedded command timeout in seconds |
-| `DEFAULT_NAMESPACE` | Name of the default environment stack (`default`) |
-| `ENVPATH` | Colon-separated paths to search for environment files |
-| `IGNORE_MISSING` | Ignore missing stack files when resolving environments |
-| `INTERACTIVE` | Force shells to run in interactive mode |
-| `STACK` | Stores the name of the current environment stack |
+See [Runtime context](runtime-context.md) for the complete control-variable table,
+when settings are read, and collision behavior. The default-stack environment
+setting is `DEFAULT_NAMESPACE`; `ENVPATH` uses the host platform's path delimiter
+(`:` on POSIX and `;` on Windows).

docs/expansion.md

@@ -1,5 +1,15 @@
 # Variable Expansion
 
+Runtime variables provide context. Parameter modifiers select and compose
+values. YAML tags transform their representation when values are resolved or
+written. These mechanisms work together, but they run at different stages.
+
+For example, a shell-supplied `ROOT` provides context, `${ROOT:-/shared}` selects
+a fallback, and `!base64` marks a stored encoded value for decoding. A tag does
+not change variable precedence or select the active stack.
+
+See [Runtime context](runtime-context.md) for control variables and name collisions.
+
 envstack supports Bash-like variable expansion inside stack values.
 
 This is the syntax behind values such as `${ROOT}/${ENV}`,
@@ -14,13 +24,18 @@ This is the syntax behind values such as `${ROOT}/${ENV}`,
 | `${VAR:-default}` | If `VAR` is unset or empty, use `default` without assigning it |
 | `${VAR:?message}` | Raise an error if `VAR` is unset or empty |
 
+This is the complete supported modifier set. Bash forms such as `${VAR-default}`,
+`${VAR:+alternate}`, `${#VAR}`, substring operations, and pattern replacements
+are not supported. Use the forms above rather than relying on unsupported
+expressions to behave like a shell.
+
 These rules apply both in stack files and when resolving values through the
 Python API.
 
 ## Simple substitution
 
-Direct references are resolved against the current stack and then fall back to
-the parent or process environment:
+Direct references use the final stack definition. A missing variable can fall
+back to the process environment; an explicit empty string does not:
 
 ```yaml
 ROOT: /mnt/pipe
@@ -43,8 +58,8 @@ OPTIONAL_SUFFIX: ${SUFFIX}
 
 ## Defaults With `:=`
 
-Use `:=` when the fallback should become the variable's effective value for the
-rest of the resolution pass:
+Use `:=` when subsequent references should be able to reuse a fallback. A
+self-assignment stays local to the definition being evaluated:
 
 ```yaml
 LOG_LEVEL: ${LOG_LEVEL:=INFO}
@@ -69,9 +84,10 @@ variable itself:
 PROMPT_COLOR: ${PROMPT_COLOR:-green}
 ```
 
-This is helpful when a value should have a display or formatting fallback, but
-you do not want that fallback to become the new resolved value of
-`PROMPT_COLOR`.
+Here `PROMPT_COLOR` itself resolves to `green` when no preceding value exists.
+For a fallback that leaves a separate variable unchanged, use
+`LABEL: ${PROMPT_COLOR:-green}`: `LABEL` gets `green`, but no assignment is made
+to `PROMPT_COLOR`.
 
 ## Required Variables With `:?`
 
@@ -96,9 +112,12 @@ If neither `FOO` nor `BAR` is already set, this resolves to:
 
 ```bash
 FOO=/foo/bar
-BAR=/foo/bar
 ```
 
+The nested `:=` also makes `BAR` available to subsequent references during
+resolution. It does not add an absent `BAR` key to the returned environment or
+to the process environment.
+
 Nested expressions are often the cleanest way to define a chain of fallbacks
 without repeating the final literal value.
 
@@ -211,16 +230,105 @@ PYVERSION: $(python -c "import sys; print(f'{sys.version_info[0]}.{sys.version_i
 PYTHONPATH: ${DEPLOY_ROOT}/lib/python${PYVERSION}
 ```
 
-Command substitution is separate from modifier syntax and is only evaluated when
-embedded commands are allowed.
+Set `ALLOW_COMMANDS=1` before launching envstack or importing it in Python.
+Command substitution recognizes a whole value of the form `$(...)`; it is not
+shell-style interpolation into arbitrary surrounding text. Successful output is
+stripped of leading and trailing whitespace. Command execution starts with the launching process environment and overlays
+the resolved default stack. Put the intended interpreter on `PATH` before
+starting envstack, and ensure the default stack preserves that selection.
+
+Loading can resolve values internally to update `ENVPATH`, and dependency lookup
+can evaluate a value more than once. Do not rely on a command running exactly
+once or only when you request that variable with `-r`.
+
+## YAML Transformation Tags
+
+Built-in tags apply to scalar values. A tag in a file describes the stored
+representation; it is not a parameter modifier.
+
+| Tag | Loading and resolution | Writing YAML |
+| --- | --- | --- |
+| `!base64` | Decode the stored Base64 payload before expanding its contents | Encode a newly constructed `Base64Node`; preserve an unchanged payload loaded from YAML |
+| `!aesgcm` | Decrypt with `ENVSTACK_SYMMETRIC_KEY`, then expand | Encrypt a new `AESGCMNode`; preserve an unchanged loaded payload |
+| `!fernet` | Decrypt with `ENVSTACK_FERNET_KEY`, then expand | Encrypt a new `FernetNode`; preserve an unchanged loaded payload |
+| `!encrypt` | Select AES-GCM, then Fernet, then Base64 according to available keys | Use the same selection for a new `EncryptedNode`; preserve an unchanged loaded payload |
+| `!md5` | Return the stored text; resolution does not compute a hash | Hash the node's current text when serializing |
+
+Base64 is encoding, not encryption. MD5 is a digest, not encryption and not a
+password-storage mechanism. See [Secrets and encryption](secrets.md) for keys
+and encrypted-file workflows. For `!encrypt`, use a key context compatible with
+the one used when writing the payload; the tag does not identify its algorithm.
+
+For example, writing `MD5Node("hello")` produces
+`!md5 '5d41402abc4b2a76b9719d911017c592'`. Loading `!md5 hello` alone does not
+hash `hello`. Serializing that loaded MD5 node hashes its current text again;
+unlike the encryption nodes, it does not preserve an unchanged loaded payload.
+
+### Compose Context, Tags, and Modifiers
 
-## Resolution Notes
+Save this as `context.env`:
+
+```yaml
+all:
+  ROOT: ${ROOT:-/shared}
+  ENV: ${ENV:-prod}
+  DEPLOY_ROOT: !base64 JHtST09UfS8ke0VOVn0=
+  LIB: ${DEPLOY_ROOT}/lib/python
+```
+
+The Base64 payload represents the literal expression `${ROOT}/${ENV}`. With
+`ROOT=/studio` and `ENV=dev` in the launching shell:
+
+```bash
+envstack context -r DEPLOY_ROOT LIB
+```
+
+produces:
+
+```text
+DEPLOY_ROOT=/studio/dev
+LIB=/studio/dev/lib/python
+```
 
-- Empty strings are treated the same as unset values for `:=`, `:-`, and `:?`.
-- Nested expressions are supported in modifier arguments.
-- Cyclical references resolve to an empty string instead of recursing forever.
-- Path-like results such as `PATH` and `PYTHONPATH` are normalized and
-  deduplicated during resolution.
+Without those shell values, the defaults produce `/shared/prod` and
+`/shared/prod/lib/python`. The tag decodes the expression, the modifiers select
+context values, and ordinary references compose the final paths. Use
+`envstack context -u DEPLOY_ROOT` to inspect the stored payload instead.
+
+## Evaluation Order
+
+1. Read YAML, construct tagged nodes, and process YAML anchors and merge keys.
+   Load included sources in order and select the requested platform section.
+   If that section is absent, use `all`; use `<<: *all` in a platform section
+   when it should inherit an anchored `all` mapping.
+2. Assemble the final stack and retain earlier definitions for self-references.
+   `load_environ()` returns unresolved values, but internally resolves the
+   environment to update `ENVPATH` between requested stacks.
+3. Decode or decrypt top-level tagged values using the available key context.
+   Process-environment encryption keys take precedence over same-named stack
+   keys during resolution.
+4. Resolve dependencies and modifiers on demand. Ordinary references use final
+   definitions; self-references fall back through preceding definitions. An
+   eligible `$(...)` value runs when evaluated if commands are enabled.
+5. Normalize and deduplicate path-like results using the selected platform,
+   then restore supported Python literal types in top-level results.
+6. Apply the result only when requested. `resolve_environ()` returns a new
+   environment; `init()` updates `os.environ` and `sys.path`, while a wrapped
+   command receives its own composed environment.
+
+This is dependency resolution, not sequential execution of YAML assignments.
+Avoid using key order as a way to control effects. `:=` can retain assignments
+for subsequent references, so use explicit variable definitions when several
+independent expressions need the same default.
+
+Lists and dictionaries are traversed for variable expansion. Keep encoded or
+encrypted expressions at the top level when composing tags with modifiers:
+nested tagged values do not follow the same top-level decode-then-expand path.
+
+A cycle uses earlier definitions when possible. Without an earlier or shell
+value, a plain recursive reference contributes an empty string; default and
+required-value operators can instead supply a fallback or raise an error.
+YAML `null` is distinct from `""`; it is not an instruction to unset a variable.
 
 ## See Also
 

docs/index.md

@@ -199,3 +199,8 @@ cat .env | envstack --set -o out.env
   values, and nested modifiers
 - [FAQ](faq.md): operational details and gotchas
 - [API](api.md): Python and CLI reference material
+
+## Expansion and Runtime Context
+
+- [Variable expansion](expansion.md): modifiers, YAML tags, and evaluation order
+- [Runtime context](runtime-context.md): control variables and name collisions

docs/runtime-context.md

@@ -0,0 +1,87 @@
+# Runtime Context
+
+Runtime context is the information available when a stack is loaded or resolved:
+the launching process environment, selected stack names, selected platform,
+search directories, and explicitly defined stack values. Modifiers select and
+compose those values; YAML tags encode, decode, encrypt, or hash representations.
+See [Variable expansion](expansion.md) for their syntax and evaluation order.
+
+## Context and Stack Data
+
+`ROOT`, `ENV`, `HOME`, and `USER` are ordinary variable names. A stack can define
+them, use a self-reference to inherit them, or leave them absent so references
+can fall back to the process environment. envstack does not inject a complete
+runtime-variable dictionary into every loaded stack.
+
+Likewise, `PLATFORM` inside a stack is ordinary data. Selecting `--platform
+windows` or `load_environ(..., platform="windows")` chooses YAML sections and
+path delimiters; it does not create or overwrite a `PLATFORM` variable. Changing
+that variable does not change platform selection.
+
+`STACK` is special: loading supplies the last requested stack name when the
+merged stack has no truthy `STACK` value. A nonempty value defined in a stack is
+preserved. Treat `STACK` as loader metadata rather than using it for unrelated
+application data.
+
+## Runtime Controls
+
+Set import-time controls in the launching shell before starting envstack, or
+before importing it in Python. A same-named value inside a stack is still data;
+it does not reconfigure constants already read by the running process.
+
+| Name | Purpose and timing |
+| --- | --- |
+| `ENVPATH` | Search directories, separated by the host's path delimiter. Read when loading; a resolved stack `ENVPATH` can affect subsequent requested stacks. The current directory is also searched. |
+| `DEFAULT_NAMESPACE` | Default stack name, read at import time; defaults to `default`. Explicit stack arguments override it. |
+| `ALLOW_COMMANDS` | Enable `$(...)`, read at import time; disabled by default. Values `1`, `true`, `True`, and `TRUE` enable it. |
+| `COMMAND_TIMEOUT` | Command timeout in seconds, read at import time; defaults to 5, including when the setting is not an integer. |
+| `IGNORE_MISSING` | Ignore missing stack files, read at import time; enabled by default. Uses the same true values as `ALLOW_COMMANDS`. The Python loading API can override it. |
+| `LOG_LEVEL` | Logging level read at import time; defaults to `INFO`. |
+| `INTERACTIVE` | Read from a command wrapper's environment to select interactive command execution. |
+| `SHELL` / `ComSpec` | Host shell detection at import time: `SHELL` on POSIX and `ComSpec` on Windows. |
+| `ENVSTACK_SHELL` | Override the shell selected when launching an envstack shell session. |
+| `ENVSTACK_SYMMETRIC_KEY` | AES-GCM key used during encryption/decryption. |
+| `ENVSTACK_FERNET_KEY` | Fernet key used during encryption/decryption. |
+
+Enabling command substitution in a stack with `ALLOW_COMMANDS: 1` does not
+enable it for the process currently loading that file. Exporting that value can
+influence a subsequently launched envstack process.
+
+## Runtime Names and Collisions
+
+Use application-specific names, such as `MYAPP_CACHE_ROOT`, for application
+settings. Keep `ENVSTACK_*` for envstack integration, and avoid repurposing the
+runtime control names above. This is naming guidance, not an enforced reserved
+namespace: the loader does not reject these keys or emit collision warnings.
+There is no separate runtime lookup syntax or implicit `ENVSTACK_*` variable
+expansion mechanism.
+
+The current collision behavior is:
+
+- Ordinary references prefer stack definitions over shell values. An explicit
+  empty string also wins. A missing variable can fall back to the shell.
+- A self-reference uses preceding stack definitions before the shell.
+- Import-time control settings and same-named stack data can differ. For example,
+  a stack's `LOG_LEVEL` does not automatically reconfigure the current logger.
+- For decryption, process-environment encryption keys override matching stack
+  keys. Do not use key names for unrelated data.
+- A nonempty stack-defined `STACK` overrides the loader's default metadata value.
+
+No new collision rejection or reserved-prefix enforcement is introduced by
+this model. Applications should not assume that writing a runtime-looking name
+inside a stack changes envstack's own state.
+
+## Inspect Before Applying
+
+```bash
+envstack myapp -u ROOT ENV DEPLOY_ROOT
+envstack myapp -r ROOT ENV DEPLOY_ROOT
+```
+
+The first command displays stored values; the second displays resolved values.
+Both load the stack, so enabled commands may run during internal `ENVPATH`
+resolution. Neither command changes the invoking shell's environment.
+
+For an application process, use `envstack myapp -- command`, or apply a stack
+inside Python with `envstack.init("myapp")`. The [Python API](api.md) describes
+loading, resolution, activation, and restoration.

docs/secrets.md

@@ -30,13 +30,13 @@ decrypt values**.
 
 ---
 
-## Supported encryption algorithms
+## Supported encryption and encoding
 
-envstack supports the following encryption algorithms:
+envstack supports the following encryption and encoding methods:
 
 | Algorithm | Key required | Environment variable |
 |---------|--------------|----------------------|
-| Base64  | No           | (default)            |
+| Base64 encoding (not encryption) | No | (default) |
 | AES-GCM | Yes          | `ENVSTACK_SYMMETRIC_KEY` |
 | Fernet  | Yes          | `ENVSTACK_FERNET_KEY` |
 
@@ -151,4 +151,7 @@ Values will automatically decrypt during resolution.
 - Use includes to inject keys at activation time
 - Combine with external secret managers for higher security needs
 
-envstack encryption is intentionally simple, explicit, and inspectable.
+envstack encryption is intentionally simple, explicit, and inspectable.
+
+For YAML tag syntax and read/write timing, see
+[YAML transformation tags](expansion.md#yaml-transformation-tags).

lib/envstack/config.py

@@ -61,7 +61,7 @@ def detect_shell():
 DEBUG = os.getenv("DEBUG")
 
 # default namespace
-DEFAULT_NAMESPACE = os.getenv("DEFAULT_ENV_STACK", "default")
+DEFAULT_NAMESPACE = os.getenv("DEFAULT_NAMESPACE", "default")
 
 # allow embedded commands
 ALLOW_COMMANDS = os.getenv("ALLOW_COMMANDS", "0") in ("1", "true", "True", "TRUE")