Skip to content

feat(jinja): add b64decode/b64encode builtin filters#5863

Merged
StuffbyYuki merged 1 commit into
SQLMesh:mainfrom
anxkhn:loop/sqlmesh__002
Jun 30, 2026
Merged

feat(jinja): add b64decode/b64encode builtin filters#5863
StuffbyYuki merged 1 commit into
SQLMesh:mainfrom
anxkhn:loop/sqlmesh__002

Conversation

@anxkhn

@anxkhn anxkhn commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Description

Core SQLMesh ships no builtin Jinja filters, so there is no in-tree way to decode a base64-encoded value inside a config YAML. As issue #5754 notes, this is awkward for secrets that are commonly stored base64-encoded in environment variables (for example a GCP/BigQuery service-account key), since they cannot be decoded where the connection config is rendered.

This adds two builtin filters, b64decode and b64encode (matching ansible's filter names referenced in the issue), and registers them on the shared Jinja environment() factory in sqlmesh/utils/jinja.py. Because ENVIRONMENT is built from that factory and config files are rendered through it, the filters work in config YAML:

env_vars:
  BIGQUERY_KEY: "{{ env_var('BIGQUERY_KEY_B64') | b64decode }}"

They are also available in models, since the model JinjaMacroRegistry builds its environment from the same factory. Both helpers are str/bytes tolerant and return str. This mirrors the existing dbt builtin-filter pattern (create_builtin_filters in sqlmesh/dbt/builtin.py).

Closes #5754

Test Plan

  • Added test_builtin_base64_filters in tests/utils/test_jinja.py: renders b64decode/b64encode (and a round-trip) via a built environment, and asserts the filter is usable in config YAML through yaml.load.
  • Confirmed the new test fails on main (No filter named 'b64decode') and passes on the branch.
  • pytest tests/utils/test_jinja.py -> 14 passed; dbt jinja/filter tests + tests/utils/test_yaml.py pass.
  • ruff check, ruff format --check, mypy sqlmesh/utils/jinja.py clean.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

@StuffbyYuki StuffbyYuki self-requested a review June 29, 2026 19:39

@StuffbyYuki StuffbyYuki left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anxkhn Thanks for your work on this!

Optional (non-blocking):

  • Add a short note in docs/guides/configuration.md (near the env_var examples) showing {{ env_var('BIGQUERY_KEY_B64') | b64decode }} for base64-encoded secrets.
  • Consider a test that uses env_var + b64decode together (mock/set env) to match the real-world use case.
  • A brief docstring on b64decode noting it returns UTF-8 text (intended for JSON/string secrets, not arbitrary binary) would help future users.

Core SQLMesh shipped no builtin Jinja filters, so base64-encoded secrets
(e.g. a BigQuery service-account key stored in an env var) could not be
decoded directly in config YAML. Add b64decode/b64encode and register them
on the shared environment factory so they are available both in config YAML
and in models, mirroring ansible's filters.

Closes SQLMesh#5754

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
@anxkhn anxkhn force-pushed the loop/sqlmesh__002 branch from fe28674 to 62e5d1b Compare June 30, 2026 04:20
@StuffbyYuki StuffbyYuki merged commit ffb30e4 into SQLMesh:main Jun 30, 2026
32 checks passed
EhabEasee pushed a commit to EhabEasee/sqlmesh that referenced this pull request Jun 30, 2026
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Signed-off-by: EhabEasee <ehab.elbadrawi@easee.com>
EhabEasee pushed a commit to EhabEasee/sqlmesh that referenced this pull request Jun 30, 2026
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Signed-off-by: EhabEasee <ehab.elbadrawi@easee.com>
@JohnSchlerf

Copy link
Copy Markdown

This is amazing. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Request: Support base64 decode as a Jinja plugin (for configuration yaml files)

3 participants