feat(jinja): add b64decode/b64encode builtin filters#5863
Merged
Conversation
StuffbyYuki
approved these changes
Jun 29, 2026
StuffbyYuki
left a comment
Collaborator
There was a problem hiding this comment.
@anxkhn Thanks for your work on this!
Optional (non-blocking):
- Add a short note in
docs/guides/configuration.md(near the env_var examples) showing{{ env_var('BIGQUERY_KEY_B64') | b64decode }}for base64-encoded secrets. - Consider a test that uses env_var + b64decode together (mock/set env) to match the real-world use case.
- A brief docstring on b64decode noting it returns UTF-8 text (intended for JSON/string secrets, not arbitrary binary) would help future users.
Core SQLMesh shipped no builtin Jinja filters, so base64-encoded secrets (e.g. a BigQuery service-account key stored in an env var) could not be decoded directly in config YAML. Add b64decode/b64encode and register them on the shared environment factory so they are available both in config YAML and in models, mirroring ansible's filters. Closes SQLMesh#5754 Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
fe28674 to
62e5d1b
Compare
EhabEasee
pushed a commit
to EhabEasee/sqlmesh
that referenced
this pull request
Jun 30, 2026
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com> Signed-off-by: EhabEasee <ehab.elbadrawi@easee.com>
EhabEasee
pushed a commit
to EhabEasee/sqlmesh
that referenced
this pull request
Jun 30, 2026
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com> Signed-off-by: EhabEasee <ehab.elbadrawi@easee.com>
|
This is amazing. Thank you! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Core SQLMesh ships no builtin Jinja filters, so there is no in-tree way to decode a base64-encoded value inside a config YAML. As issue #5754 notes, this is awkward for secrets that are commonly stored base64-encoded in environment variables (for example a GCP/BigQuery service-account key), since they cannot be decoded where the connection config is rendered.
This adds two builtin filters,
b64decodeandb64encode(matching ansible's filter names referenced in the issue), and registers them on the shared Jinjaenvironment()factory insqlmesh/utils/jinja.py. BecauseENVIRONMENTis built from that factory and config files are rendered through it, the filters work in config YAML:They are also available in models, since the model
JinjaMacroRegistrybuilds its environment from the same factory. Both helpers are str/bytes tolerant and returnstr. This mirrors the existing dbt builtin-filter pattern (create_builtin_filtersinsqlmesh/dbt/builtin.py).Closes #5754
Test Plan
test_builtin_base64_filtersintests/utils/test_jinja.py: rendersb64decode/b64encode(and a round-trip) via a built environment, and asserts the filter is usable in config YAML throughyaml.load.main(No filter named 'b64decode') and passes on the branch.pytest tests/utils/test_jinja.py-> 14 passed; dbt jinja/filter tests +tests/utils/test_yaml.pypass.ruff check,ruff format --check,mypy sqlmesh/utils/jinja.pyclean.Checklist
make styleand fixed any issuesmake fast-test)git commit -s) per the DCO