job_runs: add lifecycle.triggers.on_file_change by radakam · Pull Request #6309 · databricks/cli

radakam · 2026-08-18T12:30:32Z

Changes

Add lifecycle.triggers.on_file_change for resources.job_runs (direct engine).

resources:
  job_runs:
    migrate:
      job_id: ${resources.jobs.migrate.id}
      lifecycle:
        triggers:
          - on_file_change: migrations/*.sql

Resolve globs under the sync root and fingerprint matched files by content hash (mtime-only touch does not re-fire).
Re-fire when a hash changes or when matches appear/disappear; removing the trigger is a state-only update (no new run).
Skip files that sync would skip (.gitignore and sync.exclude).
Reject paths outside the sync root, empty patterns, directory-only matches, and combining with prevent_destroy.

Why

Users need to re-run a job when local inputs change (SQL migrations, notebooks, configs) without re-firing on every deploy. Content hashes keep the trigger stable under sync noise while still catching real edits.

Tests

Unit: resolve (glob merge, missing match, path escape, directory-only, gitignore, sync.exclude), validate, PrepareState / OverrideChangeDesc.
Acceptance:
- on_file_change — deploy, unchanged redeploy, touch, edit → recreate (including READPLAN), delete/restore file, clear trigger → update-only.
- on_file_change_glob — migrations/* plus mysubdir/*.txt; appear/disappear; nested dir and gitignored files do not re-fire; ** is not recursive; READPLAN.
- on_file_change_errors — path outside sync root, directory-only glob, no-match warning.

eng-dev-ecosystem-bot · 2026-08-18T13:54:48Z

Integration test report

Commit: 786a2d2

Run: 32369688487

	Env	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
💚	aws linux		1	4	274	1165	6:17
💚	aws windows		1	4	276	1163	3:48
🔄	azure linux	2	1	4	271	1165	5:25
💚	azure windows		1	4	275	1163	3:22
💚	gcp linux		1	4	274	1165	5:38
💚	gcp windows		1	4	276	1163	3:37

7 interesting tests: 4 SKIP, 2 flaky, 1 RECOVERED

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚	TestAccept	💚R	💚R	💚R	💚R	💚R	💚R
🙈	TestAccept/bundle/invariant/no_drift	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🔄	TestFsCpFileToDirWithOverwriteFlag	✅p	✅p	🔄f	✅p	✅p	✅p
🔄	TestFsCpFileToDirWithOverwriteFlag/uc-volumes_to_local	✅p	✅p	🔄f	✅p	✅p	✅p

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
3:51	gcp linux	TestAccept
3:44	azure linux	TestAccept
3:40	aws linux	TestAccept
3:14	azure windows	TestAccept
3:03	aws windows	TestAccept
3:03	gcp windows	TestAccept

github-actions · 2026-08-19T08:05:38Z

Approval status: pending

`/acceptance/bundle/` - needs approval

24 files changed
Suggested: @janniklasrose
Also eligible: @denik, @pietern, @andrewnester, @shreyas-goenka, @lennartkats-db, @anton-107

`/bundle/` - needs approval

11 files changed
Suggested: @janniklasrose
Also eligible: @denik, @pietern, @andrewnester, @shreyas-goenka, @lennartkats-db, @anton-107

General files (require maintainer)

Files: .nextchanges/bundles/job-runs-on-file-change.md
Based on git history:

@janniklasrose -- recent work in bundle/config/mutator/, bundle/schema/, bundle/internal/schema/

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db, @rugpanov, @rclarey) can approve all areas.

See OWNERS for ownership rules.}

Re-fire a run when matched file contents change, using stable hash fingerprints so mtime-only updates (touch) do not recreate.

Satisfy the linter on PrepareState literals after adding OnFileChange, and refresh out.fields.txt so validate-generated stays green.

Reject patterns outside the sync root and directory-only matches so the trigger cannot fingerprint escaped paths or silently disarm.

Drop size/mtime fingerprints and the resolve fast path so content identity alone drives recreate, and touch no longer needs a planner exception.

The modernize linter rejects the manual m[k]=v copy loop, failing CI lint.

denik · 2026-08-19T10:37:48Z

+	}
+	defer f.Close()
+
+	h := sha256.New()


This implementation is fine but it also does way too much potentially.

We could also do this:

store stat(x).st_size and stat(x).st_mtime in the state in addition to hash

check stat values first:
= st_size is different? can trigger update without calculating the hash
= st_size the same and t_mtime is the same? can skip hashing, file was not touched so hash can be assumed the same.

However, let's discuss this first, because we do something similar in file upload so I wonder if we should just re-use the mechanism there.

I tried size+mtime in state in an earlier revision. touch then showed up as local drift, so we needed ignore_local_changes on mtime_ns to prevent it from recreating. Since mtime isn’t part of the recreate condition, putting it in planned state felt less elegant than just hashing the content. I also wouldn’t reuse file upload for this: sync is mtime-only, so a touch would trigger a re-upload and re-fire the job. What do you think?

right, let's keep your implementation it's simple and you only pay for files you track. We can add local optimization later, maybe using cache that we have.

denik · 2026-08-19T16:46:54Z

+  "lifecycle": {
+    "action": "recreate",
+    "reason": "immutable",
+    "old": {


Could you look into why the change key is "lifecycle" and not a more detailed one like 'lifecycle.triggers.on_file_change["seed.txt"]'.

I supposed it is because remote is nil for the whole lifecycle? perhaps it can be hacked to have empty on_file_changes map?

The leaf is already at lifecycle.triggers.on_file_change['seed.txt'], and the extra "lifecycle" key is the same change at the subtree root -- I think structdiff emits both

we should not have both, it's not helpful there.

Honor .gitignore and sync.exclude when hashing matches. Cover mysubdir globs, READPLAN, and keep LF on Windows so restored files hash cleanly.

The glob test edits migrations/ignored.txt, but the test .gitignore excluded it from the commit so CI clones would miss the file.

Plan serialization is already covered by on_file_change; the glob cases only assert the hash map.

Cover deploying glob hashes from a saved plan, not only from a live re-plan.

denik · 2026-08-20T12:51:03Z

+  "lifecycle": {
+    "action": "recreate",
+    "reason": "immutable",
+    "old": {


we should not have both, it's not helpful there.

denik · 2026-08-20T12:52:37Z

+          "mysubdir/a.txt": "[FILE_HASH][2]"
+        }
+      }
+    }


try initializing remote object with {"lifecycle": {triggers: on_file_change: {}}, I think that can help to get rid of this block.

or maybe we need to fix diffing in general to recognize such cases.

denik · 2026-08-20T12:53:59Z

@@ -0,0 +1,212 @@
+package mutator_test
+
+import (


do these tests add anything on top of acc tests?

denik · 2026-08-20T12:55:53Z

+		}
+	}
+	if len(input.ResolvedFileTriggers) > 0 {
+		if triggers == nil {


do we need all this conditions? would it be bad tojust create struct unconditionally?

radakam marked this pull request as ready for review August 19, 2026 08:04

Base automatically changed from deco-job-runs-lifecycle-on-bundle-deploy to main August 19, 2026 09:41

radakam force-pushed the deco-job-runs-lifecycle-on-file-change branch from b8c20f6 to 8744e5e Compare August 19, 2026 09:47

radakam requested a review from denik August 19, 2026 11:06

radakam added 7 commits August 19, 2026 11:36

job_runs: add lifecycle.triggers.on_file_change

be0eca0

Re-fire a run when matched file contents change, using stable hash fingerprints so mtime-only updates (touch) do not recreate.

job_runs: fix exhaustruct and regenerate refschema for on_file_change

926b8cf

Satisfy the linter on PrepareState literals after adding OnFileChange, and refresh out.fields.txt so validate-generated stays green.

job_runs: tighten on_file_change path and directory checks

f181266

Reject patterns outside the sync root and directory-only matches so the trigger cannot fingerprint escaped paths or silently disarm.

job_runs: store on_file_change as path-to-hash only

7255a17

Drop size/mtime fingerprints and the resolve fast path so content identity alone drives recreate, and touch no longer needs a planner exception.

job_runs: use maps.Copy in on_file_change hash merge

a02fd3d

The modernize linter rejects the manual m[k]=v copy loop, failing CI lint.

job_runs: drop filepath.Glob jargon from on_file_change schema docs

43cf3cf

acc: refresh on_file_change goldens for resource-key progress lines

47994d8

radakam force-pushed the deco-job-runs-lifecycle-on-file-change branch from 8744e5e to 47994d8 Compare August 19, 2026 11:39

denik reviewed Aug 19, 2026

View reviewed changes

acc: cover on_file_change globs, errors, and appear/disappear

132ba7b

denik reviewed Aug 19, 2026

View reviewed changes

radakam added 3 commits August 19, 2026 19:58

acc: expand on_file_change glob coverage and skip ignored files

75eaf87

Honor .gitignore and sync.exclude when hashing matches. Cover mysubdir globs, READPLAN, and keep LF on Windows so restored files hash cleanly.

acc: force-add gitignored on_file_change fixture

4e43352

The glob test edits migrations/ignored.txt, but the test .gitignore excluded it from the commit so CI clones would miss the file.

acc: drop READPLAN from the on_file_change glob test

a7a3275

Plan serialization is already covered by on_file_change; the glob cases only assert the hash map.

radakam requested a review from denik August 20, 2026 08:12

denik reviewed Aug 20, 2026

View reviewed changes

Comment thread acceptance/bundle/resources/job_runs/on_file_change_glob/test.toml

radakam requested a review from denik August 20, 2026 09:27

acc: restore READPLAN on the on_file_change glob test

786a2d2

Cover deploying glob hashes from a saved plan, not only from a live re-plan.

denik reviewed Aug 20, 2026

View reviewed changes

Uh oh!

Conversation

radakam commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

eng-dev-ecosystem-bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

github-actions Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approval status: pending

/acceptance/bundle/ - needs approval

/bundle/ - needs approval

General files (require maintainer)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

radakam commented Aug 18, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Aug 18, 2026 •

edited

Loading

github-actions Bot commented Aug 19, 2026 •

edited

Loading

`/acceptance/bundle/` - needs approval

`/bundle/` - needs approval