Skip to content

Fix blocksToDelete shipper initialization race - #7753

Open
Rohithmatham12 wants to merge 2 commits into
cortexproject:masterfrom
Rohithmatham12:fix/6413-blocks-to-delete-race
Open

Rohithmatham12 wants to merge 2 commits into
cortexproject:masterfrom
Rohithmatham12:fix/6413-blocks-to-delete-race

Conversation

@Rohithmatham12

Copy link
Copy Markdown
Contributor

What this changes

  • Initializes the per-tenant Thanos shipper before assigning userDB.db in createTSDB.
  • Keeps blocksToDelete from observing a partially initialized userTSDB where db is visible but shipper is still being written.
  • Leaves the shipped-block cache initialization after the TSDB is fully assigned, preserving the existing behavior.

Why
tsdb.Open installs userDB.blocksToDelete as a callback and can run block reload work concurrently. Previously createTSDB assigned userDB.db before creating userDB.shipper, so the callback could read userDB.shipper while the constructor wrote it. Creating the shipper before making db visible removes that race window without adding locks to the callback path.

Fixes #6413.

Testing

  • git diff --check
  • GOCACHE=/private/tmp/cortex-go-cache-6413 GOMODCACHE=/private/tmp/cortex-go-mod-cache go test -tags "netgo slicelabels" ./pkg/ingester -run "TestIngester_shipBlocks|TestIngester_dontShipBlocksWhenTenantDeletionMarkerIsPresent|TestIngester_TSDB"
  • GOCACHE=/private/tmp/cortex-go-cache-6413-full GOMODCACHE=/private/tmp/cortex-go-mod-cache go test -tags "netgo slicelabels" ./pkg/ingester

Signed-off-by: Rohithmatham12 <rohithmatham@gmail.com>
@CharlieTLe
CharlieTLe requested review from a team and alanprot and removed request for a team August 21, 2026 18:27

@sandy2008 sandy2008 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking at this. The new order closes the "db set but shipper not yet" window, but I don't think it removes the data race from #6413; details inline.

Comment thread pkg/ingester/ingester.go
userDB.shipperMetadataFilePath = filepath.Join(db.Dir(), filepath.Clean(shipper.DefaultMetaFilename))
}

userDB.db = db

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving this assignment after the shipper changes the order of two plain writes, but it doesn't synchronize them with the reader. tsdb.Open has already started the reload goroutine (BlockReloadInterval is 1m), which calls blocksToDelete and reads u.db and u.shipper without a lock or atomic, so -race still reports this write against the u.db == nil read. I reproduced it with a small probe: a goroutine calling u.blocksToDelete(nil) in a loop while the test assigns u.db = db fails under go test -race with a data race on exactly this pair. Could we publish with synchronization instead, e.g. an atomic.Bool set after db and shipper are assigned and checked at the top of blocksToDelete, and add a -race test for it?

@Rohithmatham12
Rohithmatham12 requested a review from a team as a code owner November 2, 2026 05:18
@Rohithmatham12

Copy link
Copy Markdown
Contributor Author

Thanks for catching this. I added an atomic publication flag that is stored only after db and shipper are assigned, and blocksToDelete now checks it before reading either field. I also added a concurrent publication test and ran:

go test -race -tags "netgo slicelabels" ./pkg/ingester -run "TestIngester_UserTSDB_BlocksToDelete" -count=1

It passes at 4294d28fbf.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Race condition on blocksToDelete

2 participants