feat!: consolidate storage schemas and add Redis provider (MAPCO-11262) - #218
Open
almog8k wants to merge 12 commits into
Open
feat!: consolidate storage schemas and add Redis provider (MAPCO-11262)#218almog8k wants to merge 12 commits into
almog8k wants to merge 12 commits into
Conversation
`constants/export/constants.ts` imported `pickEnum` from the `utils` barrel, which pulls in `layer.utils` -> `schemas` barrel while `constants` is still initializing. Deep imports of 9 of 11 schema modules threw "Cannot access '<x>' before initialization"; only the package entry point worked, since `src/index.ts` loads `./utils` before `./constants`. Import from `utils/helpers.utils` directly, matching what `constants/core/constants.ts` already does. All 11 schema modules now load when imported directly.
Introduce a `StorageProvider` discriminator covering only backends that have a matching `storageSchema` member (FS, S3, REDIS) — deliberately not spread from `SourceType`, which also contains GPKG, a source format rather than a storage backend. Split the per-provider locator fields into reusable schemas (`fsSubPathSchema`, `s3BucketSchema`, `redisPrefixSchema`) and add `redisStorageSchema` to the `storageSchema` union, keyed on a Redis key prefix.
Replace `tilesDeletionParamsBaseSchema` and its `sourceProvider` union with two composable pieces: `tileRangesSchema` (the ranges to delete) and `tilePyramidSchema` (`tilesPath` + `fileExtension`, meaningful only for path-based tile stores). The provider-specific unions move to the deletion scope, where they can be keyed on `storageProvider` alongside the other deletion params.
Add `DeletionJobTypes.Cache_Deletion` and `DeletionTaskTypes.CacheDeletion`
('cache-deletion') for wiping a layer's Redis-backed tile cache.
Consolidate the deletion task params on the `storageProvider` discriminator: - `tilesDeletionParamsSchema` — S3/FS/Redis. The FS member intentionally omits `subPath`, since tile deletion locates tiles by `tilesPath`. - `deleteStoredResourcesParamsSchema` — S3/FS/Redis. The Redis member is `.strict()`, rejecting `paths` outright: for a key-value store the prefix is the locator, so `paths` is meaningless rather than merely unused. - `redisCacheDeletionParamsSchema` — params of the single `cache-deletion` task. The two shapes are told apart structurally: `ranges` present means range-based deletion, `prefix` alone means a whole-layer wipe. Both members are `.strict()`, so at most one can ever match. Both Redis members accept an optional `delaySeconds` to wait out a mapproxy reload before deleting.
Merged
|
Hi it is me, your friendly bot helper! 👋 {
"dependencies": {
"@map-colonies/raster-shared": "https://ghatmpstorage.blob.core.windows.net/npm-packages/raster-shared-a022e3faf8e92677e7990d96437cb3d41a2fa685.tgz"
}
}The link will expire in a week ⌛ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Consolidates the deletion schemas around a single
storageProviderdiscriminator, and adds Redis as a storage provider so a layer's Redis-backed tile cache can be deleted through the same shapes as FS and S3.Why
Tile deletion and stored-resource deletion each handled providers their own way — tiles keyed on
sourceProvider, stored resources on an intersection withstorageSchema. Adding Redis to both would have meant duplicating that logic twice. Instead there is now one discriminator, with each provider's params composed from small reusable pieces.Changes
StorageProvider(FS | S3 | REDIS) becomes the storage discriminator, replacingSourceType. It is deliberately not spread fromSourceType, which also containsGPKG— a source format rather than a storage backend, and one that no storage schema accepts.redisStorageSchemajoinsstorageSchema, located by a Redis key prefix.tileRangesSchema(which tiles) andtilesInfoSchema(tilesRelativePath+fileExtension, meaningful only for path-based stores), combined astilePyramidSchema.storageProviderwith an S3 / FS / Redis member each, and each member is exported as its own type.delaySeconds, to wait out a mapproxy reload before the keys are removed. Both are.strict(), so path-store fields such aspathsare rejected outright rather than silently stripped — for a key-value store the prefix is the locator.Cache_Deletionjob type andcache-deletiontask type added.Also included: an import-cycle fix
constants/export/constants.tsimportedpickEnumthrough theutilsbarrel, which pulls inlayer.utils→ theschemasbarrel whileconstantsis still initializing. As a result 9 of 11 schema modules threwCannot access '<x>' before initializationwhen imported directly — only the package entry point worked, because it happens to loadutilsfirst. It now imports the module directly, matching whatconstants/core/constants.tsalready did. All 11 modules load.This is why the package currently has no schema tests: a spec importing any schema module hits the cycle.
Breaking changes
sourceProvider→storageProvidertilesPath→tilesRelativePathtilesDeletionParamsSchema/TilesDeletionParamsmoved from the core scope to deletiontilesDeletionParamsBaseSchema,deleteStoredResourcesParamsBaseSchema