Skip to content

feat!: consolidate storage schemas and add Redis provider (MAPCO-11262) - #218

Open
almog8k wants to merge 12 commits into
alphafrom
feat/consolidate-storage-schemas-redis-MAPCO-11262
Open

feat!: consolidate storage schemas and add Redis provider (MAPCO-11262)#218
almog8k wants to merge 12 commits into
alphafrom
feat/consolidate-storage-schemas-redis-MAPCO-11262

Conversation

@almog8k

@almog8k almog8k commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator
Question Answer
New feature
Breaking change

What

Consolidates the deletion schemas around a single storageProvider discriminator, and adds Redis as a storage provider so a layer's Redis-backed tile cache can be deleted through the same shapes as FS and S3.

Why

Tile deletion and stored-resource deletion each handled providers their own way — tiles keyed on sourceProvider, stored resources on an intersection with storageSchema. Adding Redis to both would have meant duplicating that logic twice. Instead there is now one discriminator, with each provider's params composed from small reusable pieces.

Changes

  • StorageProvider (FS | S3 | REDIS) becomes the storage discriminator, replacing SourceType. It is deliberately not spread from SourceType, which also contains GPKG — a source format rather than a storage backend, and one that no storage schema accepts.
  • redisStorageSchema joins storageSchema, located by a Redis key prefix.
  • Tile schemas split into composable parts: tileRangesSchema (which tiles) and tilesInfoSchema (tilesRelativePath + fileExtension, meaningful only for path-based stores), combined as tilePyramidSchema.
  • Both deletion param schemas are now discriminated unions over storageProvider with an S3 / FS / Redis member each, and each member is exported as its own type.
  • The two Redis shapes share a base carrying the prefix plus an optional delaySeconds, to wait out a mapproxy reload before the keys are removed. Both are .strict(), so path-store fields such as paths are rejected outright rather than silently stripped — for a key-value store the prefix is the locator.
  • Cache_Deletion job type and cache-deletion task type added.

Also included: an import-cycle fix

constants/export/constants.ts imported pickEnum through the utils barrel, which pulls in layer.utils → the schemas barrel while constants is still initializing. As a result 9 of 11 schema modules threw Cannot access '<x>' before initialization when imported directly — only the package entry point worked, because it happens to load utils first. It now imports the module directly, matching what constants/core/constants.ts already did. All 11 modules load.

This is why the package currently has no schema tests: a spec importing any schema module hits the cycle.

Breaking changes

  • Tiles deletion discriminator renamed: sourceProviderstorageProvider
  • Field renamed: tilesPathtilesRelativePath
  • tilesDeletionParamsSchema / TilesDeletionParams moved from the core scope to deletion
  • Removed: tilesDeletionParamsBaseSchema, deleteStoredResourcesParamsBaseSchema

almog8k added 11 commits August 3, 2026 12:38
`constants/export/constants.ts` imported `pickEnum` from the `utils` barrel,
which pulls in `layer.utils` -> `schemas` barrel while `constants` is still
initializing. Deep imports of 9 of 11 schema modules threw "Cannot access
'<x>' before initialization"; only the package entry point worked, since
`src/index.ts` loads `./utils` before `./constants`.

Import from `utils/helpers.utils` directly, matching what
`constants/core/constants.ts` already does. All 11 schema modules now load
when imported directly.
Introduce a `StorageProvider` discriminator covering only backends that have a
matching `storageSchema` member (FS, S3, REDIS) — deliberately not spread from
`SourceType`, which also contains GPKG, a source format rather than a storage
backend.

Split the per-provider locator fields into reusable schemas (`fsSubPathSchema`,
`s3BucketSchema`, `redisPrefixSchema`) and add `redisStorageSchema` to the
`storageSchema` union, keyed on a Redis key prefix.
Replace `tilesDeletionParamsBaseSchema` and its `sourceProvider` union with two
composable pieces: `tileRangesSchema` (the ranges to delete) and
`tilePyramidSchema` (`tilesPath` + `fileExtension`, meaningful only for
path-based tile stores).

The provider-specific unions move to the deletion scope, where they can be keyed
on `storageProvider` alongside the other deletion params.
Add `DeletionJobTypes.Cache_Deletion` and `DeletionTaskTypes.CacheDeletion`
('cache-deletion') for wiping a layer's Redis-backed tile cache.
Consolidate the deletion task params on the `storageProvider` discriminator:

- `tilesDeletionParamsSchema` — S3/FS/Redis. The FS member intentionally omits
  `subPath`, since tile deletion locates tiles by `tilesPath`.
- `deleteStoredResourcesParamsSchema` — S3/FS/Redis. The Redis member is
  `.strict()`, rejecting `paths` outright: for a key-value store the prefix is
  the locator, so `paths` is meaningless rather than merely unused.
- `redisCacheDeletionParamsSchema` — params of the single `cache-deletion` task.
  The two shapes are told apart structurally: `ranges` present means range-based
  deletion, `prefix` alone means a whole-layer wipe. Both members are `.strict()`,
  so at most one can ever match.

Both Redis members accept an optional `delaySeconds` to wait out a mapproxy
reload before deleting.
@almog8k almog8k changed the title feat: consolidate storage schemas and add Redis provider (MAPCO-11262) feat!: consolidate storage schemas and add Redis provider (MAPCO-11262) Aug 3, 2026
@almog8k almog8k mentioned this pull request Aug 3, 2026
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

Hi it is me, your friendly bot helper! 👋
I've packed this commit for you 😊
You can install it like this:

{
  "dependencies": {
      "@map-colonies/raster-shared": "https://ghatmpstorage.blob.core.windows.net/npm-packages/raster-shared-a022e3faf8e92677e7990d96437cb3d41a2fa685.tgz"
  }
}

The link will expire in a week ⌛

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant