diff --git a/AGENTS.md b/AGENTS.md index 2ad6369..0d24898 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,6 +25,7 @@ When adding or moving pages: 3. Keep pages grouped by product area (`datasets/`, `workflows/`, `sdks/`, `guides/`, `api-reference/`). 4. Keep the User Guides nav and the [Tilebox Cookbook](/guides/cookbook) in sync. Whenever you add, remove, rename, or move a `guides/**` page in `docs.json`, update `guides/cookbook.mdx` with the same guide metadata, and vice versa. 5. Preserve the current pattern where high-level landing pages link to deeper pages via `Card`/`HeroCard` blocks. +6. Give overview and landing pages an icon distinct from the icon of their containing navigation group. ## Diátaxis Mapping diff --git a/api-reference/python/tilebox.datasets.assets/Asset.mdx b/api-reference/python/tilebox.datasets.assets/Asset.mdx new file mode 100644 index 0000000..5bbfb1f --- /dev/null +++ b/api-reference/python/tilebox.datasets.assets/Asset.mdx @@ -0,0 +1,70 @@ +--- +title: Asset +icon: file +--- + +```python +class Asset( + key: str, + primary: AssetLocation, + alternates: Mapping[str, AssetLocation] = {}, + media_type: MediaType | str | None = None, + title: str | None = None, + description: str | None = None, + roles: frozenset[KnownAssetRole | str] = frozenset(), + gsd: float | None = None, + bands: tuple[Band, ...] = (), + data_type: DataType = DataType.UNSPECIFIED, + nodata: float | None = None, + statistics: Statistics | None = None, + unit: str | None = None, + eo: EOProperties | None = None, + raster: RasterProperties | None = None, + projection: Projection | None = None, + view: View | None = None, + classes: tuple[ClassificationClass, ...] = (), + file: File | None = None, + sar: SARProperties | None = None, + satellite: SatelliteProperties | None = None, + product: ProductProperties | None = None, +) +``` + +Describe an asset and its primary and alternate locations. + +## Fields + + + The asset's unique key within its datapoint. + + + The primary asset location. + + + Alternate locations keyed by their STAC alternate-assets key. + + + The exact media type. + + + Known or custom STAC asset roles. + + + Ordered band metadata. + + +The other fields contain optional STAC metadata defined by extensions such as [Electro-Optical](https://github.com/stac-extensions/eo), [Projection](https://github.com/stac-extensions/projection), and [Raster](https://github.com/stac-extensions/raster). + +## Media types + +Import `MediaType` with the authoring types from `tilebox.datasets.assets`. Its members are strings, such as `MediaType.GEOTIFF`, `MediaType.CLOUD_OPTIMIZED_GEOTIFF`, `MediaType.PNG`, and `MediaType.NETCDF`. You can also pass a custom media type string. + +```python Python +from tilebox.datasets.assets import Asset, AssetLocation, MediaType + +asset = Asset( + key="image", + primary=AssetLocation("s3://bucket/image.tif"), + media_type=MediaType.CLOUD_OPTIMIZED_GEOTIFF, +) +``` diff --git a/api-reference/python/tilebox.datasets.assets/AssetCollection.from_assets.mdx b/api-reference/python/tilebox.datasets.assets/AssetCollection.from_assets.mdx new file mode 100644 index 0000000..b4682ff --- /dev/null +++ b/api-reference/python/tilebox.datasets.assets/AssetCollection.from_assets.mdx @@ -0,0 +1,23 @@ +--- +title: AssetCollection.from_assets +icon: folder-open +--- + +```python +@classmethod +def AssetCollection.from_assets( + assets: Iterable[Asset], +) -> AssetCollection +``` + +Create a normalized semantic asset collection. + +## Parameters + + + Assets with nonempty, unique keys. + + +## Returns + +A read-only `AssetCollection` keyed by each asset's key. diff --git a/api-reference/python/tilebox.datasets.assets/AssetCollection.from_datapoint.mdx b/api-reference/python/tilebox.datasets.assets/AssetCollection.from_datapoint.mdx new file mode 100644 index 0000000..3efbee2 --- /dev/null +++ b/api-reference/python/tilebox.datasets.assets/AssetCollection.from_datapoint.mdx @@ -0,0 +1,29 @@ +--- +title: AssetCollection.from_datapoint +icon: folder-open +--- + +```python +@classmethod +def AssetCollection.from_datapoint( + datapoint: xarray.Dataset, + *, + fields: AssetFieldNames | None = None, +) -> AssetCollection +``` + +Resolve the assets attached to exactly one scalar xarray datapoint. + +## Parameters + + + A scalar dataset containing exactly one datapoint. + + + Optional names for the `assets`, `storage`, and `authentication` variables + when automatic discovery is ambiguous. + + +## Returns + +An `AssetCollection` keyed by asset key. diff --git a/api-reference/python/tilebox.datasets.assets/AssetCollection.mdx b/api-reference/python/tilebox.datasets.assets/AssetCollection.mdx new file mode 100644 index 0000000..24afa90 --- /dev/null +++ b/api-reference/python/tilebox.datasets.assets/AssetCollection.mdx @@ -0,0 +1,12 @@ +--- +title: AssetCollection +icon: folder-open +--- + +```python +class AssetCollection(Mapping[str, Asset]) +``` + +Represent the assets belonging to one dataset datapoint as a read-only mapping keyed by asset key. + +Create a collection with `AssetCollection.from_assets`, resolve one from a scalar datapoint with `AssetCollection.from_datapoint`, or convert it to ingestion fields with `AssetCollection.to_fields`. diff --git a/api-reference/python/tilebox.datasets.assets/AssetCollection.to_fields.mdx b/api-reference/python/tilebox.datasets.assets/AssetCollection.to_fields.mdx new file mode 100644 index 0000000..30bd9ce --- /dev/null +++ b/api-reference/python/tilebox.datasets.assets/AssetCollection.to_fields.mdx @@ -0,0 +1,31 @@ +--- +title: AssetCollection.to_fields +icon: folder-open +--- + +```python +def AssetCollection.to_fields( + *, + fields: AssetFieldNames | None = None, + storage: Storage | None = None, + authentication: Authentication | None = None, +) -> dict[str, Assets | Storage | Authentication] +``` + +Convert the collection to fields ready for dataset ingestion. + +## Parameters + + + Optional output names for the assets, storage, and authentication fields. + + + Additional storage registry entries to retain. + + + Additional authentication registry entries to retain. + + +## Returns + +A field-name mapping containing `Assets` and any nonempty `Storage` or `Authentication` fields. diff --git a/api-reference/python/tilebox.datasets.assets/AssetLocation.mdx b/api-reference/python/tilebox.datasets.assets/AssetLocation.mdx new file mode 100644 index 0000000..e12c884 --- /dev/null +++ b/api-reference/python/tilebox.datasets.assets/AssetLocation.mdx @@ -0,0 +1,33 @@ +--- +title: AssetLocation +icon: location-dot +--- + +```python +class AssetLocation( + href: str, + alternate_name: str | None = None, + storage_schemes: Mapping[str, StorageScheme] = {}, + authentication_schemes: Mapping[str, AuthenticationScheme] = {}, +) +``` + +Describe an asset URL and the storage and authentication schemes that apply to it. + +## Fields + + + The fully resolved asset URL. + + + The optional STAC alternate-assets display name. + + + Storage schemes keyed by their exact registry keys. + + + Authentication schemes keyed by their exact registry keys. + diff --git a/api-reference/python/tilebox.datasets.assets/Band.mdx b/api-reference/python/tilebox.datasets.assets/Band.mdx new file mode 100644 index 0000000..2547fd3 --- /dev/null +++ b/api-reference/python/tilebox.datasets.assets/Band.mdx @@ -0,0 +1,40 @@ +--- +title: Band +icon: layer-group +--- + +```python +class Band( + name: str | None = None, + description: str | None = None, + data_type: DataType = DataType.UNSPECIFIED, + nodata: float | None = None, + unit: str | None = None, + eo: EOProperties | None = None, + raster: RasterProperties | None = None, + classes: tuple[ClassificationClass, ...] = (), + sar: SARProperties | None = None, +) +``` + +Describe one asset band, with unspecified values inherited from the asset when an `AssetCollection` is created. + +## Fields + + + The band name. + + + A human-readable description. + + + The raster data type. + + + The nodata value. + + + The unit name. + + +The `eo`, `raster`, `classes`, and `sar` fields hold generated extension metadata. diff --git a/api-reference/python/tilebox.datasets/Client.create_or_update_dataset.mdx b/api-reference/python/tilebox.datasets/Client.create_or_update_dataset.mdx index 632afe4..5327101 100644 --- a/api-reference/python/tilebox.datasets/Client.create_or_update_dataset.mdx +++ b/api-reference/python/tilebox.datasets/Client.create_or_update_dataset.mdx @@ -75,6 +75,24 @@ a new queryable field is only supported while the dataset is empty. A geometry field + + STAC asset metadata. Import from `tilebox.datasets.schema`. + + + STAC authentication metadata. Import from `tilebox.datasets.schema`. + + + STAC link metadata. Import from `tilebox.datasets.schema`. + + + STAC processing software metadata. Import from `tilebox.datasets.schema`. + + + STAC provider metadata. Import from `tilebox.datasets.schema`. + + + STAC storage metadata. Import from `tilebox.datasets.schema`. + Note that the type can also be a list of one of the types, indicating that the field is an array, e.g. `list[str]`. diff --git a/api-reference/python/tilebox.datasets/Collection.ingest.mdx b/api-reference/python/tilebox.datasets/Collection.ingest.mdx index b788261..de0b43d 100644 --- a/api-reference/python/tilebox.datasets/Collection.ingest.mdx +++ b/api-reference/python/tilebox.datasets/Collection.ingest.mdx @@ -24,10 +24,13 @@ Ingest data into a collection. The data to ingest. Supported `IngestionData` data types are: - - A `pandas.DataFrame`, mapping the column names to dataset fields. - - An `xarray.Dataset`, mapping variables and coordinates to dataset fields. - - An `Iterable`, `dict` or `nd-array`: ingest any object that can be converted to a `pandas.DataFrame` using - its constructor, equivalent to `ingest(pd.DataFrame(data))`. + - An iterable of mappings, with one mapping per datapoint. + - A mapping from field names to ordered sequences, `numpy.ndarray` objects, or `pandas.Series` objects. + - A `pandas.DataFrame`, with column names mapped to dataset fields. + - An `xarray.Dataset`, with variables and coordinates mapped to dataset fields. + + A mapping is always interpreted as column-oriented data. Wrap a single record in an iterable, such as `[record]`. + Every datapoint must include `time`. Tilebox generates `id` and `ingestion_time`. Missing optional values leave their corresponding fields unset. Datapoint fields are used to generate a deterministic unique `UUID` for each @@ -47,16 +50,18 @@ List of datapoint IDs that were ingested, including the IDs of existing data poi ```python Python -import pandas as pd - -collection.ingest(pd.DataFrame({ - "time": [ - "2023-05-01T12:00:00Z", - "2023-05-02T12:00:00Z", - ], - "value": [1, 2], - "sensor": ["A", "B"], -})) +collection.ingest([ + { + "time": "2023-05-01T12:00:00Z", + "value": 1, + "sensor": "A", + }, + { + "time": "2023-05-02T12:00:00Z", + "value": 2, + "sensor": "B", + }, +]) ``` diff --git a/api-reference/python/tilebox.storage.aio/AssetAccessPolicy.mdx b/api-reference/python/tilebox.storage.aio/AssetAccessPolicy.mdx new file mode 100644 index 0000000..4b91a34 --- /dev/null +++ b/api-reference/python/tilebox.storage.aio/AssetAccessPolicy.mdx @@ -0,0 +1,20 @@ +--- +title: AssetAccessPolicy +icon: list-ol +--- + +```python +class AssetAccessPolicy( + preferred_schemes: tuple[str, ...] = ( + "file", "s3", "gs", "az", "https", "http" + ), +) +``` + +Control the order in which `Client.resolve` considers asset location schemes. + +## Fields + + + URI schemes in descending preference order. + diff --git a/api-reference/python/tilebox.storage.aio/Client.download.mdx b/api-reference/python/tilebox.storage.aio/Client.download.mdx new file mode 100644 index 0000000..ba7d191 --- /dev/null +++ b/api-reference/python/tilebox.storage.aio/Client.download.mdx @@ -0,0 +1,31 @@ +--- +title: Client.download +icon: download +--- + +```python +async def Client.download( + asset: Asset, + destination: str | PathLike[str], + *, + overwrite: bool = False, +) -> Path +``` + +Atomically download an asset to an exact local path. + +## Parameters + + + The asset to resolve and download. + + + The destination path. Missing parent directories are created. + + + Whether to replace an existing destination. Defaults to `False`. + + +## Returns + +The destination as a `Path`. diff --git a/api-reference/python/tilebox.storage.aio/Client.iter_bytes.mdx b/api-reference/python/tilebox.storage.aio/Client.iter_bytes.mdx new file mode 100644 index 0000000..a190692 --- /dev/null +++ b/api-reference/python/tilebox.storage.aio/Client.iter_bytes.mdx @@ -0,0 +1,22 @@ +--- +title: Client.iter_bytes +icon: hard-drive +--- + +```python +async def Client.iter_bytes( + asset: Asset, +) -> AsyncIterator[bytes] +``` + +Stream an asset as provider-dependent byte chunks. + +## Parameters + + + The asset to resolve and stream. + + +## Yields + +Byte chunks whose sizes are selected by the storage provider. diff --git a/api-reference/python/tilebox.storage.aio/Client.mdx b/api-reference/python/tilebox.storage.aio/Client.mdx new file mode 100644 index 0000000..1fa41a8 --- /dev/null +++ b/api-reference/python/tilebox.storage.aio/Client.mdx @@ -0,0 +1,27 @@ +--- +title: Client +icon: hard-drive +--- + +```python +class Client( + *, + policy: AssetAccessPolicy | None = None, +) +``` + +Create an asynchronous client that resolves and accesses dataset assets through reusable object stores. + +## Parameters + + + The location-selection policy. The default prefers local, S3, Google Cloud, + Azure, HTTPS, and HTTP locations in that order. + + +```python Python +from tilebox.storage.aio import Client + +client = Client() +data = await client.read_bytes(asset) +``` diff --git a/api-reference/python/tilebox.storage.aio/Client.open_geotiff.mdx b/api-reference/python/tilebox.storage.aio/Client.open_geotiff.mdx new file mode 100644 index 0000000..33e8ae4 --- /dev/null +++ b/api-reference/python/tilebox.storage.aio/Client.open_geotiff.mdx @@ -0,0 +1,33 @@ +--- +title: Client.open_geotiff +icon: map +--- + +```python +async def Client.open_geotiff( + asset: Asset, + *, + prefetch: int = 32768, + multiplier: float = 2.0, +) -> GeoTIFF +``` + +Open a GeoTIFF through its resolved object store without reading pixel data. + +This method requires Python 3.11 or newer. + +## Parameters + + + The GeoTIFF asset to resolve and open. + + + The initial metadata range size in bytes. Defaults to `32768`. + + + The growth factor for later metadata range requests. Defaults to `2.0`. + + +## Returns + +An async-geotiff `GeoTIFF` backed by the resolved object store. diff --git a/api-reference/python/tilebox.storage.aio/Client.read_bytes.mdx b/api-reference/python/tilebox.storage.aio/Client.read_bytes.mdx new file mode 100644 index 0000000..a4d4234 --- /dev/null +++ b/api-reference/python/tilebox.storage.aio/Client.read_bytes.mdx @@ -0,0 +1,27 @@ +--- +title: Client.read_bytes +icon: hard-drive +--- + +```python +async def Client.read_bytes( + asset: Asset, + *, + max_bytes: int | None = None, +) -> bytes +``` + +Read an entire asset into memory. + +## Parameters + + + The asset to resolve and read. + + + An optional maximum accepted object size in bytes. + + +## Returns + +The complete object contents as `bytes`. diff --git a/api-reference/python/tilebox.storage.aio/Client.resolve.mdx b/api-reference/python/tilebox.storage.aio/Client.resolve.mdx new file mode 100644 index 0000000..dc43e8a --- /dev/null +++ b/api-reference/python/tilebox.storage.aio/Client.resolve.mdx @@ -0,0 +1,20 @@ +--- +title: Client.resolve +icon: hard-drive +--- + +```python +def Client.resolve(asset: Asset) -> ResolvedAsset +``` + +Select the best supported asset location without making a network request. + +## Parameters + + + The asset to resolve. + + +## Returns + +A `ResolvedAsset` containing the selected location, object store, path, and access metadata. diff --git a/api-reference/python/tilebox.storage.aio/ResolvedAsset.mdx b/api-reference/python/tilebox.storage.aio/ResolvedAsset.mdx new file mode 100644 index 0000000..fe273cc --- /dev/null +++ b/api-reference/python/tilebox.storage.aio/ResolvedAsset.mdx @@ -0,0 +1,42 @@ +--- +title: ResolvedAsset +icon: file-circle-check +--- + +```python +class ResolvedAsset( + asset: Asset, + location: AssetLocation, + href: str, + store: ObjectStore, + path: str, + storage_scheme: StorageScheme | None, + authentication_scheme: AuthenticationScheme | None, +) +``` + +Describe the selected asset location and object-store access details returned by `Client.resolve`. + +## Fields + + + The source asset. + + + The selected location. + + + The selected location's URL. + + + The configured object store. + + + The object path within the store. + + + The applicable storage scheme. + + + The applicable authentication scheme. + diff --git a/api-reference/python/tilebox.storage.aio/window_from_bounds.mdx b/api-reference/python/tilebox.storage.aio/window_from_bounds.mdx new file mode 100644 index 0000000..207f555 --- /dev/null +++ b/api-reference/python/tilebox.storage.aio/window_from_bounds.mdx @@ -0,0 +1,41 @@ +--- +title: window_from_bounds +icon: crop-simple +--- + +```python +def window_from_bounds( + geotiff: GeoTIFF, + bounds: tuple[float, float, float, float], + *, + crs: str | int | CRS | Proj, + require_fully_contained: bool = False, +) -> Window +``` + +Convert geographic bounds to an outward-rounded GeoTIFF pixel window clipped to the image. + + + This function requires Python 3.11 or newer and is imported from + `tilebox.storage.geotiff`. + + +## Parameters + + + The open async-geotiff image. + + + Coordinates in `(left, bottom, right, top)` order. + + + The coordinate reference system of `bounds`. + + + Whether to reject bounds that extend beyond the image instead of clipping + them. Defaults to `False`. + + +## Returns + +An async-geotiff `Window` containing every pixel touched by the bounds. diff --git a/assets/changelog/2026-07-27-assets.webp b/assets/changelog/2026-07-27-assets.webp new file mode 100644 index 0000000..a02eee0 Binary files /dev/null and b/assets/changelog/2026-07-27-assets.webp differ diff --git a/assets/changelog/2026-07-31-queryable-fields.webp b/assets/changelog/2026-07-31-queryable-fields.webp new file mode 100644 index 0000000..933774f Binary files /dev/null and b/assets/changelog/2026-07-31-queryable-fields.webp differ diff --git a/assets/changelog/2026-08-01-sentinel2-aws-earth.webp b/assets/changelog/2026-08-01-sentinel2-aws-earth.webp new file mode 100644 index 0000000..cf7a3e5 Binary files /dev/null and b/assets/changelog/2026-08-01-sentinel2-aws-earth.webp differ diff --git a/assets/datasets/assets-and-storage/overview-dark.png b/assets/datasets/assets-and-storage/overview-dark.png new file mode 100644 index 0000000..fdf0653 Binary files /dev/null and b/assets/datasets/assets-and-storage/overview-dark.png differ diff --git a/assets/datasets/assets-and-storage/overview-light.png b/assets/datasets/assets-and-storage/overview-light.png new file mode 100644 index 0000000..aaa9348 Binary files /dev/null and b/assets/datasets/assets-and-storage/overview-light.png differ diff --git a/changelog.mdx b/changelog.mdx index 0bfd863..2aaaef1 100644 --- a/changelog.mdx +++ b/changelog.mdx @@ -5,9 +5,33 @@ icon: rss mode: center --- - + + ## Sentinel-2 imagery, ready to query and read + + + Access Sentinel-2 imagery credentials-free with Tilebox + + + The new `open_data.aws_earth.sentinel2` dataset provides credentials-free access to Sentinel-2 metadata and Cloud Optimized GeoTIFFs (COGs). You can start working with satellite imagery using only your Tilebox API key—no external data provider account, separate credentials, or storage configuration required. + + Query Level-2A scenes by time, location, cloud cover, or satellite platform, then resolve a result directly into Tilebox assets. From the same datapoint, you can inspect the available spectral bands, download a complete image, or open a COG remotely and read only the pixels covering your area of interest. + + This makes it practical to move from catalog search to image processing in one workflow: find a low-cloud scene, select a band, crop it to a geographic region, and pass the resulting data into your analysis without first downloading an entire scene. + + + + Query Sentinel-2 scenes, resolve their assets, and read or download image data. + + + + + ## Queryable custom dataset fields + + Query sentinel-2 dataset with custom filters on cloud_cover + + Custom dataset schemas can now mark selected fields as queryable. Tilebox evaluates these field expressions on the server together with temporal, spatial, and collection filters, so clients only receive matching datapoints. @@ -37,13 +61,33 @@ mode: center - + + ## Assets and storage access + - Deploying a workflow release to a cluster and rolling the cluster back to an earlier release in the Tilebox Console + Assets and storage access in Tilebox + Dataset datapoints can now describe file assets and their storage locations. The Python SDK resolves that metadata into asset collections, and the asynchronous storage client can stream, download, or open those assets across supported storage providers. + + The GeoTIFF integration also supports remote COG access and window reads, so you can fetch only the pixels needed for an area of interest. + + Provider-specific storage clients are now deprecated. Migration guidance for existing integrations will follow. + + + + Learn how dataset metadata connects to files in object storage. + + + + + ## Workflow Management and Operations + + Deploying a workflow release to a cluster and rolling the cluster back to an earlier release in the Tilebox Console + + Tilebox now exposes more workflow operations across the Console, SDKs, CLI, MCP server, and runner deployments. You can manage workflow releases and clusters from the Console, filter jobs by compute location, schedule automations in local time, and start release runners from an official Tilebox container image. ### What changed diff --git a/datasets/assets-and-storage/overview.mdx b/datasets/assets-and-storage/overview.mdx new file mode 100644 index 0000000..e08e72a --- /dev/null +++ b/datasets/assets-and-storage/overview.mdx @@ -0,0 +1,38 @@ +--- +title: Assets in Tilebox +sidebarTitle: Overview +description: Understand how Tilebox datapoints reference files in external storage. +icon: diagram-project +--- + +Assets connect searchable Tilebox datapoints to files such as images and previews. Tilebox stores those references with the datapoint, while the files remain in their original storage. + + + An application queries datapoint metadata from Tilebox and fetches the referenced files from an external storage bucket + An application queries datapoint metadata from Tilebox and fetches the referenced files from an external storage bucket + + +## Reusable file access model + +A structured asset model lets the Tilebox storage client access files through one interface. The same client reads assets from Tilebox open data datasets and from datasets you create that reference files in private buckets. + +## Assets belong to datapoints + +A datapoint can reference named assets such as `red`, `nir`, or `thumbnail`. Each asset contains a primary location, optional alternate locations, and metadata such as its media type, roles, or bands. + +Asset locations can reference storage and authentication schemes. These describe provider settings such as region and requester-pays behavior and how access is authenticated, allowing the storage client to select and configure a compatible location. + +Assets can also include metadata compatible with common STAC extensions, including [Electro-Optical](https://github.com/stac-extensions/eo), [Projection](https://github.com/stac-extensions/projection), and [Raster](https://github.com/stac-extensions/raster). + +In Python, [`AssetCollection`](/api-reference/python/tilebox.datasets.assets/AssetCollection) provides read-only, key-based access to the assets of one datapoint. + +## Work with assets + + + + Query an asset-enabled dataset, then read, stream, download, or open its files. + + + Add references to your own files when ingesting datapoints. + + diff --git a/datasets/assets-and-storage/read-and-download.mdx b/datasets/assets-and-storage/read-and-download.mdx new file mode 100644 index 0000000..21df02a --- /dev/null +++ b/datasets/assets-and-storage/read-and-download.mdx @@ -0,0 +1,157 @@ +--- +title: Read and download assets +description: Use the Python storage client to read, stream, download, and open files referenced by Tilebox datapoints. +icon: download +--- + +The storage client reads assets from local files, S3, Google Cloud Storage, Azure, and HTTP locations. It selects a compatible location from the metadata attached to each asset. + +## Resolve assets from a datapoint + +Query an asset-enabled dataset and select one datapoint. See [Querying data](/datasets/query/querying-data) for the complete query API. + +```python Python +from shapely import box +from tilebox.datasets import Client, field +from tilebox.datasets.assets import AssetCollection + +collection = Client().dataset("open_data.aws_earth.sentinel2").collection("L2A") +data = collection.query( + temporal_extent=("2026-07-20", "2026-07-28"), + spatial_extent=box(16.25, 48.15, 16.35, 48.22), + filter=field("cloud_cover") < 10, +) + +assets = AssetCollection.from_datapoint(data.isel(time=0)) +print(list(assets)) +red = assets["red"] +thumbnail = assets["thumbnail"] +``` + +An asset exposes its media type, roles, bands, primary location, and alternate locations. `from_datapoint` accepts one selected datapoint; for multiple results, resolve each datapoint separately. + +Create one client and reuse it across assets so it can reuse the underlying object stores: + +```python Python +from tilebox.storage.aio import Client as StorageClient + +storage = StorageClient() +await storage.download(thumbnail, "thumbnail.jpg") +``` + +## Storage operations + +### Read an asset into memory + +Use `read_bytes` for small files. Set `max_bytes` to reject unexpectedly large objects. + +```python Python +content = await storage.read_bytes(thumbnail, max_bytes=10_000_000) +``` + +### Stream asset bytes + +Use `iter_bytes` when you can process the file incrementally. + +```python Python +async for chunk in storage.iter_bytes(red): + process(chunk) +``` + +### Download an asset + +`download` writes atomically to the exact destination path and does not replace an existing file unless requested. + +```python Python +path = await storage.download(red, "data/red.tif") +``` + +### Open a GeoTIFF + +`open_geotiff` opens TIFF metadata without downloading the complete file. The [COG and GeoTIFF section](#work-with-cog-and-geotiff-assets) shows how to read a region. + +```python Python +geotiff = await storage.open_geotiff(red) +``` + +### Inspect the selected location + +`resolve` selects a location without making a network request. Most code can let the other operations call it automatically. + +```python Python +resolved = storage.resolve(red) +print(resolved.href, resolved.path) +``` + +## Work with COG and GeoTIFF assets + +A Cloud Optimized GeoTIFF (COG) supports range requests, so you can read the pixels for one region without downloading the complete image. Use `window_from_bounds` to convert geographic bounds into a pixel window. + +```python Python +from tilebox.storage.geotiff import window_from_bounds + +geotiff = await storage.open_geotiff(red) +window = window_from_bounds( + geotiff, + bounds=(16.25, 48.15, 16.35, 48.22), # west, south, east, north + crs="EPSG:4326", +) +pixels = await geotiff.read(window=window) +``` + +The storage client does not apply scale, offset, no-data masks, band stacking, or coordinate transformations. GeoTIFF access requires Python 3.11 or newer. + +## Read RGB bands in parallel + +Use `asyncio.gather` to fetch independent assets concurrently. This example reads the same 512 × 512 pixel window from the red, green, and blue COGs and stacks the results into an RGB array. + +```python Python +import asyncio + +import numpy as np +from async_geotiff import Window + +window = Window(col_off=4096, row_off=4096, width=512, height=512) + +async def read_band(key): + geotiff = await storage.open_geotiff(assets[key]) + raster = await geotiff.read(window=window) + return raster.data[0] + +red_data, green_data, blue_data = await asyncio.gather( + read_band("red"), + read_band("green"), + read_band("blue"), +) +rgb = np.stack((red_data, green_data, blue_data), axis=-1) +print(rgb.shape) # (512, 512, 3) +``` + +## Choose a location + +Assets can provide primary and alternate locations. By default, the client prefers local files, S3, Google Cloud Storage, Azure, HTTPS, and HTTP, in that order. Configure an [`AssetAccessPolicy`](/api-reference/python/tilebox.storage.aio/AssetAccessPolicy) to change that order. + +```python Python +from tilebox.storage.aio import AssetAccessPolicy, Client as StorageClient + +storage = StorageClient( + policy=AssetAccessPolicy(preferred_schemes=("https", "s3")), +) +``` + +Storage metadata can provide regions, endpoints, and requester-pays settings. + +## Authentication + +Asset locations can reference authentication metadata. The storage client currently supports S3 credentials from the standard AWS credential environment. Authenticated HTTP locations are not yet supported. + +## Next steps + + + + Query Sentinel-2 and read a selected image region. + + + Attach file references to datapoints you ingest. + + diff --git a/datasets/assets-and-storage/reference-assets.mdx b/datasets/assets-and-storage/reference-assets.mdx new file mode 100644 index 0000000..bd1ee28 --- /dev/null +++ b/datasets/assets-and-storage/reference-assets.mdx @@ -0,0 +1,156 @@ +--- +title: Reference assets in a dataset +sidebarTitle: Reference your assets +description: Add references to files in external storage when ingesting Tilebox datapoints. +icon: link +--- + +Reference files that already exist in object storage, behind HTTP URLs, or on a local filesystem by adding asset fields to your dataset schema and ingestion records. + +## Add structured STAC fields + +Tilebox provides the following structured field types for STAC-compatible datasets. See [dataset field types](/datasets/concepts/datasets#field-types) for the complete list of supported types. + +| Field type | Purpose | +| --- | --- | +| `Assets` | References files associated with a datapoint, including their locations, media types, roles, and optional metadata. | +| `Storage` | Describes reusable storage schemes. | +| `Authentication` | Describes reusable access methods. | +| `Links` | References related STAC resources. | +| `Provider` | Identifies organizations that produce, process, license, or host data. | +| `ProcessingSoftware` | Records software and versions used to process data. | + +Both `Assets` and `Links` can reference entries in the storage and authentication registries. In Python, `AssetCollection` combines assets with optional storage and authentication entries needed to access them. + +Add the assets, storage, and authentication fields explicitly when you create the dataset: + + +```python Python +from tilebox.datasets import Client +from tilebox.datasets.data.datasets import DatasetKind +from tilebox.datasets.schema import Assets, Authentication, Storage + +client = Client() +dataset = client.create_or_update_dataset( + kind=DatasetKind.SPATIOTEMPORAL, + code_name="imagery_catalog", + fields=[ + {"name": "product_id", "type": str}, + {"name": "assets", "type": Assets}, + {"name": "storage", "type": Storage}, + {"name": "authentication", "type": Authentication}, + ], + name="Imagery catalog", +) +collection = dataset.get_or_create_collection("products") +``` + +```go Go +import ( + "github.com/tilebox/tilebox-go/datasets/v1" + "github.com/tilebox/tilebox-go/datasets/v1/field" + stacv1 "github.com/tilebox/tilebox-go/protogen/datasets/stac/v1" +) + +fields := []datasets.Field{ + field.String("product_id"), + field.Message("assets", &stacv1.Assets{}), + field.Message("storage", &stacv1.Storage{}), + field.Message("authentication", &stacv1.Authentication{}), +} + +dataset, err := client.Datasets.CreateOrUpdate( + ctx, + datasets.KindSpatiotemporal, + "imagery_catalog", + "Imagery catalog", + fields, +) +``` + + +## Prepare assets for ingestion + +Construct assets from their source locations, then check and normalize the collection. This ensures that the storage client can read the referenced bytes. `to_fields()` converts the collection into fields for a complete datapoint record. + + +```python Python +from tilebox.datasets.assets import ( + Asset, + AssetCollection, + AssetLocation, + MediaType, +) +from shapely import box + +assets = AssetCollection.from_assets([ + Asset( + key="image", + primary=AssetLocation("s3://example-bucket/scenes/scene-1.tif"), + media_type=MediaType.CLOUD_OPTIMIZED_GEOTIFF, + roles=frozenset({"data"}), + ), +]) + +record = { + "time": "2026-07-31T10:00:00Z", + "geometry": box(16.25, 48.15, 16.35, 48.22), + "product_id": "scene-1", + **assets.to_fields(), +} + +collection.ingest([record]) +``` + +```go Go +import stacv1 "github.com/tilebox/tilebox-go/protogen/datasets/stac/v1" + +profileIndex := uint32(0) +href := "scenes/scene-1.tif" +mediaType := stacv1.KnownMediaType_KNOWN_MEDIA_TYPE_CLOUD_OPTIMIZED_GEOTIFF + +assets := stacv1.Assets_builder{ + AccessProfiles: []*stacv1.AssetAccessProfile{ + stacv1.AssetAccessProfile_builder{ + BaseHref: "https://example-bucket.s3.eu-central-1.amazonaws.com/", + }.Build(), + }, + Assets: []*stacv1.Asset{ + stacv1.Asset_builder{ + Key: "image", + Primary: stacv1.AssetLocation_builder{ + AccessProfileIndex: &profileIndex, + Href: &href, + }.Build(), + MediaType: stacv1.MediaType_builder{Known: &mediaType}.Build(), + Roles: []stacv1.KnownAssetRole{ + stacv1.KnownAssetRole_KNOWN_ASSET_ROLE_DATA, + }, + }.Build(), + }, +}.Build() + +// Use the generated datapoint type for your dataset. +record := catalogv1.Scene_builder{ + Time: timestamp, + Geometry: geometry, + ProductId: new("scene-1"), + Assets: assets, +}.Build() +``` + + +Pass the complete record to the [standard datapoint ingestion API](/datasets/ingest). + +## Add more asset metadata + +Assets can also describe alternate locations, bands, and metadata from STAC extensions such as Raster, Electro-Optical, and Projection. See the [`Asset` API reference](/api-reference/python/tilebox.datasets.assets/Asset) for the available fields. + + + + Create a dataset that combines searchable metadata with file references. + + + Access the referenced files with the storage client. + + diff --git a/datasets/concepts/datasets.mdx b/datasets/concepts/datasets.mdx index 5629646..d3cb16a 100644 --- a/datasets/concepts/datasets.mdx +++ b/datasets/concepts/datasets.mdx @@ -4,10 +4,6 @@ description: Datasets are strongly typed containers in which every data point wi icon: database --- - - You can create your own, Custom Datasets via the [Tilebox Console](/console). - - ## Related Guides @@ -90,6 +86,21 @@ When defining the data schema, you can specify each field's type. The following | --- | --- | --- | | Geometry | Geospatial geometries of type Point, LineString, Polygon or MultiPolygon. | `POLYGON ((12.3 -5.4, 12.5 -5.4, ...))` | +### STAC metadata + +Structured STAC types preserve common metadata without flattening it into separate custom fields. In Python, import these types from `tilebox.datasets.schema`. + +| Type | Description | Example contents | +| --- | --- | --- | +| Assets | Files associated with a datapoint, based on the [STAC Asset Object](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#asset-object). | Image bands and thumbnails | +| Authentication | Authentication schemes described by the [STAC Authentication Extension](https://github.com/stac-extensions/authentication). | S3 credentials | +| Links | Relationships to other resources, based on the [STAC Link Object](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#link-object). | Canonical and related links | +| ProcessingSoftware | Software and versions described by the [STAC Processing Extension](https://github.com/stac-extensions/processing). | Processor name and version | +| Provider | Organizations that produced, processed, hosted, or licensed the data. | Producer and host details | +| Storage | Storage-system details described by the [STAC Storage Extension](https://github.com/stac-extensions/storage). | S3 region and requester-pays settings | + +See [Assets and storage](/datasets/assets-and-storage/overview) for the asset access model. + ### Arrays Every type is also available as an array, allowing to ingest multiple values of the underlying type for each data point. The size of the array is flexible, and can be different for each data point. diff --git a/datasets/ingest.mdx b/datasets/ingest.mdx index 6ae7f29..423983f 100644 --- a/datasets/ingest.mdx +++ b/datasets/ingest.mdx @@ -76,148 +76,103 @@ func main() { ``` -## Preparing data for ingestion +## Prepare data for ingestion -Ingestion can be done either in Python or Go. +Ingestion is available in Python and Go. ### Python -[`collection.ingest`](/api-reference/python/tilebox.datasets/Collection.ingest) supports a wide range of input types. Below is an example of using either a `pandas.DataFrame` or an `xarray.Dataset` as input. +Every datapoint passed to [`collection.ingest`](/api-reference/python/tilebox.datasets/Collection.ingest) must include `time`. Omit `id` and `ingestion_time`; Tilebox generates both fields during ingestion. -#### pandas DataFrame - -A [pandas.DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) is a representation of two-dimensional, potentially heterogeneous tabular data. It's a powerful tool for working with structured data, and Tilebox supports it as input for `ingest`. +#### Record-oriented data -The example below shows how to construct a `pandas.DataFrame` from scratch, that matches the schema of the `MyCustomDataset` dataset and can be ingested into it. +Use an iterable of mappings when you construct datapoints individually. Optional fields can be absent from individual records. `None` and common tabular missing values also leave optional fields unset. - ```python Python -import pandas as pd +records = [ + { + "time": "2025-03-28T11:44:23Z", + "value": 45.16, + "sensor": "A", + "sensor_history": [-12.15, 13.45, -8.2, 16.5, 45.16], + }, + { + "time": "2025-03-28T11:45:19Z", + "value": 273.15, + "sensor": "B", + }, +] + +datapoint_ids = collection.ingest(records) +``` -data = pd.DataFrame({ +#### Column-oriented data + +Use a mapping of field names to equally sized sequences when your data is already organized by column. + +```python Python +columns = { "time": [ - "2025-03-28T11:44:23Z", - "2025-03-28T11:45:19Z", + "2025-03-28T11:44:23Z", + "2025-03-28T11:45:19Z", ], "value": [45.16, 273.15], "sensor": ["A", "B"], - "precise_time": [ - "2025-03-28T11:44:23.345761444Z", - "2025-03-28T11:45:19.128742312Z", - ], - "sensor_history": [ - [-12.15, 13.45, -8.2, 16.5, 45.16], - [300.16, 280.12, 273.15], - ], -}) -print(data) -``` - +} - -```plaintext Output - time value sensor precise_time sensor_history -0 2025-03-28T11:44:23Z 45.16 A 2025-03-28T11:44:23.345761444Z [-12.15, 13.45, -8.2, 16.5, 45.16] -1 2025-03-28T11:45:19Z 273.15 B 2025-03-28T11:45:19.128742312Z [300.16, 280.12, 273.15] +collection.ingest(columns) ``` - -Once you have the data ready in this format, you can `ingest` it into a collection. + + A mapping is always interpreted as column-oriented data. To ingest one record, wrap it in a list: `collection.ingest([record])`. + + +#### pandas DataFrame + +Tilebox treats each DataFrame row as one datapoint and maps column names to dataset fields. - ```python Python -# now that we have the data frame in the correct format -# we can ingest it into the Tilebox dataset -collection.ingest(data) +import pandas as pd -# To verify it now contains the 2 data points -print(collection.info()) -``` - +data = pd.DataFrame({ + "time": [ + "2025-03-28T11:44:23Z", + "2025-03-28T11:45:19Z", + ], + "value": [45.16, 273.15], + "sensor": ["A", "B"], +}) - -```plaintext Output -Measurements: [2025-03-28T11:44:23.000 UTC, 2025-03-28T11:45:19.000 UTC] (2 data points) +collection.ingest(data) ``` - - - - You can now also head on over to the [Tilebox Console](/console) and view the newly ingested data points there. - #### xarray Dataset -[`xarray.Dataset`](/sdks/python/xarray) is the default format in which Tilebox Datasets returns data when -[querying data](/datasets/query/querying-data) from a collection. -Tilebox also supports it as input for ingestion. The example below shows how to construct an `xarray.Dataset` -from scratch, that matches the schema of the `MyCustomDataset` dataset and can then be ingested into it. -To learn more about `xarray.Dataset`, visit Tilebox dedicated [Xarray documentation page](/sdks/python/xarray). +Tilebox also accepts [`xarray.Dataset`](/sdks/python/xarray), the format returned when [querying data](/datasets/query/querying-data). - ```python Python -import pandas as pd +import numpy as np +import xarray as xr data = xr.Dataset({ "time": ("time", [ - "2025-03-28T11:46:13Z", - "2025-03-28T11:46:54Z", + "2025-03-28T11:46:13Z", + "2025-03-28T11:46:54Z", ]), "value": ("time", [48.1, 290.12]), - "sensor": ("time", ["A", "B"]), - "precise_time": ("time", [ - "2025-03-28T11:46:13.345761444Z", - "2025-03-28T11:46:54.128742312Z", - ]), "sensor_history": (("time", "n_sensor_history"), [ - [13.45, -8.2, 16.5, 45.16, 48.1], - [280.12, 273.15, 290.12, np.nan, np.nan], + [13.45, -8.2, 16.5, 45.16, 48.1], + [280.12, 273.15, 290.12, np.nan, np.nan], ]), }) -print(data) -``` - - -```plaintext Output - Size: 504B -Dimensions: (time: 2, n_sensor_history: 5) -Coordinates: - * time (time) - Array fields manifest in xarray using an extra dimension, in this case `n_sensor_history`. In case - of different array sizes for each data point, remaining values are filled up with a fill value, depending on the - `dtype` of the array. For `float64` this is `np.nan` (not a number). - Don't worry - when ingesting data into a Tilebox dataset, Tilebox will automatically skip those padding fill values - and not store them in the dataset. + Array fields use an extra xarray dimension, such as `n_sensor_history`. If array lengths differ, pad shorter values at the end with the fill value for that data type. Tilebox omits this trailing padding during ingestion. -Now that you have the `xarray.Dataset` in the correct format, you can ingest it into the Tilebox dataset collection. - - -```python Python -collection = dataset.get_or_create_collection("OtherMeasurements") -collection.ingest(data) - -# To verify it now contains the 2 data points -print(collection.info()) -``` - - - -```plaintext Output -OtherMeasurements: [2025-03-28T11:46:13.000 UTC, 2025-03-28T11:46:54.000 UTC] (2 data points) -``` - - ### Go [`Client.Datapoints.Ingest`](/api-reference/go/datasets/Datapoints.Ingest) supports ingestion of data points in the form of a slice of protobuf messages. @@ -400,6 +355,10 @@ formats, such as CSV, [Parquet](https://parquet.apache.org/), [Feather](https:// Check out the [Ingestion from common file formats](/guides/datasets/ingest-format) guide for examples of how to achieve this. +## Assets + +To ingest datapoints that reference files in external storage, see [Reference assets in a dataset](/datasets/assets-and-storage/reference-assets). + ## Geometries Ingesting Geometries can traditionally be a bit tricky, especially when working with geometries that cross the antimeridian or cover a pole. diff --git a/datasets/introduction.mdx b/datasets/introduction.mdx index 5f6bd0e..c5d72ce 100644 --- a/datasets/introduction.mdx +++ b/datasets/introduction.mdx @@ -25,6 +25,9 @@ Learn more about datasets by exploring the following sections: Learn how to ingest data into a collection. + + Connect dataset metadata to files in object storage. + diff --git a/datasets/query/querying-data.mdx b/datasets/query/querying-data.mdx index 496d38d..bdc578f 100644 --- a/datasets/query/querying-data.mdx +++ b/datasets/query/querying-data.mdx @@ -113,6 +113,8 @@ if err != nil { ``` +Some datasets include assets that point to files in object storage. After selecting one datapoint, use an [asset collection and the storage client](/datasets/assets-and-storage/read-and-download) to read or download those files. + To learn more about how to narrow down query results, see the following pages about filtering by time, geometry, custom fields, or datapoint ID. diff --git a/datasets/storage/clients.mdx b/datasets/storage/clients.mdx index 5aec9b0..844dfb2 100644 --- a/datasets/storage/clients.mdx +++ b/datasets/storage/clients.mdx @@ -1,9 +1,13 @@ --- -title: Storage Clients -description: Configure and use storage clients in the Tilebox Python SDK to access satellite data products from public providers and local file systems. +title: Legacy storage clients +description: Deprecated provider-specific clients for downloading open data products. icon: hard-drive --- + + The provider-specific storage clients on this page are deprecated. Use [asset collections and the storage client](/datasets/assets-and-storage/overview) for new integrations. Migration guidance for existing integrations will follow. + + Tilebox does not host the actual open data satellite products but instead relies on publicly accessible storage providers for data access. Tilebox ingests available metadata as [datasets](/datasets/concepts/datasets) to enable high performance querying and structured access of the data as [xarray.Dataset](/sdks/python/xarray). diff --git a/docs.json b/docs.json index 1f556bb..c418e7f 100644 --- a/docs.json +++ b/docs.json @@ -75,7 +75,16 @@ "datasets/delete", "datasets/geometries", "datasets/open-data", - "datasets/storage/clients" + { + "group": "Assets and storage", + "icon": "boxes-stacked", + "pages": [ + "datasets/assets-and-storage/overview", + "datasets/assets-and-storage/read-and-download", + "datasets/assets-and-storage/reference-assets", + "datasets/storage/clients" + ] + } ] }, { @@ -150,7 +159,6 @@ "pages": [ "guides/datasets/query-satellite-data", "guides/datasets/access-sentinel2-data", - "guides/datasets/access-usgs-landsat-data", "guides/datasets/build-spatiotemporal-catalog", "guides/datasets/ingest-into-spatiotemporal-catalog", "guides/datasets/ingest-format" @@ -235,6 +243,32 @@ "api-reference/python/tilebox.datasets/Collection.query" ] }, + { + "group": "tilebox.datasets.assets", + "pages": [ + "api-reference/python/tilebox.datasets.assets/Asset", + "api-reference/python/tilebox.datasets.assets/AssetLocation", + "api-reference/python/tilebox.datasets.assets/Band", + "api-reference/python/tilebox.datasets.assets/AssetCollection", + "api-reference/python/tilebox.datasets.assets/AssetCollection.from_datapoint", + "api-reference/python/tilebox.datasets.assets/AssetCollection.from_assets", + "api-reference/python/tilebox.datasets.assets/AssetCollection.to_fields" + ] + }, + { + "group": "tilebox.storage.aio", + "pages": [ + "api-reference/python/tilebox.storage.aio/Client", + "api-reference/python/tilebox.storage.aio/AssetAccessPolicy", + "api-reference/python/tilebox.storage.aio/ResolvedAsset", + "api-reference/python/tilebox.storage.aio/Client.resolve", + "api-reference/python/tilebox.storage.aio/Client.read_bytes", + "api-reference/python/tilebox.storage.aio/Client.iter_bytes", + "api-reference/python/tilebox.storage.aio/Client.download", + "api-reference/python/tilebox.storage.aio/Client.open_geotiff", + "api-reference/python/tilebox.storage.aio/window_from_bounds" + ] + }, { "group": "tilebox.workflows", "pages": [ diff --git a/guides/cookbook.mdx b/guides/cookbook.mdx index 5190baa..31155ec 100644 --- a/guides/cookbook.mdx +++ b/guides/cookbook.mdx @@ -21,31 +21,22 @@ export const cookbookSections = [ tags: ["Open data", "Sentinel-2", "Metadata queries", "Spatial filters"], }, { - title: "Access Copernicus data", + title: "Access Sentinel-2 assets", href: "/guides/datasets/access-sentinel2-data", - description: "Download Copernicus product files with the storage client, using Sentinel-2 as an example.", + description: "Read a COG window or download a Sentinel-2 image with the storage client.", icon: "magnifying-glass-location", level: "Beginner", time: "10 min", - tags: ["Copernicus", "Storage clients", "Sentinel-2", "Product files"], - }, - { - title: "Access USGS Landsat data", - href: "/guides/datasets/access-usgs-landsat-data", - description: "Download USGS Landsat product files with the storage client, using Landsat 8 as an example.", - icon: "satellite-dish", - level: "Beginner", - time: "10 min", - tags: ["USGS", "Landsat 8", "Storage clients", "Product files"], + tags: ["Assets", "COG", "Sentinel-2", "Storage client"], }, { title: "Build a spatio-temporal catalog", href: "/guides/datasets/build-spatiotemporal-catalog", - description: "Create, document, ingest, and query a custom geospatial catalog with the Python SDK.", + description: "Create, document, ingest, and query a geospatial catalog with asset references.", icon: "globe", level: "Intermediate", time: "20 min", - tags: ["Spatio-temporal datasets", "Dataset schemas", "Ingestion", "Python SDK"], + tags: ["Spatio-temporal datasets", "Assets", "Ingestion", "Python SDK"], }, { title: "Ingest into a spatio-temporal catalog", diff --git a/guides/datasets/access-sentinel2-data.mdx b/guides/datasets/access-sentinel2-data.mdx index ba299d2..08bbd2b 100644 --- a/guides/datasets/access-sentinel2-data.mdx +++ b/guides/datasets/access-sentinel2-data.mdx @@ -1,19 +1,19 @@ --- -title: Access Copernicus data -description: Download Copernicus Data Space products with the Tilebox Copernicus storage client, using Sentinel-2 as an example. -icon: database +title: Access Sentinel-2 assets +description: Query Sentinel-2 metadata and read or download the corresponding image assets. +icon: satellite --- -Use this guide when you already have a Copernicus datapoint from a Tilebox metadata query and want to access the product files behind it. The example uses Sentinel-2 Level-2A data, but the same storage client pattern applies to Copernicus products supported by Tilebox. +Tilebox indexes Sentinel-2 metadata and asset locations in the `open_data.aws_earth.sentinel2` dataset. Query the metadata first, then use the storage client to read only the image data you need. -Tilebox indexes product metadata as datasets. Product files remain in the Copernicus Data Space Ecosystem, so file access uses the `CopernicusStorageClient` with Copernicus S3 credentials. + + Asset collections and the storage client are currently available in the Python SDK. + ## Prerequisites - You have a [Tilebox API key](/authentication). -- You have installed the [Python SDK](/sdks/python/install). -- You have a [Copernicus Data Space](https://dataspace.copernicus.eu/) account. -- You have generated Copernicus [S3 credentials](https://eodata-s3keysmanager.dataspace.copernicus.eu/panel/s3-credentials). +- You have installed the [Python SDK](/sdks/python/install) with Python 3.11 or newer. ```bash uv add tilebox shapely @@ -21,125 +21,88 @@ uv add tilebox shapely ## Select a Sentinel-2 datapoint -Start with a small metadata query and select one datapoint to access. For a deeper guide to open data discovery and metadata filtering, see [Query open data metadata](/guides/datasets/query-satellite-data). +Query a small time and area of interest, then select one low-cloud observation: ```python Python -from shapely import Polygon -from tilebox.datasets import Client - -area = Polygon( - [ - (-109.05, 37.0), - (-102.05, 37.0), - (-102.05, 41.0), - (-109.05, 41.0), - (-109.05, 37.0), - ] -) +from shapely import box +from tilebox.datasets import Client, field -client = Client() -collection = client.dataset("open_data.copernicus.sentinel2_msi").collection("S2A_S2MSI2A") +datasets = Client() +collection = datasets.dataset("open_data.aws_earth.sentinel2").collection("L2A") scenes = collection.query( temporal_extent=("2025-10-01", "2025-11-01"), - spatial_extent=area, - show_progress=True, + spatial_extent=box(-106.0, 38.0, -105.9, 38.1), + filter=field("cloud_cover") < 10, ) -selected = scenes.where(scenes.cloud_cover < 10, drop=True).isel(time=0) -print(selected.granule_name.item()) +datapoint = scenes.isel(time=0) +print(datapoint.stac_id.item()) ``` -## Create the Copernicus storage client - -Create a `CopernicusStorageClient` with your Copernicus S3 credentials. The optional `cache_directory` controls where downloaded files are stored locally. - -```python Python -from pathlib import Path +See [Query open data metadata](/guides/datasets/query-satellite-data) for more query patterns. -from tilebox.storage import CopernicusStorageClient +## Resolve the assets -storage = CopernicusStorageClient( - access_key="YOUR_COPERNICUS_ACCESS_KEY", - secret_access_key="YOUR_COPERNICUS_SECRET_ACCESS_KEY", - cache_directory=Path("./data"), -) -``` - - - These credentials are Copernicus Data Space S3 credentials, not your Tilebox API key. - - -## Download the complete product - -Use `download` when you need the complete Sentinel-2 product directory. The storage client resolves the product location from the Tilebox datapoint metadata and downloads the matching files into the local cache directory. +Turn the selected datapoint into an asset collection. Each asset describes one file and the locations from which it can be accessed. ```python Python -product_path = storage.download(selected) +from tilebox.datasets.assets import AssetCollection -print(f"Downloaded {product_path.name} to {product_path}") -print("Contents:") -for path in product_path.iterdir(): - print(f"- {path.relative_to(product_path)}") -``` +assets = AssetCollection.from_datapoint(datapoint) -```plaintext Output -Downloaded S2A_MSIL2A_20251002T180751_N0511_R084_T13TEE_20251002T225842.SAFE to data/Sentinel-2/MSI/L2A/2025/10/02/S2A_MSIL2A_20251002T180751_N0511_R084_T13TEE_20251002T225842.SAFE -Contents: -- manifest.safe -- GRANULE -- INSPIRE.xml -- MTD_MSIL2A.xml -- DATASTRIP -- HTML -- rep_info -- S2A_MSIL2A_20251002T180751_N0511_R084_T13TEE_20251002T225842-ql.jpg +for key, asset in assets.items(): + print(key, asset.media_type) + +red = assets["red"] ``` -## Download selected product files +## Read a Cloud Optimized GeoTIFF window -Sentinel-2 products contain many files, including metadata, masks, quicklook images, and bands at different resolutions. Use `list_objects` and `download_objects` when you only need specific files. +The Sentinel-2 image assets are Cloud Optimized GeoTIFFs (COGs). Open an image remotely and request a pixel window without downloading the complete file: ```python Python -objects = storage.list_objects(selected) - -wanted_bands = ["B02_10m", "B03_10m", "B04_10m", "B08_10m"] -band_objects = [ - obj for obj in objects - if any(band in obj for band in wanted_bands) -] - -for obj in band_objects: - print(obj) - -downloaded_files = storage.download_objects(selected, band_objects) -print(downloaded_files) +import asyncio + +from tilebox.storage.aio import Client +from tilebox.storage.geotiff import window_from_bounds + +async def read_area(): + storage = Client() + geotiff = await storage.open_geotiff(red) + window = window_from_bounds( + geotiff, + (-106.0, 38.0, -105.9, 38.1), + crs="EPSG:4326", + ) + return await geotiff.read(window=window) + +pixels = asyncio.run(read_area()) +print(pixels.shape) ``` -Use this pattern when a workflow only needs a few bands or metadata files. It reduces transfer time and local storage compared with downloading the full `.SAFE` product. +`window_from_bounds` transforms geographic bounds into the image coordinate system and clips the resulting window to the image. -## Preview the product +## Download an asset -Many Copernicus products include a quicklook image. In a notebook, use `quicklook` to display the product preview without downloading the full product first. +Use `download` when you need the complete file locally: ```python Python -storage.quicklook(selected) -``` +async def download_red_band(): + storage = Client() + return await storage.download(red, "data/sentinel-2-red.tif") - - Sentinel-2 quicklook image - +path = asyncio.run(download_red_band()) +print(path) +``` ## Next steps - - Find Copernicus products by time, location, and metadata fields. - - - Learn about the other Tilebox storage clients for open data products. + + Learn about streaming, downloads, GeoTIFF access, and location selection. - - Download Landsat product files with the USGS Landsat storage client. + + Understand how asset metadata connects dataset queries to file storage. diff --git a/guides/datasets/access-usgs-landsat-data.mdx b/guides/datasets/access-usgs-landsat-data.mdx deleted file mode 100644 index 38fa5f3..0000000 --- a/guides/datasets/access-usgs-landsat-data.mdx +++ /dev/null @@ -1,138 +0,0 @@ ---- -title: Access USGS Landsat data -description: Download USGS Landsat products with the Tilebox Landsat storage client, using Landsat 8 as an example. -icon: satellite-dish ---- - -Use this guide when you already have a Landsat datapoint from a Tilebox metadata query and want to access the product files behind it. The example uses Landsat 8 Collection 2 Level-2 surface reflectance data. - -Tilebox indexes Landsat metadata as datasets. Product files remain in the USGS public cloud archive, so file access uses the `USGSLandsatStorageClient` and your AWS requester-pays setup. - -## Prerequisites - -- You have a [Tilebox API key](/authentication). -- You have installed the [Python SDK](/sdks/python/install). -- You have AWS credentials configured in your environment. -- Your AWS account can access [requester-pays S3 buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html). - -```bash -uv add tilebox shapely -``` - - - USGS Landsat data is stored in a requester-pays S3 bucket. AWS charges for requests and data transfer according to your AWS account settings. - - -## Select a Landsat 8 datapoint - -Start with a small metadata query and select one datapoint to access. For a deeper guide to open data discovery and metadata filtering, see [Query open data metadata](/guides/datasets/query-satellite-data). - -```python Python -from shapely import Polygon -from tilebox.datasets import Client - -area = Polygon( - [ - (-109.05, 37.0), - (-102.05, 37.0), - (-102.05, 41.0), - (-109.05, 41.0), - (-109.05, 37.0), - ] -) - -client = Client() -collection = client.dataset("open_data.usgs.landsat8_oli_tirs").collection("L2_SR") - -scenes = collection.query( - temporal_extent=("2024-08-01", "2024-08-15"), - spatial_extent=area, - show_progress=True, -) - -selected = scenes.where(scenes.cloud_cover < 10, drop=True).isel(time=0) -print(selected.granule_name.item()) -``` - -## Create the Landsat storage client - -Create a `USGSLandsatStorageClient`. The client uses AWS credentials from your environment, such as `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` when needed. - -```python Python -from tilebox.storage import USGSLandsatStorageClient - -storage = USGSLandsatStorageClient() -``` - -## Download the complete product - -Use `download` when you need the complete Landsat product directory. The storage client resolves the product location from the Tilebox datapoint metadata and downloads the matching files into the local cache. - -```python Python -product_path = storage.download(selected) - -print(f"Downloaded {product_path.name} to {product_path}") -print("Contents:") -for path in product_path.iterdir(): - print(f"- {path.relative_to(product_path)}") -``` - -```plaintext Output -Downloaded LC08_L2SP_033033_20240808_20240814_02_T1 to ~/.cache/tilebox/collection02/level-2/standard/oli-tirs/2024/033/033/LC08_L2SP_033033_20240808_20240814_02_T1 -Contents: -- LC08_L2SP_033033_20240808_20240814_02_T1_SR_B1.TIF -- LC08_L2SP_033033_20240808_20240814_02_T1_SR_B2.TIF -- LC08_L2SP_033033_20240808_20240814_02_T1_SR_B3.TIF -- LC08_L2SP_033033_20240808_20240814_02_T1_SR_B4.TIF -- LC08_L2SP_033033_20240808_20240814_02_T1_SR_B5.TIF -- LC08_L2SP_033033_20240808_20240814_02_T1_SR_B6.TIF -- LC08_L2SP_033033_20240808_20240814_02_T1_SR_B7.TIF -- LC08_L2SP_033033_20240808_20240814_02_T1_QA_PIXEL.TIF -- LC08_L2SP_033033_20240808_20240814_02_T1_MTL.json -- LC08_L2SP_033033_20240808_20240814_02_T1_thumb_small.jpeg -``` - -## Download selected product files - -Landsat products contain surface reflectance bands, quality masks, thermal bands, metadata, and preview images. Use `list_objects` and `download_objects` when you only need specific files. - -```python Python -objects = storage.list_objects(selected) - -rgb_bands = ["B4", "B3", "B2"] -rgb_objects = [ - obj for obj in objects - if any(obj.endswith(f"_{band}.TIF") for band in rgb_bands) -] - -for obj in rgb_objects: - print(obj) - -downloaded_files = storage.download_objects(selected, rgb_objects) -print(downloaded_files) -``` - -Use this pattern when a workflow only needs a few bands, masks, or metadata files. It reduces transfer time and local storage compared with downloading the full product. - -## Preview the product - -Many Landsat products include a thumbnail image. In a notebook, use `quicklook` to display the product preview without downloading the full product first. - -```python Python -storage.quicklook(selected) -``` - - - USGS Landsat quicklook image - - -## Next steps - - - - Find Landsat products by time, location, and metadata fields. - - - Learn about the other Tilebox storage clients for open data products. - - diff --git a/guides/datasets/build-spatiotemporal-catalog.mdx b/guides/datasets/build-spatiotemporal-catalog.mdx index 50aabb3..a578658 100644 --- a/guides/datasets/build-spatiotemporal-catalog.mdx +++ b/guides/datasets/build-spatiotemporal-catalog.mdx @@ -6,7 +6,7 @@ icon: globe Use a spatio-temporal dataset when each datapoint has both a time and a geometry. This is useful for internal imagery catalogs, derived products, ground truth data, regions of interest, and processing outputs that need geospatial lookup. -This guide creates an imagery catalog from code. You will define the dataset schema with the Python SDK, add field descriptions and examples for generated schema documentation, create a collection, ingest geospatial metadata, and query the catalog by time, location, and custom fields. +This guide creates an imagery catalog from code. You will define the dataset schema with the Python SDK, reference the image files as assets, ingest geospatial metadata, and query the catalog by time, location, and custom fields. ## Prerequisites @@ -21,11 +21,12 @@ uv add tilebox geopandas shapely Start by choosing the spatio-temporal dataset kind and the custom fields for your catalog. Tilebox adds the required `time`, `id`, `ingestion_time`, and `geometry` fields automatically. -The example catalog tracks imagery products with a provider product ID, a storage location, cloud cover, and processing level. Field descriptions and example values become part of the generated schema documentation. +The example catalog tracks imagery products with a provider product ID, file assets, cloud cover, and processing level. Field descriptions and example values become part of the generated schema documentation. ```python Python from tilebox.datasets import Client from tilebox.datasets.data.datasets import DatasetKind +from tilebox.datasets.schema import Assets client = Client() @@ -37,10 +38,9 @@ fields = [ "example_value": "LC08_L2SP_033033_20240808_20240814_02_T1", }, { - "name": "location", - "type": str, - "description": "Storage path, object key, or provider-specific product location.", - "example_value": "s3://example-bucket/landsat/LC08_L2SP_033033_20240808_20240814_02_T1", + "name": "assets", + "type": Assets, + "description": "Files associated with the imagery product.", }, { "name": "cloud_cover", @@ -97,7 +97,7 @@ For this catalog, the complete schema includes: | `ingestion_time` | Required | No | Time when Tilebox ingested the datapoint. | | `geometry` | Required | Dedicated spatial filter | Geometry used for spatial queries. | | `product_id` | Custom | No | Stable product or scene identifier. | -| `location` | Custom | No | Storage path or provider product location. | +| `assets` | Custom | No | Files associated with the imagery product. | | `cloud_cover` | Custom | Yes | Cloud cover percentage for filtering. | | `processing_level` | Custom | Yes | Provider processing level or product type. | @@ -149,21 +149,46 @@ products = products.rename( columns={ "timestamp": "time", "scene": "product_id", - "path": "location", + "path": "source_href", } ) products = products[ - ["time", "geometry", "product_id", "location", "cloud_cover", "processing_level"] + ["time", "geometry", "product_id", "source_href", "cloud_cover", "processing_level"] ] ``` +## Add asset references + +Convert each source file into an asset collection, then add its dataset fields to the record: + +```python Python +from tilebox.datasets.assets import Asset, AssetCollection, AssetLocation, MediaType + +records = [] +for record in products.to_dict(orient="records"): + source_href = record.pop("source_href") + assets = AssetCollection.from_assets( + [ + Asset( + key="image", + primary=AssetLocation(source_href), + media_type=MediaType.CLOUD_OPTIMIZED_GEOTIFF, + roles=frozenset({"data"}), + ) + ] + ) + records.append({**record, **assets.to_fields()}) +``` + +`AssetCollection.from_assets` validates and normalizes the metadata into the structure consumed by the storage client. It does not upload the referenced file or test its availability. + ## Ingest the catalog Ingest the prepared records into a collection. ```python Python -collection.ingest(products) +collection.ingest(records) ``` ## Query by time, location, and custom fields diff --git a/guides/datasets/query-satellite-data.mdx b/guides/datasets/query-satellite-data.mdx index d417192..584af04 100644 --- a/guides/datasets/query-satellite-data.mdx +++ b/guides/datasets/query-satellite-data.mdx @@ -1,12 +1,10 @@ --- title: Query open satellite data -description: Explore available Tilebox open data catalogs and query Sentinel-2 metadata by time and location. +description: Query Sentinel-2 metadata by time, location, and cloud cover. icon: satellite --- -Use this guide when you want to find satellite products in Tilebox open data catalogs before downloading any files. You will first inspect the available open data datasets, then query Sentinel-2 metadata by time and location. - -Tilebox Datasets stores searchable metadata for open Earth observation catalogs. Metadata queries are the fastest way to narrow a large catalog to the scenes that match your workflow, notebook, or agent task. +Tilebox indexes searchable metadata for public Earth observation catalogs. Use metadata queries to find relevant observations before reading or downloading their image assets. ## Prerequisites @@ -17,106 +15,55 @@ Tilebox Datasets stores searchable metadata for open Earth observation catalogs. uv add tilebox shapely ``` -## Explore available open data datasets +## Select the Sentinel-2 catalog -Tilebox exposes open data catalogs through the same dataset API as your private datasets. To get a list of available open data satellite datasets, run the following snippet. +Open the Sentinel-2 dataset and its Level-2A collection: ```python Python -from tilebox.datasets import Client +from tilebox.datasets import Client, field client = Client() -datasets = client.datasets() -print(datasets.open_data) +sentinel2 = client.dataset("open_data.aws_earth.sentinel2") +collection = sentinel2.collection("L2A") ``` -The output groups datasets by provider. Open data datasets include Copernicus Sentinel missions, USGS Landsat products, ASF SAR products, and other public catalogs that Tilebox has indexed. - -```plaintext Output -asf: - ers_sar: European Remote Sensing Satellite (ERS) Synthetic Aperture Radar ... -copernicus: - sentinel1_sar: The Sentinel-1 mission is the European Radar Observatory ... - sentinel2_msi: Sentinel-2 is equipped with an optical instrument payload ... - sentinel3_olci: OLCI (Ocean and Land Colour Instrument) is an optical ... - ... -usgs: - ... - landsat8_oli_tirs: Landsat-8 Operational Land Imager and Thermal Infrared ... - landsat9_oli_tirs: Landsat-9 Operational Land Imager and Thermal Infrared ... -``` +You can browse other open datasets and inspect their schemas in the [Tilebox Console](https://console.tilebox.com/datasets/open-data). -You can also browse open data datasets in the [Tilebox Console](https://console.tilebox.com/datasets/open-data) when you want descriptions, provider details, the dataset schema, and available collections before writing code. - -## Select the Sentinel-2 catalog +## Query observation metadata -Access the Sentinel-2 MSI dataset by its slug. The dataset contains collections for Sentinel-2 products such as `S2A_S2MSI2A`. +Query by time and area of interest. The result contains metadata and asset references, but does not download image bytes. ```python Python -sentinel2 = client.dataset("open_data.copernicus.sentinel2_msi") +from shapely import box -for name, collection in sentinel2.collections().items(): - print(name, collection) -``` - -## Define the search area - -Create a polygon for the area you want to inspect. This example uses a bounding box around Colorado. - -```python Python -from shapely import Polygon - -area = Polygon( - [ - # lon, lat - (-109.05, 37.0), - (-102.05, 37.0), - (-102.05, 41.0), - (-109.05, 41.0), - # close the square (repeat the first element) - (-109.05, 37.0), - ] -) -``` - -## Query Sentinel-2 metadata - -Query the Sentinel-2 Level-2A collection by time and location. This returns metadata for matching scenes; it does not download image products. - -```python Python -collection = sentinel2.collection("S2A_S2MSI2A") +area = box(-109.05, 37.0, -102.05, 41.0) scenes = collection.query( temporal_extent=("2025-10-01", "2025-11-01"), spatial_extent=area, + filter=field("cloud_cover") < 10, show_progress=True, ) -print(scenes[["granule_name", "processing_level", "product_type"]]) +print(scenes[["stac_id", "cloud_cover", "platform"]]) ``` -The result is an `xarray.Dataset` containing scene metadata. Use it to inspect candidate scenes, filter by metadata fields, or pass selected datapoints to a workflow task. - -## Filter the metadata result - -Metadata results behave like regular `xarray.Dataset` objects. You can filter, sort, or select scenes before deciding what to process next. +The result is an `xarray.Dataset`. Use regular xarray operations to sort or select observations: ```python Python -low_cloud = scenes.where(scenes.cloud_cover < 10, drop=True) -latest = low_cloud.sortby("time").isel(time=-1) +latest = scenes.sortby("time").isel(time=-1) -print(latest.granule_name.item()) +print(latest.stac_id.item()) print(latest.cloud_cover.item()) ``` -Metadata queries do not download product files. Use a [storage client](/datasets/storage/clients) when you want to read or download the files referenced by a datapoint. - ## Next steps - - Download Copernicus product files with the storage client. + + Read a COG window or download an image from a selected datapoint. - - Configure provider-specific clients for product access. + + Learn more dataset query patterns. diff --git a/index.mdx b/index.mdx index 58fa7a3..e9d3dec 100644 --- a/index.mdx +++ b/index.mdx @@ -309,7 +309,7 @@ func (t *ComputeVisibleChange) Execute(ctx context.Context) error {
- + diff --git a/sdks/python/async.mdx b/sdks/python/async.mdx index 3e8bcc8..1fbf6c2 100644 --- a/sdks/python/async.mdx +++ b/sdks/python/async.mdx @@ -78,6 +78,25 @@ datapoint = await collection.find(datapoint_uuid) `await some_async_call()` as the output of a code cell. +## Accessing assets asynchronously + +The storage client is asynchronous. Resolve the assets from one queried datapoint, then await the storage operation: + +```python Python +from tilebox.datasets.assets import AssetCollection +from tilebox.storage.aio import Client as StorageClient + +datasets = await client.datasets() +collections = await datasets.open_data.aws_earth.sentinel2.collections() +data = await collections["L2A"].query(temporal_extent=("2025-01-01", "2025-01-02")) + +assets = AssetCollection.from_datapoint(data.isel(time=0)) +storage = StorageClient() +contents = await storage.read_bytes(assets["thumbnail"], max_bytes=10_000_000) +``` + +See [Read and download assets](/datasets/assets-and-storage/read-and-download) for streaming, downloads, and GeoTIFF window reads. + ## Fetching data concurrently The primary benefit of the async client is that it allows concurrent requests, enhancing performance. diff --git a/sdks/python/install.mdx b/sdks/python/install.mdx index 311cefe..363661f 100644 --- a/sdks/python/install.mdx +++ b/sdks/python/install.mdx @@ -15,6 +15,9 @@ Tilebox offers a Python SDK for accessing Tilebox services. The SDK includes sep Workflow client and runner for Tilebox + + Read and download assets from object storage + ## Installation @@ -68,8 +71,8 @@ from tilebox.datasets import Client client = Client() datasets = client.datasets() -collection = datasets.open_data.copernicus.landsat8_oli_tirs.collection("L1T") -data = collection.query(temporal_extent=("2015-01-01", "2020-01-01"), show_progress=True) +collection = datasets.open_data.aws_earth.sentinel2.collection("L2A") +data = collection.query(temporal_extent=("2025-01-01", "2025-01-02"), show_progress=True) data ```