feat: first running workflow (retrieve, clean, build) + interactive map output

.gitignore

@@ -10,6 +10,7 @@
 \#*\#
 **/.vscode
 .$*
+.claude/
 
 ### Byte-compiled files
 __pycache__
@@ -37,6 +38,7 @@ gurobi.log
 
 # earth-osm cache
 data/earth-osm/
+tests/data/earth-osm/
 
 # test_snakemake_integration_testing's working directory; normally cleaned
 # up automatically, this is only a backstop if that's ever interrupted

INTERFACE.yaml

@@ -7,10 +7,12 @@ pathvars:
     resources:
       default: "<resources>"
       description: >-
-        Raw OSM files are written to osm/retrieve/{country}_{feature}.json.
-        Clean features are written below osm/clean; generic network outputs
-        below osm/build (buses/lines/transformers as CSV under osm/build/csv
-        and GeoJSON, including substation polygons, under osm/build/geojson).
+        Raw OSM files are written to retrieve/{country}_{feature}.json.
+        Clean features are written below clean; generic network outputs
+        below build (buses/lines/transformers as CSV under build/csv
+        and GeoJSON, including substation polygons, under build/geojson).
+        An interactive PyDeck map of the network, and the default target
+        of this workflow, is written to map.html.
     results:
       default: "<results>"
       description: location of module result files.

README.md

@@ -21,6 +21,7 @@ Currently implemented:
 1. Retrieve OSM substations, lines, cables, and (optionally) circuit relations by country, either from a cached local Geofabrik PBF extract or the live Overpass API.
 2. Clean the raw retrieval output, filtering voltage, frequency, construction status, and future assets, and grouping relation member ways into one line per real-world circuit.
 3. Merge nearby stations and line endpoints into generic buses, AC lines, and transformers.
+4. Build a self-contained interactive map of the resulting network (`map.html`), with layer toggles, voltage/text filtering, and click-through OSM links — this is the workflow's default target.
 
 ## Configuration
 
@@ -30,15 +31,17 @@ Configuration lives in [`config/config.yaml`](./config/config.yaml), validated a
 
 Please consult the [interface file](./INTERFACE.yaml) for more information.
 
-Raw retrieval outputs use `<resources>/osm/retrieve/{country}_{feature}.json`, one
+Raw retrieval outputs use `<resources>/retrieve/{country}_{feature}.json`, one
 file per country and feature (`lines_way`, `cables_way`, `substations_way`,
 `substations_node`, `substations_relation`, `routes_relation`). Both retrieval
 backends write the same raw-Overpass-JSON shape, so downstream cleaning doesn't
-need to know which one ran. Clean features use `<resources>/osm/clean/*.geojson`;
-generic network components use `<resources>/osm/build/csv/{buses,lines,transformers}.csv`
-and matching GeoJSON files under `<resources>/osm/build/geojson/`, which also
+need to know which one ran. Clean features use `<resources>/clean/*.geojson`;
+generic network components use `<resources>/build/csv/{buses,lines,transformers}.csv`
+and matching GeoJSON files under `<resources>/build/geojson/`, which also
 includes `stations_polygon.geojson` (clustered station shapes) and
 `buses_polygon.geojson` (substation polygons scoped to the buses in the output).
+An interactive map of the network is written to `<resources>/map.html`; it is
+a standalone HTML file (no server required) and the workflow's default target.
 Country logs use `<logs>/retrieve_osm_pbf/{country}.log` or
 `<logs>/retrieve_osm_overpass/{country}.log`, depending on `retrieve.source`. The
 integration example sets these roots to `resources/grid-builder` and

config/README.md

@@ -6,17 +6,23 @@ mapping in the calling configuration overrides values from those regional files.
 `retrieve.source` picks the retrieval backend: `geofabrik` reads a cached local
 PBF extract (`retrieve_osm_pbf.py`), `overpass` queries the live Overpass API
 (`retrieve_osm_overpass.py`). Both produce the same output shape, so
-`clean_osm_data` doesn't need to know which one ran. `retrieve.include_relations`
-additionally retrieves `route=power`/`power=circuit` relations, so member ways
-are grouped into one line per real-world circuit. `network` controls the
-minimum retained AC voltage, station merge buffer radius, construction filtering, and
-planned-asset cutoff date.
+`clean` doesn't need to know which one ran. `network.include_relations`
+decides whether the network should consider `route=power`/`power=circuit`
+relations, grouping their member ways into one line per real-world circuit;
+retrieval respects this too, so relations aren't fetched at all when it's off.
+`network` also controls the minimum retained AC voltage, station merge buffer
+radius, construction filtering, and planned-asset cutoff date.
 
 The [BE+NL example](./examples/config.BE-NL.yaml) is a small European development
 scope. Country files under `config/regions` are intentionally small defaults for
 now; community-maintained local corrections belong there rather than in workflow
 code.
 
+`interactive_map` controls the size of `map.html`: `coordinate_decimals` rounds
+embedded coordinates, and `simplify_geometries` sets per-geometry-type
+Douglas-Peucker tolerances (in metres) for station polygons, bus polygons, and
+lines, or disables simplification entirely via `simplify_geometries.enable`.
+
 ### Personal settings and Overpass fair use
 
 Keep `config/config.yaml` as pure defaults — a test enforces that it matches the

config/config.schema.json

@@ -31,11 +31,6 @@
           "description": "Force refresh of cached data",
           "type": "boolean"
         },
-        "include_relations": {
-          "default": true,
-          "description": "Additionally retrieve OSM route=power/power=circuit relations, so clean_osm_data can group their member ways into a single line matching the relation's real-world circuit",
-          "type": "boolean"
-        },
         "target_date": {
           "anyOf": [
             {
@@ -99,6 +94,11 @@
       "additionalProperties": false,
       "description": "Global assumptions for cleaning and connecting OSM grid features.",
       "properties": {
+        "include_relations": {
+          "default": true,
+          "description": "Whether the network should consider OSM route=power/power=circuit relations, grouping their member ways into a single line matching the relation's real-world circuit; retrieval respects this too, so relations aren't fetched at all when it's off",
+          "type": "boolean"
+        },
         "minimum_voltage_kv": {
           "default": 220.0,
           "description": "Minimum nominal AC voltage retained from OSM, in kV",
@@ -209,9 +209,50 @@
       "description": "Country-specific network overrides loaded from config/regions",
       "type": "object"
     },
+    "interactive_map": {
+      "additionalProperties": false,
+      "description": "Geometry simplification and coordinate rounding for build_interactive_map.py.",
+      "properties": {
+        "coordinate_decimals": {
+          "default": 5,
+          "description": "Decimal places kept for coordinates embedded in the interactive map; 5 is about 1.1m of precision at the equator, comfortably below every simplification tolerance below",
+          "minimum": 0,
+          "type": "integer"
+        },
+        "simplify_geometries": {
+          "additionalProperties": false,
+          "description": "Douglas-Peucker simplification tolerances for the interactive map.",
+          "properties": {
+            "enable": {
+              "default": true,
+              "description": "Whether to simplify station/bus-polygon/line geometries before embedding them in the interactive map",
+              "type": "boolean"
+            },
+            "stations_m": {
+              "default": 100.0,
+              "description": "Simplification tolerance for station polygon outlines, in metres",
+              "minimum": 0,
+              "type": "number"
+            },
+            "buses_polygon_m": {
+              "default": 5.0,
+              "description": "Simplification tolerance for individual bus/substation footprint polygons, in metres",
+              "minimum": 0,
+              "type": "number"
+            },
+            "lines_m": {
+              "default": 30.0,
+              "description": "Simplification tolerance for line geometries, in metres",
+              "minimum": 0,
+              "type": "number"
+            }
+          }
+        }
+      }
+    },
     "crs": {
       "additionalProperties": false,
-      "description": "Coordinate reference systems used throughout clean_osm_data/build_osm_network.",
+      "description": "Coordinate reference systems used throughout clean/build_network.",
       "properties": {
         "geo": {
           "default": "EPSG:4326",

config/config.yaml

@@ -7,7 +7,6 @@ countries:
 retrieve:
   source: geofabrik
   force_redownload: false
-  include_relations: true
   target_date:
   overpass_api:
     url: "https://overpass-api.de/api/interpreter"
@@ -19,6 +18,7 @@ retrieve:
       website: "https://github.com/pypsa/grid-builder"
 
 network:
+  include_relations: true
   minimum_voltage_kv: 220.0
   frequency_hz:
     AC: 50.0
@@ -29,6 +29,14 @@ network:
 
 regions: {}
 
+interactive_map:
+  coordinate_decimals: 5
+  simplify_geometries:
+    enable: true
+    stations_m: 100.0
+    buses_polygon_m: 5.0
+    lines_m: 30.0
+
 crs:
   geo: "EPSG:4326"
   distance: "EPSG:3035"

config/examples/config.BE-NL.yaml

@@ -5,9 +5,9 @@ countries:
 
 retrieve:
   source: geofabrik
-  include_relations: true
 
 network:
+  include_relations: true
   minimum_voltage_kv: 220.0
   station_merge_radius_m: 500.0
   remove_after: 2026-12-31

config/examples/config.BR.yaml

@@ -0,0 +1,12 @@
+# Brazil scope for the OSM cleaning and network-building pipeline.
+countries:
+  - BR
+
+retrieve:
+  source: overpass
+
+network:
+  include_relations: true
+  minimum_voltage_kv: 60.0
+  station_merge_radius_m: 250.0
+  remove_after: 2026-12-31

config/examples/config.MX.yaml

@@ -0,0 +1,12 @@
+# Mexico scope for the OSM cleaning and network-building pipeline.
+countries:
+  - MX
+
+retrieve:
+  source: overpass
+
+network:
+  include_relations: true
+  minimum_voltage_kv: 100.0
+  station_merge_radius_m: 150.0
+  remove_after: 2026-12-31

config/examples/config.PH.yaml

@@ -0,0 +1,12 @@
+# Philippines scope for the OSM cleaning and network-building pipeline.
+countries:
+  - PH
+
+retrieve:
+  source: overpass
+
+network:
+  include_relations: true
+  minimum_voltage_kv: 120.0
+  station_merge_radius_m: 500.0
+  remove_after: 2026-12-31

config/examples/config.US.yaml

@@ -0,0 +1,12 @@
+# United States scope for the OSM cleaning and network-building pipeline.
+countries:
+  - US
+
+retrieve:
+  source: overpass
+
+network:
+  include_relations: true
+  minimum_voltage_kv: 100.0
+  station_merge_radius_m: 200.0
+  remove_after: 2026-12-31

config/regions/config.BR.yaml

@@ -0,0 +1,6 @@
+# Brazil: 60 Hz grid; lower minimum voltage than the European default to
+# retain its sub-transmission network.
+network:
+  minimum_voltage_kv: 60
+  frequency_hz:
+    AC: 60.0

config/regions/config.MX.yaml

@@ -0,0 +1,6 @@
+# Mexico: 60 Hz grid; lower minimum voltage than the European default to
+# retain its sub-transmission network.
+network:
+  minimum_voltage_kv: 100
+  frequency_hz:
+    AC: 60.0

config/regions/config.PH.yaml

@@ -0,0 +1,6 @@
+# Philippines: 60 Hz grid; lower minimum voltage than the European default
+# to retain its sub-transmission network.
+network:
+  minimum_voltage_kv: 120
+  frequency_hz:
+    AC: 60.0

config/regions/config.US.yaml

@@ -0,0 +1,6 @@
+# United States: 60 Hz grid; lower minimum voltage than the European
+# default to retain its sub-transmission network.
+network:
+  minimum_voltage_kv: 100
+  frequency_hz:
+    AC: 60.0