diff --git a/pages/memgraph-zero/memgql/changelog.mdx b/pages/memgraph-zero/memgql/changelog.mdx index b2a01d723..bc635cf94 100644 --- a/pages/memgraph-zero/memgql/changelog.mdx +++ b/pages/memgraph-zero/memgql/changelog.mdx @@ -5,6 +5,180 @@ description: MemGQL release notes # MemGQL Changelog +## MemGQL v0.14.0 - unreleased + +### ๐Ÿƒ New features & Improvements + +- **Amazon Redshift connector.** `ADD CONNECTOR โ€ฆ TYPE redshift` maps + Redshift tables, on provisioned clusters and Serverless workgroups, as a + graph, with reads and writes pushed down as Redshift SQL. Authenticate with a + database password, or with IAM temporary credentials (`redshift+iam://โ€ฆ`) + fetched for your AWS identity on every connection. The endpoint from the AWS + console, or a `jdbc:redshift://` URL, works as the connector URI. See + [Amazon Redshift](/memgraph-zero/memgql/connect/redshift). +- **Schema discovery for native connectors.** A graph that lists a Memgraph or + Neo4j connector under `"discover": ["mg"]` reads that connector's schema at + boot and on `REFRESH SCHEMA` instead of repeating it: every node type becomes + a vertex (its label set spelled `CAT:Product`, properties as attributes), + every relationship type one edge per endpoint pair. Anything the graph + declares itself wins, so declaring an element is how you restrict what is + exposed. `EXPORT SCHEMA` writes `discover` back, not the discovered elements. + See [Discovered native schema](/memgraph-zero/memgql/schema-file#discovered-native-schema). +- **Multi-label native vertices and shared relationship types.** A native + vertex spelled `"label": "CAT:Product"` routes under each of its labels, so + `MATCH (p:CAT)`, `` MATCH (p:CAT&`Product`) `` and `CYPHER MATCH (p:CAT:Product)` + all reach it; a backticked `` :`CAT:Product` `` in a `CYPHER` passthrough is + sent as the label conjunction (it used to match nothing). A + `mappedGraphSource` edge label may be declared once per `from`/`to` pair, + and the declared pairs constrain a pattern end left unlabelled. `SHOW SCHEMA` + lists the declared or discovered attributes of native labels instead of + "(not introspected)". See [Vertices](/memgraph-zero/memgql/schema-file#vertices) + and [Edges](/memgraph-zero/memgql/schema-file#edges). +- **Query parameters are bound.** `MATCH (s:Store) WHERE s.region = $r โ€ฆ` with + `{r: 'North'}` sent by the driver now binds on every backend (as a prepared + placeholder on SQL, natively on Memgraph and Neo4j, including a `CYPHER` + passthrough). A parameter the client did not send is reported by name. +- **More of the expression language in GQL mode.** String and numeric + functions (`lower(s.name)`, `char_length`, `size`, `abs`, `floor`, โ€ฆ), + `CAST(x AS STRING | INT | FLOAT | BOOLEAN)`, concatenation with `||` or + `+`, and `EXISTS { โ€ฆ }` are translated per backend instead of being copied + into the backend query as text, so they work on renamed attributes and on + every SQL dialect (`CAST(x AS TEXT)`, `CONCAT`, `||`). On MongoDB the + functions with an aggregation operator (`upper`, `lower`, `trim`, `abs`, + `char_length`, `CAST`, โ€ฆ) now run as well, where every scalar function used + to be refused. `EXISTS { โ€ฆ }` runs natively on Memgraph and Neo4j and is + refused with a rewrite hint on SQL backends. See + [Functions and reserved words](/memgraph-zero/memgql/reference#functions-and-reserved-words-in-gql-mode). +- **Richer cross-backend queries.** A query joining `USE` blocks on different + backends can return whole nodes and relationships (`RETURN p, co`), and can + aggregate over the joined rows (`count`, `sum`, `min`, `max`, `avg`, + `count(DISTINCT โ€ฆ)`); the aggregate is partly pushed into each backend and + finished after the join. A `WITH` may follow the first `USE` block to + rename, filter or aggregate its rows before the next block runs. Joins on + several keys and chains of three or more blocks are now + [piped](/memgraph-zero/memgql/multiple-graphs#piping-fetching-only-what-joins) + too, each join deciding on its own. See + [Focused Multi-Graph Queries](/memgraph-zero/memgql/multiple-graphs#focused-multi-graph-queries). +- **Session defaults.** `SESSION SET GRAPH ` gives the session a soft + default graph: it breaks ties when a label exists in several graphs and runs + queries with no label to route by, while an explicit `USE` still wins. + `SESSION SET CONNECTION ` names the fallback connection when no + default graph is set. Both last until `SESSION RESET โ€ฆ` or the end of the + session. See + [Graph Management Query Syntax](/memgraph-zero/memgql/reference#graph-management-query-syntax). +- **`max_in_list_size` per connector.** `ADD CONNECTOR โ€ฆ OPTIONS + '{"max_in_list_size": N}'` (schema file key `maxInListSize`) caps how many + join-key values one piped cross-backend join sends to that backend; above it + the join runs unpiped. +- **Re-running `ADD CONNECTOR` is safe.** An identical declaration replies + `already exists (unchanged)`; a changed one reconnects and replies + `replaced (changed: uri, password)`, naming the changed options but never + their values. +- **`INTERSECT` / `EXCEPT` on more backends.** PostgreSQL, MySQL, DuckDB, + ClickHouse and MongoDB now run them; on MongoDB MemGQL runs each arm and + combines the results itself. A `UNION`, + `INTERSECT` or `EXCEPT` over a cache-enabled graph checks each arm against + the [cache](/memgraph-zero/memgql/multiple-graphs#caching-a-graph-in-memgraph) + on its own. +- **`CALL` on PostgreSQL and MySQL.** `CALL fn(โ€ฆ) YIELD โ€ฆ` reads a PostgreSQL + set-returning function, and `CALL proc(โ€ฆ) YIELD โ€ฆ` the first result set of a + MySQL stored procedure. See [PostgreSQL](/memgraph-zero/memgql/connect/postgres) + and [MySQL](/memgraph-zero/memgql/connect/mysql). +- **Renaming native labels.** `mappedGraphSource.label` exposes a Memgraph or + Neo4j label or relationship type under another name: queries use the + exposed name, and returned nodes and relationships carry it. See + [Vertices](/memgraph-zero/memgql/schema-file#vertices). +- **More list and map results.** `collect()` runs on ClickHouse + (`groupArray`) and Oracle (`JSON_ARRAYAGG`, without `DISTINCT`), and map + projections on Oracle. On SQL Server `INSERT โ€ฆ RETURN` returns the inserted + values instead of a row count. +- **Large results are paged.** MemGQL answers the driver's fetch size and + signals `has_more`, so a result of any size reaches the client page by page. +- **Concurrent queries.** Queries to the same backend no longer wait for each + other; each connection keeps its own pool of handlers. + +### โš ๏ธ Behavior changes + +- **A routed query no longer changes the session's connection.** A query that + routed to a graph by its labels used to become the session's default for the + next query with nothing to route by. Use `SESSION SET GRAPH` or + `SESSION SET CONNECTION` to set a default on purpose. +- **Unsupported writes and projections fail instead of doing something + else.** On SQL backends `DETACH DELETE`, `REMOVE` and `DELETE r` on a + relationship return an actionable error; `DETACH DELETE` used to run as a + plain `DELETE` and `DELETE r` deleted every row of the relationship table. + Map projections (`RETURN n {.a, .b}`) on SQL Server, SAP HANA and Redshift + return an actionable error instead of `null`. +- **A label-less `MATCH (n)` on a SQL backend** returns an error listing the + labels the mapping defines, instead of a raw backend error. +- **`USE CONNECTION` / `USE CONNECTOR`** return an error pointing to + `USE `. + +### ๐Ÿž Bug fixes + +- **`collect()` / `collect_list()` without a grouping key on SQL backends.** + `MATCH (p:Person) RETURN collect(p.name) AS names` sent an invalid + `COLLECT(...)` call to PostgreSQL, MySQL, and DuckDB instead of their list + aggregate, so the query failed. It now returns a list, the same as when the + `RETURN` also has a grouping key. On ClickHouse, SQL Server, and SAP HANA, + every `collect()` form, including `collect(DISTINCT โ€ฆ)`, now fails with an + actionable error; `collect(DISTINCT โ€ฆ)` used to return `null` there. +- **`SET` and `REMOVE` on a `READ ONLY` graph.** They were not routed by the + labels they touch, so the read-only check never saw a graph and the write + went to whichever backend the session had used last. They now route like + `DELETE` and are refused before any backend, the same as `INSERT`. +- **`OPTIONAL MATCH` filters.** On Memgraph a `WHERE` on the `MATCH` before an + `OPTIONAL MATCH` was attached to the optional pattern, so every row came + back; on SQL backends `count(n)` after an `OPTIONAL MATCH โ€ฆ WHERE` counted + the rows the filter had turned into `null`s (`500` instead of `1, 0, 0`). + Each filter now stays with its own clause, and `count(n)` counts matched + elements only. +- **An unlabelled end of a native edge.** `(p:Product)-[:inGroup]->(g)` could + return nodes of labels the graph never declared; the edge's declared `to` + label (or labels) now constrains `g`. +- **`CONTAINS` / `STARTS WITH` / `ENDS WITH` inside `CASE` on SQL backends** + always fell through to `ELSE`: the pattern was emitted without wildcards. +- **Functions on renamed attributes** (`lower(s.name)` where `name` maps to + `store_name`) emitted the GQL name into the SQL. `CAST(x AS STRING)` lost its + spaces, `size()` passed through unchanged, `||` and `+` on strings were not + translated. +- **`n.id` on a table whose key is not called `id`** emitted a column `id`; + it now reads the `metaFields.id` column. +- **Variable-length paths over `varchar(n)` keys** failed with a recursive + CTE type mismatch on PostgreSQL. +- **A later `MATCH` that reaches an already bound variable only at its end** + (`MATCH (i:Item) โ€ฆ MATCH (s:Store)-[:stocks]->(i)`) referenced a table + before it was in scope; it now joins back to the bound variable. +- **`count(r)` of a relationship carried through `WITH`** referenced a column + the derived table did not have. +- **Parse errors inside a backticked label** (`` (p:`CAT:Product`) ``) no + longer carry an unrelated "`Product` is a reserved word" hint. +- **Oracle** queries failed on quoted identifiers, derived-table aliases and + `FLOAT` casts; `NUMBER` values came back as text, and integer division did + not truncate. +- **ClickHouse** `UNION` failed without `DISTINCT`, unmatched `OPTIONAL MATCH` + values came back as defaults instead of `null`, and integer division + returned a float. +- **DuckDB** dropped the fraction of a float division. +- **A `UNION` without `USE` across graphs** could return no rows; each arm is + now routed and its rows collected. +- **A `NEXT` chain** dropped the `USE` of its first part. +- **Whole-node `WITH n` carry-through** on SQL backends read columns the + derived table did not have. +- **Two Memgraph or Neo4j connections with different schemas** shared one + introspected schema; each connection is now introspected at boot. +- **Single-connector Cypher mode** ran only the first arm of `INTERSECT`, + `EXCEPT` and `OTHERWISE`. + +### ๐Ÿšง Known limitations + +- Whole paths cannot be returned across backends, and `collect()` is not + supported over a cross-backend join. +- Results are paged to the client, but each connector still reads its full + result before the first page. +- On ClickHouse, integer division needs the attribute typed as an integer + (`"type": "Int"`) in the mapping; otherwise it divides as floating point. + ## MemGQL v0.13.0 - October 7th, 2026 ### ๐Ÿƒ New features & Improvements diff --git a/pages/memgraph-zero/memgql/connect.mdx b/pages/memgraph-zero/memgql/connect.mdx index e1b76f414..f90c73d90 100644 --- a/pages/memgraph-zero/memgql/connect.mdx +++ b/pages/memgraph-zero/memgql/connect.mdx @@ -7,6 +7,7 @@ description: MemGQL connector details and configuration. - [ClickHouse](/memgraph-zero/memgql/connect/clickhouse) - [DuckDB](/memgraph-zero/memgql/connect/duckdb) +- [Amazon Redshift](/memgraph-zero/memgql/connect/redshift) - [Iceberg](/memgraph-zero/memgql/connect/iceberg) - [Microsoft Fabric](/memgraph-zero/memgql/connect/fabric) - [Memgraph](/memgraph-zero/memgql/connect/memgraph) diff --git a/pages/memgraph-zero/memgql/connect/_meta.ts b/pages/memgraph-zero/memgql/connect/_meta.ts index 401a8a8b8..1478ab073 100644 --- a/pages/memgraph-zero/memgql/connect/_meta.ts +++ b/pages/memgraph-zero/memgql/connect/_meta.ts @@ -12,5 +12,6 @@ export default { "mysql": "to MySQL", "sqlserver": "to SQL Server", "pinot": "to Pinot", + "redshift": "to Amazon Redshift", "snowflake": "to Snowflake", } diff --git a/pages/memgraph-zero/memgql/connect/clickhouse.mdx b/pages/memgraph-zero/memgql/connect/clickhouse.mdx index b822aaea9..40378f403 100644 --- a/pages/memgraph-zero/memgql/connect/clickhouse.mdx +++ b/pages/memgraph-zero/memgql/connect/clickhouse.mdx @@ -104,8 +104,18 @@ For environment variables, see [Reference](/memgraph-zero/memgql/reference#click | Whole-node `RETURN n` / whole-rel `RETURN r` | โœ“ | | Pattern-level `WHERE` (`MATCH (n WHERE โ€ฆ)`) | โœ“ | | Typed edge `(a)-[r:R]->(b)` | โœ“ | +| `OPTIONAL MATCH` (unmatched values are `NULL`) | โœ“ | +| Whole-node `WITH n` carry-through | โœ“ | | `ORDER BY` / `LIMIT` | โœ“ | | `DISTINCT` | โœ“ | +| Map projections `RETURN n {.a, .b}` | โœ“ | +| `collect()` aggregate | โœ“ | +| `UNION` / `UNION DISTINCT` | โœ“ | +| `INTERSECT` / `EXCEPT` | โœ“ | +| Integer division (`5 / 2` is `2`)ยน | โœ“ | | `INSERT (a {โ€ฆ})` | โœ“ | | `DELETE` (via `ALTER TABLE โ€ฆ DELETE`) | โœ“ | | Boolean literals as 1/0 | โœ“ | + +ยน Integer division needs the property typed as an integer (`"type": "Int"`) in +the mapping; otherwise it divides as floating point. diff --git a/pages/memgraph-zero/memgql/connect/duckdb.mdx b/pages/memgraph-zero/memgql/connect/duckdb.mdx index e14426c8f..655db37fd 100644 --- a/pages/memgraph-zero/memgql/connect/duckdb.mdx +++ b/pages/memgraph-zero/memgql/connect/duckdb.mdx @@ -89,6 +89,7 @@ For environment variables, see [Reference](/memgraph-zero/memgql/reference#duckd | Typed edge `(a)-[r:R]->(b)` | โœ“ | | Untyped edge `()-[]->(b)` | โœ— | | `UNION` / `UNION ALL` / `UNION DISTINCT` | โœ“ | +| `INTERSECT` / `EXCEPT` | โœ“ | | Quantified path `(){m,n}` (bounded) | โœ“ | | Whole-node `RETURN n` / whole-rel `RETURN r` | โœ“ | | Map projections `RETURN n {.a, .b}` | โœ“ | @@ -104,7 +105,6 @@ For environment variables, see [Reference](/memgraph-zero/memgql/reference#duckd | `DELETE` | โœ“ | | `SET` / `REMOVE` (property update / delete) | โœ— | | `DETACH DELETE` | โœ— | -| `INTERSECT` / `EXCEPT` | โœ— | | Quantified path `(){m,}` (unbounded) | โœ— | | Shortest-path (`ALL SHORTEST` / `SHORTEST k`) | โœ— | diff --git a/pages/memgraph-zero/memgql/connect/fabric.mdx b/pages/memgraph-zero/memgql/connect/fabric.mdx index e4e5aaef0..54e352499 100644 --- a/pages/memgraph-zero/memgql/connect/fabric.mdx +++ b/pages/memgraph-zero/memgql/connect/fabric.mdx @@ -233,9 +233,10 @@ These are Fabric engine limits, not MemGQL gaps: Warehouse does not support recursive queries. MemGQL rejects such a query with a clear error rather than returning a partial answer โ€” use a fixed number of hops, or run it against a Cypher backend. -- **`collect()` and map projections** (`RETURN n {.a, .b}`) are not pushed down: - Fabric's `FOR JSON` must be the last operator in a statement and is invalid in - subqueries. +- **`collect()` and map projections** (`RETURN n {.a, .b}`) return an + actionable error: Fabric's `FOR JSON` must be the last operator in a statement + and is invalid in subqueries. +- **Temporal functions** (`date()`, `datetime()`, โ€ฆ) return an actionable error. - **Generated ids are not returned** on insert. Fabric has no usable `IDENTITY_INSERT`, so write a natural key and read the row back by it. - **Column types** are limited to what a Fabric table can store: `nvarchar`, diff --git a/pages/memgraph-zero/memgql/connect/hana.mdx b/pages/memgraph-zero/memgql/connect/hana.mdx index 050debb2b..7920bd988 100644 --- a/pages/memgraph-zero/memgql/connect/hana.mdx +++ b/pages/memgraph-zero/memgql/connect/hana.mdx @@ -287,8 +287,7 @@ MAPPING_FILE=/data/mapping.json \ [cache the graph in Memgraph](/memgraph-zero/memgql/multiple-graphs#caching-a-graph-in-memgraph), which answers the pattern from the cache. - **`collect()`** is not emitted yet; the query fails with an actionable error. -- **Map projections** (`RETURN n {.a, .b}`) currently return `NULL` instead of - a map (no error is raised); don't rely on them on this backend. +- **Map projections** (`RETURN n {.a, .b}`) return an actionable error. - **`INSERT โ€ฆ RETURN`** executes the insert but returns the affected-row count instead of the projected values. diff --git a/pages/memgraph-zero/memgql/connect/iceberg.mdx b/pages/memgraph-zero/memgql/connect/iceberg.mdx index cd9eef0d9..6120ab0fb 100644 --- a/pages/memgraph-zero/memgql/connect/iceberg.mdx +++ b/pages/memgraph-zero/memgql/connect/iceberg.mdx @@ -120,18 +120,18 @@ tables = { pa.table({"id": [1], "name": ["Acme Corp"]}), ), "knows": ( - pa.schema([("from_id", pa.int32()), ("to_id", pa.int32())]), - pa.table({"from_id": [1], "to_id": [2]}), + pa.schema([("id", pa.int32()), ("from_id", pa.int32()), ("to_id", pa.int32())]), + pa.table({"id": [1], "from_id": [1], "to_id": [2]}), ), "works_at": ( - pa.schema([("person_id", pa.int32()), ("company_id", pa.int32())]), - pa.table({"person_id": [1], "company_id": [1]}), + pa.schema([("id", pa.int32()), ("person_id", pa.int32()), ("company_id", pa.int32())]), + pa.table({"id": [1], "person_id": [1], "company_id": [1]}), ), } for name, (schema, data) in tables.items(): table = catalog.create_table_if_not_exists(f"default.{name}", schema=schema) - table.append(data) + table.append(data.cast(schema)) ``` ### 3. Start MemGQL @@ -191,19 +191,40 @@ REST API. Trino performs joins, pruning, aggregation, and writes server-side. ### 1. Start Trino with Iceberg +Trino reads Iceberg through a catalog. This one uses the MinIO storage and REST +catalog from step 1 of the direct connector above, so start those first. + ```bash -docker network create memgql-net +mkdir -p catalog +cat > catalog/iceberg.properties << 'EOF' +connector.name=iceberg +iceberg.catalog.type=rest +iceberg.rest-catalog.uri=http://iceberg-rest-dev:8181 +fs.native-s3.enabled=true +s3.endpoint=http://minio-dev:9000 +s3.path-style-access=true +s3.region=us-east-1 +s3.aws-access-key=admin +s3.aws-secret-key=password +EOF docker run -d --rm \ --name trino-dev \ --network memgql-net \ -p 8080:8080 \ + -v ./catalog/iceberg.properties:/etc/trino/catalog/iceberg.properties:ro \ trinodb/trino:latest ``` ### 2. Seed data +Both connectors read the same tables, so skip this step if you already seeded +them for the direct connector. + ```bash +# Wait until Trino accepts queries. +until docker exec trino-dev trino --execute 'SELECT 1' >/dev/null 2>&1; do sleep 2; done + docker exec -i trino-dev trino << 'SQL' CREATE SCHEMA IF NOT EXISTS iceberg.default; @@ -217,18 +238,20 @@ CREATE TABLE IF NOT EXISTS iceberg.default.companies ( name VARCHAR ); CREATE TABLE IF NOT EXISTS iceberg.default.knows ( + id INTEGER, from_id INTEGER, to_id INTEGER ); CREATE TABLE IF NOT EXISTS iceberg.default.works_at ( + id INTEGER, person_id INTEGER, company_id INTEGER ); INSERT INTO iceberg.default.persons VALUES (1, 'Alice', 30), (2, 'Bob', 25); INSERT INTO iceberg.default.companies VALUES (1, 'Acme Corp'); -INSERT INTO iceberg.default.knows VALUES (1, 2); -INSERT INTO iceberg.default.works_at VALUES (1, 1); +INSERT INTO iceberg.default.knows VALUES (1, 1, 2); +INSERT INTO iceberg.default.works_at VALUES (1, 1, 1); SQL ``` @@ -292,3 +315,6 @@ aggregation run **in-process** over Arrow batches. | Time-travel (snapshot) | โœ“ | โœ“ | | `INSERT (a {โ€ฆ})` | โŒ (read-only) | โœ“ | | `DELETE` (no alias; Trino requirement) | โŒ (read-only) | โœ“ | + +On the Trino connector, map projections (`RETURN n {.a, .b}`) and `collect()` +return an actionable error. diff --git a/pages/memgraph-zero/memgql/connect/memgraph.mdx b/pages/memgraph-zero/memgql/connect/memgraph.mdx index 3e7423e5b..0fcb15997 100644 --- a/pages/memgraph-zero/memgql/connect/memgraph.mdx +++ b/pages/memgraph-zero/memgql/connect/memgraph.mdx @@ -104,7 +104,7 @@ GQL-specific syntax (INSERT, LOCAL_DATETIME, etc.) only works on `memgraph-gql`; | Typed edge `(a)-[r:R]->(b)` | โœ“ | | Untyped edge `()-[]->(b)` | โœ“ | | `UNION` / `UNION ALL` / `UNION DISTINCT` | โœ“ | -| `INTERSECT` / `EXCEPT` | โœ— | +| `INTERSECT` / `EXCEPT` | โœ“ | | Quantified path `(){m,n}` โ€” bounded | โœ“ | | Quantified path `(){m,}` โ€” unbounded | โœ“ | | Shortest-path (`ALL SHORTEST` / `SHORTEST k`) | โœ“ | diff --git a/pages/memgraph-zero/memgql/connect/mongodb.mdx b/pages/memgraph-zero/memgql/connect/mongodb.mdx index 8bf5b4814..bef88f402 100644 --- a/pages/memgraph-zero/memgql/connect/mongodb.mdx +++ b/pages/memgraph-zero/memgql/connect/mongodb.mdx @@ -204,7 +204,8 @@ across databases. Spanning two databases needs two connectors. | `SET` / `REMOVE` (properties) | โœ“ | | `SET` / `REMOVE` (labels) | โœ— โ€” a label is a collection | | `UNION` / `UNION ALL` | โœ— | -| Scalar functions in `RETURN` (`upper(x)`, โ€ฆ) | โœ— โ€” see below | +| `INTERSECT` / `EXCEPT` | โœ“ | +| Scalar functions in `RETURN` (`upper(x)`, โ€ฆ) | partial โ€” see below | | Untyped edge `(a)-[]->(b)` | โœ— โ€” name the edge type | | Undirected variable-length `(-[:R]-()){m,n}` | โœ— โ€” give it a direction | | Path binding on a quantified pattern | โœ— | @@ -213,9 +214,12 @@ Traversal is reachability-based: `$graphLookup` returns the set of nodes reachable through the edge collection, so a node reachable by several routes appears once. Queries that count distinct paths need a Cypher backend. -Scalar functions inside `RETURN` (`upper(n.name)`, `abs(-7)`, `char_length(s)`) -are not available on MongoDB. The other backends receive such a call as raw -query text and evaluate it in their own dialect; an aggregation pipeline has no -expression string to run, so the function has to be applied client-side for now. -Aggregates (`count`, `sum`, `avg`, `min`, `max`, `collect`, and their `DISTINCT` -forms) are unaffected and run natively as `$group`. +Scalar functions inside `RETURN` run as aggregation operators where MongoDB has +one: `upper` / `lower` (`$toUpper` / `$toLower`), `trim`, `abs`, `ceil`, +`floor`, `sqrt`, `size`, `char_length` (`$strLenCP`), `round`, `coalesce`, +`nullif`, `CAST` to string / integer / float, and `substring`. A function +without an operator (`left`, `right`, `ltrim`, `exp`, `ln`, `power`, โ€ฆ) returns +an actionable error naming the supported set; apply it client-side, or front +the graph with a Memgraph cache, which evaluates it. Aggregates (`count`, +`sum`, `avg`, `min`, `max`, `collect`, and their `DISTINCT` forms) run natively +as `$group`. diff --git a/pages/memgraph-zero/memgql/connect/mysql.mdx b/pages/memgraph-zero/memgql/connect/mysql.mdx index efad65318..e89ebef96 100644 --- a/pages/memgraph-zero/memgql/connect/mysql.mdx +++ b/pages/memgraph-zero/memgql/connect/mysql.mdx @@ -120,6 +120,7 @@ For environment variables, see [Reference](/memgraph-zero/memgql/reference#mysql | Typed edge `(a)-[r:R]->(b)` | โœ“ | | Untyped edge `()-[]->(b)` | โœ— | | `UNION` / `UNION ALL` / `UNION DISTINCT` | โœ“ | +| `INTERSECT` / `EXCEPT` | โœ“ | | Quantified path `(){m,n}` (bounded) | โœ“ | | Whole-node `RETURN n` / whole-rel `RETURN r` | โœ“ | | Map projections `RETURN n {.a, .b}` | โœ“ | @@ -133,9 +134,9 @@ For environment variables, see [Reference](/memgraph-zero/memgql/reference#mysql | `COALESCE`, `NULLIF` | โœ“ | | `INSERT (a {โ€ฆ})` (MySQL has no `RETURNING`) | โœ“ | | `DELETE` | โœ“ | +| `CALL proc(โ€ฆ) YIELD โ€ฆ` (stored procedure) | โœ“ | | `SET` / `REMOVE` (property update / delete) | โœ— | | `DETACH DELETE` | โœ— | -| `INTERSECT` / `EXCEPT` | โœ— | | Quantified path `(){m,}` (unbounded) | โœ— | | Shortest-path (`ALL SHORTEST` / `SHORTEST k`) | โœ— | @@ -143,4 +144,5 @@ For environment variables, see [Reference](/memgraph-zero/memgql/reference#mysql - **Unbounded variable-length paths** (`()-[*]->()`) are rejected. Bound the depth (`{1,5}`) or run the query against a Cypher backend. - **`FOR x IN [...]` (UNWIND-style)** is rejected with an actionable error. Cypher backends (Memgraph, Neo4j) handle this syntax natively. +- **`CALL proc(โ€ฆ) YIELD โ€ฆ`** returns the procedure's first result set. `ORDER BY` / `SKIP` / `LIMIT` after it return an actionable error; sort inside the procedure. - **Path variables on variable-length patterns**: `MATCH p = (a){1,3}(b) RETURN p` is not supported. Drop the `p =` binding (or query a Cypher backend) and `RETURN` the individual nodes / edges instead. diff --git a/pages/memgraph-zero/memgql/connect/neo4j.mdx b/pages/memgraph-zero/memgql/connect/neo4j.mdx index 449330e25..ebfceb5ed 100644 --- a/pages/memgraph-zero/memgql/connect/neo4j.mdx +++ b/pages/memgraph-zero/memgql/connect/neo4j.mdx @@ -91,7 +91,7 @@ GQL-specific syntax (INSERT, LOCAL_DATETIME, etc.) only works on `neo4j-gql`; th | Typed edge `(a)-[r:R]->(b)` | โœ“ | | Untyped edge `()-[]->(b)` | โœ“ | | `UNION` / `UNION ALL` / `UNION DISTINCT` | โœ“ | -| `INTERSECT` / `EXCEPT` | โœ— | +| `INTERSECT` / `EXCEPT` | โœ“ | | Quantified path `(){m,n}` โ€” bounded | โœ“ | | Quantified path `(){m,}` โ€” unbounded | โœ“ | | Shortest-path (`ALL SHORTEST` / `SHORTEST k`) | โœ“ | diff --git a/pages/memgraph-zero/memgql/connect/oracle.mdx b/pages/memgraph-zero/memgql/connect/oracle.mdx index 2ee215b89..7ec2e5175 100644 --- a/pages/memgraph-zero/memgql/connect/oracle.mdx +++ b/pages/memgraph-zero/memgql/connect/oracle.mdx @@ -127,6 +127,12 @@ MATCH (a:Person)-[:KNOWS]->(b:Person) RETURN a.name, b.name; - No `INSERT โ€ฆ RETURNING *`. Oracle supports `RETURNING INTO :out`, but this is bind-variable based and the planner stays bind-agnostic; the executor uses it only when the caller needs the auto-generated key. +- `collect()` renders as `JSON_ARRAYAGG` and map projections as `JSON_OBJECT`; + both reach the client as a list / map. `collect(DISTINCT โ€ฆ)` is not + supported. +- Integer division truncates (`5 / 2` is `2`), matching Cypher. `NUMBER` + columns return as integers or floats, never as text. +- Temporal functions (`date()`, `datetime()`, โ€ฆ) return an actionable error. - Identifiers from the mapping file are emitted unquoted, so Oracle's default UPPERCASE folding applies; match your mapping's `table` and column spellings to your physical schema accordingly. diff --git a/pages/memgraph-zero/memgql/connect/pinot.mdx b/pages/memgraph-zero/memgql/connect/pinot.mdx index bc57565fe..3f0dafab7 100644 --- a/pages/memgraph-zero/memgql/connect/pinot.mdx +++ b/pages/memgraph-zero/memgql/connect/pinot.mdx @@ -102,3 +102,4 @@ For environment variables, see [Reference](/memgraph-zero/memgql/reference#apach | Whole-node `RETURN n` / whole-rel `RETURN r` | โœ“ | | Pattern-level `WHERE` (`MATCH (n WHERE โ€ฆ)`) | โœ“ | | Typed edge `(a)-[r:R]->(b)` | โœ“ | +| Map projections / `collect()` | โœ— (actionable error) | diff --git a/pages/memgraph-zero/memgql/connect/postgres.mdx b/pages/memgraph-zero/memgql/connect/postgres.mdx index 776455417..b592ac100 100644 --- a/pages/memgraph-zero/memgql/connect/postgres.mdx +++ b/pages/memgraph-zero/memgql/connect/postgres.mdx @@ -171,6 +171,7 @@ the comparison numeric instead of lexicographic. See | Typed edge `(a)-[r:R]->(b)` | โœ“ | | Untyped edge `()-[]->(b)` | โœ— | | `UNION` / `UNION ALL` / `UNION DISTINCT` | โœ“ | +| `INTERSECT` / `EXCEPT` | โœ“ | | Quantified path `(){m,n}` (bounded) | โœ“ | | Whole-node `RETURN n` / whole-rel `RETURN r` | โœ“ | | Map projections `RETURN n {.a, .b}` | โœ“ | @@ -184,9 +185,9 @@ the comparison numeric instead of lexicographic. See | `COALESCE`, `NULLIF` | โœ“ | | `INSERT (a {โ€ฆ}) RETURN a.x` | โœ“ | | `DELETE` | โœ“ | +| `CALL fn(โ€ฆ) YIELD โ€ฆ` (set-returning function) | โœ“ | | `SET` / `REMOVE` (property update / delete) | โœ— | | `DETACH DELETE` | โœ— | -| `INTERSECT` / `EXCEPT` | โœ— | | Quantified path `(){m,}` (unbounded) | โœ— | | Shortest-path (`ALL SHORTEST` / `SHORTEST k`) | โœ— | diff --git a/pages/memgraph-zero/memgql/connect/redshift.mdx b/pages/memgraph-zero/memgql/connect/redshift.mdx new file mode 100644 index 000000000..f52e98ec7 --- /dev/null +++ b/pages/memgraph-zero/memgql/connect/redshift.mdx @@ -0,0 +1,224 @@ +--- +title: Amazon Redshift +description: Connect MemGQL to Amazon Redshift. +--- + +# Amazon Redshift + +The Redshift connector (type `redshift`) translates GQL queries into Redshift +SQL and runs them on a provisioned cluster or a Serverless workgroup. It +requires a [mapping file](/memgraph-zero/memgql/reference#mapping-schema) that +maps graph patterns to relational tables, in the same format as every other SQL +backend. + +Redshift speaks the PostgreSQL wire protocol, so there is no ODBC layer or +Redshift client to install. The connector supports **both reads and writes**, +and authenticates with a database password or with **IAM temporary +credentials**. It is available both in +[`multi` mode](/memgraph-zero/memgql/multiple-graphs) (`ADD CONNECTOR โ€ฆ TYPE +redshift`) and as the standalone `CONNECTOR_TYPE=redshift` mode. + +## 1. Prepare a database user + +Connect to your cluster or workgroup as an admin with any PostgreSQL client or +the Redshift query editor, and create a user for MemGQL: + +```sql +CREATE USER memgql PASSWORD ''; +GRANT USAGE ON SCHEMA public TO memgql; +GRANT SELECT ON ALL TABLES IN SCHEMA public TO memgql; +-- Only if you want writes: +GRANT INSERT, DELETE ON ALL TABLES IN SCHEMA public TO memgql; +``` + +The endpoint is shown on the cluster or workgroup page in the AWS console, for +example `analytics.abc123xyz.us-east-1.redshift.amazonaws.com:5439/dev` +(provisioned) or +`my-workgroup.123456789012.us-east-1.redshift-serverless.amazonaws.com:5439/dev` +(Serverless). + +## 2. Write the mapping + +Save as `mapping.json`, the standard +[mapping format](/memgraph-zero/memgql/reference#mapping-schema). Declare every +property you plan to query. In `multi` mode the mapping is also the routing +schema, so an undeclared property is a routing error, not a passthrough. + +```json +{ + "vertices": [ + { + "label": "Person", + "mappedTableSource": { + "connector": "rs", + "schema": "public", + "table": "persons", + "metaFields": { "id": "id" } + }, + "attributes": [ + { "name": "name" }, + { "name": "age", "type": "Int" } + ] + }, + { + "label": "Company", + "mappedTableSource": { + "connector": "rs", + "schema": "public", + "table": "companies", + "metaFields": { "id": "id" } + }, + "attributes": [ { "name": "name" } ] + } + ], + "edges": [ + { + "label": "KNOWS", + "from": "Person", + "to": "Person", + "mappedTableSource": { + "connector": "rs", + "schema": "public", + "table": "knows", + "metaFields": { "id": "id", "from": "from_id", "to": "to_id" } + } + }, + { + "label": "WORKS_AT", + "from": "Person", + "to": "Company", + "mappedTableSource": { + "connector": "rs", + "schema": "public", + "table": "works_at", + "metaFields": { "id": "id", "from": "person_id", "to": "company_id" } + } + } + ] +} +``` + +## 3. Start MemGQL in multi mode + +```bash +docker run --rm \ + --name memgql \ + --stop-timeout 2 \ + -p 7688:7688 \ + --env CONNECTOR_TYPE=multi \ + -v ./mapping.json:/data/mapping.json \ + memgraph/memgql:latest +``` + +For [IAM authentication](#authentication), also pass your AWS credentials, for +example `--env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY --env +AWS_SESSION_TOKEN`. + +## 4. Connect and register the backend + +```bash +mgconsole --port 7688 +``` + +Give the endpoint as the connector `URI`. The connector accepts the same URL +shape as the Redshift JDBC driver, so you can paste the endpoint from the +console: + +```gql +ADD CONNECTOR rs TYPE redshift + URI 'redshift://analytics.abc123xyz.us-east-1.redshift.amazonaws.com:5439/dev' + USER 'memgql' PASSWORD ''; +CREATE GRAPH social FROM FILE '/data/mapping.json'; +MATCH (n:Person) RETURN n.name LIMIT 5; +``` + +The port defaults to `5439` and the database to `dev`. TLS is required by +default. The server certificate is verified against the system roots, which +include Amazon's. + +Because the parameters live on the connector, **changing them needs no MemGQL +restart**. To rotate a password, re-register the connector: + +```gql +DROP CONNECTOR rs; +ADD CONNECTOR rs TYPE redshift + URI 'redshift://analytics.abc123xyz.us-east-1.redshift.amazonaws.com:5439/dev' + USER 'memgql' PASSWORD ''; +``` + +## 5. Query + +```gql +MATCH (p:Person) RETURN p.name, p.age; +``` + +```gql +MATCH (p:Person)-[:WORKS_AT]->(c:Company) RETURN p.name, c.name; +``` + +```gql +MATCH (a:Person)((-[:KNOWS]->()){1,3})(b:Person) RETURN DISTINCT b.name; +``` + +## Authentication + +The URI scheme selects the method: + +| Method | URI | +|---|---| +| **Database password** | `redshift://:@:5439/`, or `USER` / `PASSWORD` options | +| **IAM temporary credentials** | `redshift+iam://[@]:5439/` | + +`jdbc:redshift://โ€ฆ` and `jdbc:redshift:iam://โ€ฆ` URLs work too. + +With IAM, MemGQL exchanges your AWS identity for a short-lived database password +on every connection, so there is no password to store or rotate. The identity +comes from the standard AWS credential chain: environment variables, a profile +(`?profile=`), SSO, web identity, or an instance or task role. + +- **Provisioned clusters** call `GetClusterCredentials` when a user is given + (`redshift+iam://memgql@โ€ฆ`), with optional `?AutoCreate=true` and + `?DbGroups=group1,group2`. Without a user they call + `GetClusterCredentialsWithIAM`, which logs in as your IAM identity. +- **Serverless workgroups** call `GetCredentials`, which always logs in as your + IAM identity, so a user in the URI is rejected. + +The cluster or workgroup and the region are read from the AWS endpoint name. +Behind a custom DNS name or a VPC endpoint, add them to the URI: +`?cluster=` or `?workgroup=`, plus `?region=`. + +Every option you omit falls back to its environment variable (`REDSHIFT_URI`, +`REDSHIFT_USER`, `REDSHIFT_PASSWORD`, `REDSHIFT_DATABASE`). The standalone +`CONNECTOR_TYPE=redshift` mode takes all of them from the environment. + +## Dialect notes + +- Table references are `schema.table`. A `mappedTableSource` with a `database` + is spelled `database.schema.table`, for cross-database queries and data + sharing. +- `AVG()` casts its argument to `DOUBLE PRECISION`. Redshift's `AVG` over + integers otherwise truncates. +- Bounded variable-length patterns (`(){1,3}`) run as a recursive CTE with + trail semantics. Redshift limits recursion to 100 levels. +- JSON attributes with a `path` read `VARCHAR` JSON columns with + `JSON_EXTRACT_PATH_TEXT`. +- `DESCRIBE CONNECTOR` lists tables, columns, primary keys, and foreign keys. + Redshift records these constraints but does not enforce them. + +## Known limitations + +- **`collect()`** is not available: Redshift has no array type to return a + list in. Every `collect()` form fails with an actionable error. +- **Map projections** (`RETURN n {.a, .b}`) are not available: Redshift has no + object constructor to build the map with, so they fail with an actionable + error. +- **Inserting a node and an edge in one statement** requires the node to carry + its id (`INSERT (:Person {id: 7, name: 'Ann'})-[:KNOWS]->(โ€ฆ)`). Redshift + cannot report generated ids back. Without an explicit id, insert the node + first and connect it with `MATCH โ€ฆ INSERT`. +- **Unbounded variable-length paths**, shortest path and graph algorithms are + not pushed down. Use a bounded form or a Cypher backend. +- `SUPER` columns are not mapped yet. + +For connector configuration, see the +[mapping reference](/memgraph-zero/memgql/reference#mapping-schema). diff --git a/pages/memgraph-zero/memgql/connect/sqlserver.mdx b/pages/memgraph-zero/memgql/connect/sqlserver.mdx index 7749c0e51..86d6a4450 100644 --- a/pages/memgraph-zero/memgql/connect/sqlserver.mdx +++ b/pages/memgraph-zero/memgql/connect/sqlserver.mdx @@ -223,8 +223,8 @@ MATCH (a:Person)-[:KNOWS]->(b:Person) RETURN a.name, b.name; is synthesized because T-SQL requires one for `OFFSET`/`FETCH`. - Booleans are emitted as `BIT` (`1`/`0`). - `char_length()` / `length()` on a column translate to T-SQL `LEN()`. -- No `INSERT โ€ฆ RETURNING`; SQL Server's `OUTPUT` clause is used only when the - caller needs the auto-generated key. +- `INSERT โ€ฆ RETURN` projects the inserted row through SQL Server's + `OUTPUT INSERTED.` clause, so it returns the values, not a row count. ## Known limitations @@ -232,11 +232,8 @@ MATCH (a:Person)-[:KNOWS]->(b:Person) RETURN a.name, b.name; semantics are not available on SQL Server yet and return an actionable error. Use a bounded fixed-hop chain or a Cypher backend. - **`collect()`** is not available; there is no portable array aggregate in - the T-SQL surface MemGQL emits yet; the query fails with the backend's error. -- **Map projections** (`RETURN n {.a, .b}`) currently return `NULL` instead of - a map (no error is raised); don't rely on them on this backend. -- **`INSERT โ€ฆ RETURN`** executes the insert but returns the affected-row count - instead of the projected values. + the T-SQL surface MemGQL emits yet; the query fails with an actionable error. +- **Temporal functions** (`date()`, `datetime()`, โ€ฆ) return an actionable error. For connector configuration, see [Reference](/memgraph-zero/memgql/reference#sql-server-sqlserver). diff --git a/pages/memgraph-zero/memgql/features.mdx b/pages/memgraph-zero/memgql/features.mdx index 66774cdf5..0bcbc61ce 100644 --- a/pages/memgraph-zero/memgql/features.mdx +++ b/pages/memgraph-zero/memgql/features.mdx @@ -26,6 +26,7 @@ description: MemGQL Community and Enterprise feature comparison. | [Microsoft Fabric](/memgraph-zero/memgql/connect/fabric) | Yes | Yes | | [MongoDB](/memgraph-zero/memgql/connect/mongodb) | Yes | Yes | | [SAP HANA](/memgraph-zero/memgql/connect/hana) | Yes | Yes | +| [Amazon Redshift](/memgraph-zero/memgql/connect/redshift) | Yes | Yes | | **Multi-Connection Mode** | Yes | Yes | | Max connectors | 2 | Unlimited | | Max simultaneous connections | 2 | Unlimited | diff --git a/pages/memgraph-zero/memgql/multiple-graphs.mdx b/pages/memgraph-zero/memgql/multiple-graphs.mdx index c850c3852..f038a338a 100644 --- a/pages/memgraph-zero/memgql/multiple-graphs.mdx +++ b/pages/memgraph-zero/memgql/multiple-graphs.mdx @@ -102,11 +102,13 @@ Routing is strict and deterministic: pointing you at `SHOW SCHEMA`. - **Two or more candidates** โ†’ an *ambiguous* hard error listing the candidates. Disambiguate by referencing a property or relationship type that exists in - only one of them, or add an explicit `USE`. + only one of them, add an explicit `USE`, or set a session default with + `SESSION SET GRAPH ` (it wins when it is one of the candidates). - **Explicit `USE` always wins** and bypasses inference entirely. - **No-signal queries** (`MATCH (n) RETURN n` with no label, relationship type, or property to route by) have nothing to infer from; pin them with an - explicit `USE `. + explicit `USE `, or run them on the session default set with + `SESSION SET GRAPH `. Identifier matching is **exact**: `:person` does not match a mapping (or introspected schema) that declares `Person`. Routing, translation, and the @@ -137,15 +139,13 @@ follow-up read routes without needing a `REFRESH SCHEMA`. ### What still needs an explicit `USE` -- A graph bound to a **non-default remote database** (Memgraph multi-tenancy) is - fenced to explicit `USE`; per-tenant introspection isn't wired up yet. - A **single `MATCH` pattern that spans two backends** errors with guidance to split it into one `MATCH` clause per graph โ€” unless the link between them is declared as a [cross-connector edge](#traversing-across-backends-with-an-edge), which is split automatically. -- A label-less `MATCH (n)` against a SQL backend still surfaces a raw backend - error (there's nothing to route or translate by). +- A label-less `MATCH (n)` against a SQL backend returns an actionable error + listing the labels the mapping defines (a label is what picks the table). ## Schema discovery @@ -224,6 +224,28 @@ Execution flow: 2. The `USE warehouse` part executes on `postgres_dw`, using `c.name` values as filters 3. Results are joined locally and returned +Whole nodes and relationships can be returned from either side (`RETURN p, co`); +whole paths can't, so return their nodes and relationships instead. Node ids in +a cross-backend result are unique across backends, but differ from the id the +same node gets in a single-backend query. + +Aggregates over the joined rows work too: `count`, `sum`, `min`, `max`, `avg` +and `count(DISTINCT โ€ฆ)`. `collect()` is not supported over a cross-backend +join. + +A `WITH` may follow the first `USE` block (and chain, `WITH โ€ฆ WITH โ€ฆ`). MemGQL +evaluates it over that block's rows before the next block runs, so it can +rename, filter, aggregate, and apply `DISTINCT` / `ORDER BY` / `LIMIT`; later +blocks see only its output columns. A `WITH` after a later block returns an +actionable error. + +```gql +USE social MATCH (p:Person)-[:WORKS_AT]->(c:Company) + WITH c.name AS company, count(p) AS staff WHERE staff > 10 +USE warehouse MATCH (co:Company {name: company}) +RETURN company, staff, co.revenue; +``` + Another example correlating social connections with event logs: ```gql @@ -257,6 +279,12 @@ or without it. It engages for backends registered as catalog graphs locally. If the selective side matches nothing, the other backend is never queried. +A chain of three or more `USE` blocks is piped at each join on its own, and a +join on several keys ships the matching key tuples together. How many keys one +pipe may ship to a backend is capped by its +[`max_in_list_size`](/memgraph-zero/memgql/reference#graph-management-query-syntax) +option; above the cap that join runs unpiped. + ### Traversing across backends with an edge The joins above are written as predicates over two query parts. A graph can @@ -547,20 +575,20 @@ routed by its labels: In a cross-backend query (split by explicit `USE`, or routed by label), each cache-enabled part is served from its cache independently while the other parts -run on their own sources. `UNION` / `UNION ALL` queries are the exception: they -dispatch as a whole and never consult the cache, even when every scope they -touch is resident. +run on their own sources. The same holds for the arms of a `UNION`, +`INTERSECT` or `EXCEPT`: each arm over a cache-enabled graph is checked against +the cache on its own. Inspect what is cached with `SHOW GRAPH CACHES` (returns each cache-enabled graph, its cache connector, `TTL`, `MAX_BYTES`, and the number of cached fragments). Cache activity is logged at `DEBUG` as `[CACHE] MISS โ€ฆ teeing โ€ฆ` and `[CACHE] HIT โ€ฆ serving from cache connector โ€ฆ`. -> **Note:** The query-driven cache is exercised in CI against PostgreSQL, MySQL, -> DuckDB, SQL Server, SAP HANA, ClickHouse and the native Iceberg connector, each -> replaying the same query corpus from its cache that it runs against the -> source, and asserting identical rows. Graph-native sources (Memgraph, Neo4j) cannot be cached โ€” -> there is no relational mapping to copy from. +> **Note:** The query-driven cache is verified against PostgreSQL, MySQL, +> DuckDB, SQL Server, SAP HANA, ClickHouse and the native Iceberg connector: +> the same queries return identical rows from the cache and from the source. +> Graph-native sources (Memgraph, Neo4j) cannot be cached โ€” there is no +> relational mapping to copy from. ## Cross-Graph Query Rules @@ -572,7 +600,7 @@ and `[CACHE] HIT โ€ฆ serving from cache connector โ€ฆ`. 4. **Compatible column types**: All branches of a composite query must produce result sets with the same number of columns, in the same order, with compatible types. -5. **Routing is per part**: With [routing](#routing) each query part (and each composite branch) routes on its own schema signals. A part that matches zero or multiple sources is a hard error; add an explicit `USE` or reference a label/property unique to one source. +5. **Routing is per part**: With [routing](#routing) each query part (and each composite branch) routes on its own schema signals. A part that matches zero sources is a hard error. A part that matches several is an error too, unless one of them is the session default set with `SESSION SET GRAPH`; otherwise add an explicit `USE` or reference a label/property unique to one source. ## Complete Example diff --git a/pages/memgraph-zero/memgql/reference.mdx b/pages/memgraph-zero/memgql/reference.mdx index a508aee46..4401ab42f 100644 --- a/pages/memgraph-zero/memgql/reference.mdx +++ b/pages/memgraph-zero/memgql/reference.mdx @@ -10,7 +10,7 @@ Memgraph and Neo4j (translation is largely passthrough); "SQL backends" means PostgreSQL, MySQL and DuckDB. **MongoDB** is neither โ€” it translates to aggregation pipelines โ€” so it gets its own column. -SQL Server, SAP HANA, ClickHouse, Apache Iceberg, and Apache Pinot are also supported +SQL Server, SAP HANA, Amazon Redshift, ClickHouse, Apache Iceberg, and Apache Pinot are also supported as connectors but with a narrower verified surface. See each connector's page for the exact list of features each one supports. @@ -28,7 +28,7 @@ supports. | Multi-label pattern `(n:A&B)`; shared label over several mappings | โœ“ | โœ“ | โœ— | | Untyped edge `()-[]->(b)` (union over types) | โœ“ | โœ— | โœ— | | `UNION` / `UNION ALL` / `UNION DISTINCT` | โœ“ | โœ“ | โœ— | -| `INTERSECT` / `EXCEPT` | โœ“ | โœ“ | โœ— | +| `INTERSECT` / `EXCEPT` | โœ“ | โœ“ | โœ“ | | Quantified path `(){m,n}` (bounded) | โœ“ | โœ“ | โœ“ | | Quantified path `(){m,}` (unbounded) | โœ“ | โœ— | โœ“ | | Shortest-path (`ALL SHORTEST` / `ANY SHORTEST` / `SHORTEST k`) | โœ“ | โœ— | โœ— | @@ -37,6 +37,10 @@ supports. | Connection-less `RETURN 1` / `RETURN 1 + 2` (liveness) | โœ“ | โœ“ | โœ“ | | `IN` list membership `WHERE x IN [โ€ฆ]` | โœ“ | โœ“ | โœ“ | | `STARTS WITH` / `ENDS WITH` / `CONTAINS` | โœ“ | โœ“ | โœ“ | +| Query parameters (`WHERE n.id = $id`, sent by the driver) | โœ“ | โœ“ | โœ“ | +| String concatenation (`a \|\| b`, `'x' + a`) | โœ“ | โœ“ | โœ“ | +| `CAST(x AS STRING \| INT \| FLOAT \| BOOLEAN)` | โœ“ | โœ“ | โœ— | +| `EXISTS { (n)-[:R]->() }` / `EXISTS { MATCH โ€ฆ }` in `WHERE` | โœ“ | โœ— | โœ— | | `FOR x IN [โ€ฆ]` (UNWIND-style loop) | โœ“ | โœ— | โœ“ | | `collect()` / `collect_list()` (aggregate) | โœ“ | โœ“ | โœ“ | | `count`, `sum`, `avg`, `min`, `max` | โœ“ | โœ“ | โœ“ | @@ -44,7 +48,7 @@ supports. | Arithmetic `+ - * / %` | โœ“ | โœ“ | โœ“ | | `CASE WHEN โ€ฆ THEN โ€ฆ ELSE โ€ฆ END` | โœ“ | โœ“ | โœ“ | | `COALESCE`, `NULLIF` | โœ“ | โœ“ | โœ“ | -| Scalar functions in `RETURN` (`upper(x)`, `abs(x)`, โ€ฆ) | โœ“ | โœ“ | โœ— | +| Scalar functions in `RETURN` (`upper(x)`, `abs(x)`, โ€ฆ) | โœ“ | โœ“ | partial | | Temporals (`date`, `datetime`, `localTime`, โ€ฆ) | โœ“ | โœ— | โœ— | | `INSERT (a {โ€ฆ})` | โœ“ | โœ“ | โœ“ | | `INSERT (a {โ€ฆ}) RETURN a.x` (post-insert projection) | โœ“ | โœ“ | โœ— | @@ -64,12 +68,21 @@ supports. separate table (or collection), so an untyped hop would have to union across every registered edge mapping. The form is still accepted natively on Cypher backends. +- **`collect()` on SQL backends** runs on PostgreSQL, MySQL, DuckDB, + ClickHouse, and Oracle (Oracle without `DISTINCT`). SQL Server, SAP HANA, + Amazon Redshift, Fabric, Iceberg (Trino), and Pinot return an actionable + error; see each connector's page. - **`FOR x IN [...]` (UNWIND-style) on SQL backends** returns an actionable error pointing users at running the query on a Cypher backend. The form is still accepted natively on Cypher backends. -- **`DETACH DELETE` on SQL backends** currently executes as a plain `DELETE`; - relationship rows are not detached. It deletes a node that has no relationship - rows and otherwise surfaces the backend's constraint error; don't rely on it. +- **`DETACH DELETE` on SQL backends** returns an actionable error: delete the + relationships first, or use a Cypher backend. `REMOVE` on SQL backends + returns an actionable error as well. +- **Label-less `MATCH (n)` on SQL backends** returns an actionable error listing + the labels the mapping defines; a label is what picks the table. +- **`EXISTS { โ€ฆ }` on SQL backends** returns an actionable error. Write the + check as a traversal (`MATCH (s)-[:R]->(t) โ€ฆ RETURN DISTINCT s`) or as + `OPTIONAL MATCH โ€ฆ WHERE t IS NOT NULL`; Cypher backends run it as written. - **Path variables on variable-length patterns**: `MATCH p = (a){1,3}(b) RETURN p` is not yet supported on SQL backends or MongoDB. Drop the `p =` binding (or query a Cypher backend) and `RETURN` the @@ -84,8 +97,9 @@ supports. #### MongoDB-specific -- **`UNION` / `UNION ALL` / `INTERSECT` / `EXCEPT`** return an actionable error. - Compose the arms client-side, or run the query on another backend. +- **`UNION` / `UNION ALL`** return an actionable error. Compose the arms + client-side, or run the query on another backend. `INTERSECT` / `EXCEPT` work: + MemGQL runs each arm and combines the results itself. - **Quantified paths are reachability, not path enumeration.** MongoDB runs `(){m,n}` on `$graphLookup`, which never revisits an edge document. That matches trail semantics (no edge repeats) and terminates on cycles, but it @@ -95,12 +109,12 @@ supports. - **Undirected variable-length** (`(-[:R]-()){m,n}`) returns an actionable error: `$graphLookup` follows a single connect-from/connect-to field pair. Give the pattern a direction. -- **Scalar functions inside `RETURN`** (`upper(n.name)`, `abs(-7)`, - `char_length(s)`) return an actionable error. MemGQL passes such a call to - the other backends as raw query text, which happens to parse in their SQL or - Cypher dialect; an aggregation pipeline has no expression string to run one. - Aggregates (`count`, `sum`, `avg`, `min`, `max`, `collect`, and their - `DISTINCT` forms) are unaffected and run natively. +- **Scalar functions inside `RETURN`** run where MongoDB has an aggregation + operator (`upper`, `lower`, `trim`, `abs`, `ceil`, `floor`, `sqrt`, `size`, + `char_length`, `round`, `coalesce`, `nullif`, `CAST`, `substring`); one + without an operator (`left`, `right`, `exp`, `power`, โ€ฆ) returns an + actionable error naming the supported set. Aggregates run natively. See + [MongoDB](/memgraph-zero/memgql/connect/mongodb). - **`INSERT โ€ฆ RETURN` drops the projection** and reports the affected count instead. Re-read the inserted node with a follow-up `MATCH`. - **Every collection in one graph must live in the same database.** MongoDB's @@ -115,6 +129,38 @@ supports. variable it constrains; one spanning two variables returns an actionable error rather than silently dropping rows. +### Functions and reserved words in GQL mode + +GQL (the default language) has a fixed set of function names; Cypher's +`camelCase` spellings are not among them. Use the GQL name, or the `CYPHER` +prefix for the Cypher one. + +| Kind | GQL mode (every backend) | Cypher mode (Memgraph / Neo4j) | +|------|--------------------------|--------------------------------| +| Case and trimming | `UPPER`, `LOWER`, `TRIM`, `BTRIM`, `LTRIM`, `RTRIM`, `LEFT`, `RIGHT` | `toUpper`, `toLower`, `trim`, `lTrim`, `rTrim`, `left`, `right` | +| Length | `CHAR_LENGTH`, `SIZE`, `CARDINALITY` | `size` | +| Numeric | `ABS`, `MOD`, `FLOOR`, `CEIL`, `SQRT`, `EXP`, `LN`, `LOG10`, `LOG(base, x)`, `POWER`, `SIN`, `COS`, `TAN`, `ASIN`, `ACOS`, `ATAN` | the same, lower-case; `^` for `POWER` | +| Conversion | `CAST(x AS STRING)`, `CAST(x AS INT)`, `CAST(x AS FLOAT)`, `CAST(x AS BOOLEAN)` | `toString`, `toInteger`, `toFloat`, `toBoolean` | +| Null handling | `COALESCE`, `NULLIF`, `IS [NOT] NULL` | the same | +| Strings | `\|\|` and `+` concatenate; `STARTS WITH`, `ENDS WITH`, `CONTAINS` | `+`; `STARTS WITH`, `ENDS WITH`, `CONTAINS`, `=~` | +| Aggregates | `count`, `sum`, `avg`, `min`, `max`, `collect`, each with `DISTINCT` | the same | +| Elements | โ€” | `labels(n)`, `keys(n)`, `type(r)`, `id(n)`, `properties(n)` | + +On SQL backends `SIZE(x)` is the string length (`CHAR_LENGTH`), and a `+` +concatenates only when one side is a string literal or a string function; a +`+` between two columns stays numeric. A function GQL does not know fails to +parse (`mismatched input '('`); `toUpper(p.sku)` becomes `UPPER(p.sku)`, or +`CYPHER MATCH โ€ฆ RETURN toUpper(p.sku)`. + +Some words that are ordinary labels or properties elsewhere are **reserved in +GQL** and must be backtick-quoted when used as a label, relationship type, +property or alias: among the ones that come up in practice are `Product`, +`Project`, `path`, `reference`, `size`, `unit`, `value` and `year`. A parse +error on such a word says so and shows the quoted form +(`` MATCH (p:`Product`) ``). A quoted name may contain a colon: `` :`CAT:Product` `` +is the [multi-label native vertex](/memgraph-zero/memgql/schema-file#vertices) +spelling. + ## Query Languages MemGQL speaks **GQL** (the default) and **Cypher**. Which language a @@ -176,12 +222,17 @@ ADD CONNECTOR TYPE [WAREHOUSE ''] [ROLE ''] [PRIVATE_KEY_PATH ''] [TOKEN ''] [TENANT_ID ''] [CLIENT_ID ''] [CLIENT_SECRET ''] - [LANGUAGE '']; + [LANGUAGE ''] + [OPTIONS '{"max_in_list_size": }']; DROP CONNECTOR ; PING ; -- Session SET SESSION LANGUAGE GQL|CYPHER []; -- language for bare queries on this session +SESSION SET GRAPH ; -- soft default graph for this session +SESSION RESET GRAPH; +SESSION SET CONNECTION ; -- fallback connection for queries nothing else routes +SESSION RESET CONNECTION; -- Graphs (mappings over connectors) CREATE GRAPH FROM ''; -- inline { "vertices": โ€ฆ, "edges": โ€ฆ } body @@ -202,7 +253,7 @@ apply depends on the type: | Option | Read by | |---------------------------------------------|----------------------------------------------------------------------| | `GRAPH ''` | Memgraph, Neo4j โ€” selects the Cypher database (it is not a mapping) | -| `DATABASE ''` | PostgreSQL, MySQL, SQL Server, Oracle (service name), ClickHouse, MongoDB, Snowflake, Fabric (warehouse/lakehouse item), SAP HANA (tenant database of an MDC system) | +| `DATABASE ''` | PostgreSQL, MySQL, SQL Server, Oracle (service name), ClickHouse, MongoDB, Snowflake, Fabric (warehouse/lakehouse item), SAP HANA (tenant database of an MDC system), Amazon Redshift | | `CATALOG ''` | Iceberg (Trino catalog), Iceberg Direct (warehouse) | | `SCHEMA ''` | Iceberg, Snowflake, Fabric (default `dbo`), SAP HANA (defaults to the connection user's own schema); MongoDB accepts it as a fallback for `DATABASE` | | `WAREHOUSE` / `ROLE` | Snowflake session settings | @@ -210,10 +261,25 @@ apply depends on the type: | `TENANT_ID` / `CLIENT_ID` / `CLIENT_SECRET` | Fabric service-principal auth (Microsoft Entra ID) | | `PATH ''` | DuckDB (database file; `:memory:` by default) | | `LANGUAGE ''` | Any connector โ€” default [query language](/memgraph-zero/memgql/reference#query-languages) (`gql` / `cypher`) for `USE ` queries whose graph maps to it | +| `OPTIONS '{"max_in_list_size": }'` | Any connector โ€” most join-key values MemGQL sends it in one [piped](/memgraph-zero/memgql/multiple-graphs#piping-fetching-only-what-joins) cross-backend join (default 10 000; Oracle and Pinot 1 000, Snowflake 16 384, ClickHouse 50 000). Above it the join runs unpiped. Schema file key: `maxInListSize` | An option a connector doesn't read is ignored, and one that is omitted falls -back to that connector's environment variable. Re-adding a connector replaces -its config; `DROP CONNECTOR` is refused while a graph still references it. +back to that connector's environment variable. `DROP CONNECTOR` is refused +while a graph still references it. + +Running `ADD CONNECTOR` again for an existing name is safe: an identical +declaration replies `Connector '' already exists (unchanged)`, and a +changed one replaces the config, reconnects, and replies +`Connector '' replaced (changed: uri, password)`, naming the changed +options but never their values. A new name replies `Connector '' added`. + +`SESSION SET GRAPH ` gives the session a **soft default graph**. An +explicit `USE` always wins and a query whose labels match exactly one graph +still routes there; the default graph breaks ties when a label exists in +several graphs, and runs queries with no label to route by. +`SESSION SET CONNECTION ` names the connection such a query falls back +to when no default graph is set (`SHOW CONNECTIONS` marks it with `*`). Both last until +the matching `SESSION RESET` or the end of the Bolt session. `CREATE GRAPH โ€ฆ FROM` registers a graph from a `{ "vertices": โ€ฆ, "edges": โ€ฆ }` body and auto-connects the connectors it references. The mapping format is @@ -251,7 +317,7 @@ SHOW MAPPINGS; -- per-graph mappings SHOW SCHEMA [FOR ] [AS GRAPH]; -- unified routing index: labels, rel-types, properties; -- AS GRAPH returns it as Bolt nodes and relationships EXPORT SCHEMA [TO '']; -- merged catalog as canonical schema JSON (round-trippable) -REFRESH SCHEMA; -- re-introspect live Cypher connections (Memgraph/Neo4j) +REFRESH SCHEMA; -- re-introspect live Cypher connections (Memgraph/Neo4j) and re-run schema discovery -- Source onboarding (see below) DESCRIBE CONNECTOR ; -- the source's physical catalog: tables, columns, types, keys @@ -297,7 +363,7 @@ Writing a [mapping](/memgraph-zero/memgql/schema-file) shouldn't require out-of-band database access. `DESCRIBE CONNECTOR` reads a source's physical catalog **live through the connector** โ€” tables, columns, types, nullability, primary keys, and single-column foreign keys as `โ†’ table.column` -(PostgreSQL and SAP HANA; a graph backend answers via `SHOW SCHEMA` / +(PostgreSQL, SAP HANA and Amazon Redshift; a graph backend answers via `SHOW SCHEMA` / `REFRESH SCHEMA` instead and says so): ``` @@ -522,6 +588,7 @@ refused with a `Compute limit reached` error; the client can retry. | `pinot` | GQL -> SQL | Apache Pinot | | `mongodb` | GQL -> aggregation pipeline | MongoDB 5.0+ | | `hana` | GQL -> SQL | SAP HANA 2.0 SPS05+, HANA Cloud, HANA Express | +| `redshift` | GQL -> SQL | Amazon Redshift (provisioned and Serverless) | | `multi` | Per-connector | Multiple backends simultaneously | @@ -653,6 +720,21 @@ There is no `CATALOG` level for HANA: the tenant database is chosen at login, not spelled into a qualified table name, so a HANA table reference is at most `schema.table`. See the [SAP HANA connector page](/memgraph-zero/memgql/connect/hana). +#### Amazon Redshift (`redshift`) + +| Variable | Default | Description | +|---------------------|--------------------------|----------------------------------------------------------------| +| `REDSHIFT_URI` | _(required)_ | `redshift://:@:5439/` (password) or `redshift+iam://:5439/` (IAM temporary credentials); `jdbc:redshift://` URLs are accepted | +| `REDSHIFT_USER` | _(from the URI)_ | Database user, when not written into the URI | +| `REDSHIFT_PASSWORD` | _(from the URI)_ | Password, when not written into the URI (password auth only) | +| `REDSHIFT_DATABASE` | `dev` | Database, when not written into the URI | +| `MAPPING_FILE` | _(required)_ | Path to JSON mapping file (same format as Postgres) | + +IAM auth uses the standard AWS credential chain, so the usual `AWS_*` +variables (or an instance or task role) apply. TLS is required unless the URI +says `?sslmode=disable`. See the +[Amazon Redshift connector page](/memgraph-zero/memgql/connect/redshift). + #### Iceberg (`iceberg`) | Variable | Default | Description | diff --git a/pages/memgraph-zero/memgql/schema-file.mdx b/pages/memgraph-zero/memgql/schema-file.mdx index f46495e8e..507b60f27 100644 --- a/pages/memgraph-zero/memgql/schema-file.mdx +++ b/pages/memgraph-zero/memgql/schema-file.mdx @@ -53,6 +53,9 @@ The file has two top-level sections: } ``` +An optional `_comment` string is allowed at the top level and on each graph for +free-text notes; it is ignored at load. + ## A minimal schema The smallest useful file is one connector and one vertex: a label mapped to a @@ -143,11 +146,15 @@ Every `connection` field is optional; supply what a given backend needs: | `path` | File-backed backends (DuckDB). | | `sslRootCert` | PEM bundle of CA certificates to trust instead of the system roots (PostgreSQL). | | `trustServerCertificate` | Encrypt without verifying the server's certificate (PostgreSQL). Mutually exclusive with `sslRootCert`. | +| `warehouse` / `role` | Snowflake session context. Each falls back to its `SNOWFLAKE_*` environment variable when omitted. | +| `private_key_path` / `token` | Snowflake key-pair or token authentication instead of a password. Each falls back to its `SNOWFLAKE_*` environment variable when omitted. | +| `tenant_id` / `client_id` / `client_secret` | Microsoft Fabric service-principal authentication. Each falls back to its `FABRIC_*` environment variable when omitted. | | `language` | Default [query language](/memgraph-zero/memgql/reference#query-languages) (`gql` / `cypher`) for `USE ` queries whose graph maps to this connector. | +| `maxInListSize` | Most join-key values sent to this backend in one [piped](/memgraph-zero/memgql/multiple-graphs#piping-fetching-only-what-joins) cross-backend join; the same as `ADD CONNECTOR โ€ฆ OPTIONS '{"max_in_list_size": N}'`. | **Supported `type` values:** `memgraph`, `neo4j`, `postgres` (`postgresql`), `mysql`, `sqlserver`, `oracle`, `duckdb`, `iceberg`, `iceberg-direct`, -`clickhouse`, `pinot`, `snowflake`, `mongodb`, `fabric`, `hana`. +`clickhouse`, `pinot`, `snowflake`, `mongodb`, `fabric`, `hana`, `redshift`. For PostgreSQL, the TLS **mode** stays in the URI (`sslmode=`), where libpq users expect it; `sslRootCert` / `trustServerCertificate` declare what a URI @@ -192,6 +199,48 @@ runtime and persists it back to this file; `EXPORT SCHEMA` round-trips it. } ``` +### Discovered native schema + +A Memgraph or Neo4j connector already knows its own labels, relationship types +and properties, so a graph does not have to repeat them. List the connector +under `discover` and its schema is read at boot and on `REFRESH SCHEMA`: +every node type becomes a vertex, with the label set spelled `A:B` and the +introspected properties as attributes, and every relationship type becomes one +edge per `(type, start labels, end labels)` pair. + +```json +{ + "name": "shop", + "accessMode": "readOnly", + "discover": ["mg"], + "vertices": [ + { + "label": "Item", + "mappedTableSource": { "connector": "pg", "table": "item", "metaFields": { "id": "item_key" } }, + "attributes": [{ "name": "sku" }] + } + ], + "edges": [ + { + "label": "catalogEntry", + "from": "Item", + "to": "CAT:Product", + "mappedJoinSource": { "fromKey": "sku", "toKey": "sku" } + } + ] +} +``` + +What the graph declares itself wins: a declared vertex (matched by its label +set) keeps its attributes and renames, and a declared relationship type keeps +exactly its declared pairs, so declaring an element is how you restrict or +refine what discovery exposes. An edge may name a vertex that is discovered +later, as above, as long as exactly one connector is discovered from. `EXPORT +SCHEMA` and the persisted file write the `discover` entry back, not the +discovered elements. Memgraph must run with `--schema-info-enabled`; +`discover` is refused on a relational connector (declare the tables, or draft +them with `GENERATE MAPPING FOR CONNECTOR`). + ### Vertices A vertex maps a **label** to a backend source. Exactly one source kind is set: @@ -199,7 +248,12 @@ A vertex maps a **label** to a backend source. Exactly one source kind is set: - **`mappedTableSource`**: a relational table (PostgreSQL, MySQL, Oracle, SQL Server, DuckDB, ClickHouse, Iceberg). Rows become nodes. - **`mappedGraphSource`**: a native graph backend (Memgraph / Neo4j). Queries - pass through as Cypher; only the connector is declared. + pass through as Cypher; only the connector is declared, plus `label` when + the backend stores the element under another name. + +A vertex may also add a **`mappedContentSource`** on top of either one, to read +some of its properties from a table in another connector (see +[Content in another connector](#content-in-another-connector)). ```json { @@ -278,6 +332,15 @@ start from one table, so on a shared label both are refused with a pointer to the identifying label. A label no mapping carries together with the others matches nothing. +A **native** node type that carries several labels is one vertex spelled with +a colon, `"label": "CAT:Product"`, and written backticked in GQL +(`` MATCH (p:`CAT:Product`) ``). Each member label routes as well: +`MATCH (p:CAT)` reads every `CAT` node whatever its other labels, +`` MATCH (p:CAT&`Product`) `` and Cypher's `MATCH (p:CAT:Product)` reach the +same vertex, and a backticked `` :`CAT:Product` `` in a `CYPHER` passthrough is +sent to the backend as the label conjunction `:CAT:Product`. This is also the +spelling [schema discovery](#discovered-native-schema) produces. + | Field | Source kind | Description | |-------|-------------|-------------| | `label` / `labels` | both | The node label, or a list of labels whose first entry identifies the mapping (unique per graph) and whose others may be shared with other mappings. | @@ -287,6 +350,71 @@ matches nothing. | `mappedTableSource.table` | relational | The backing table or view. | | `mappedTableSource.metaFields.id` | relational | Primary-key column, or a list of columns for a composite key (`["region", "order_no"]`); the node's identity and the join key for edges. Each column is also exposed as a property, so `RETURN n` includes it and `n.` reads it by name. | | `mappedGraphSource.connector` | native | Connector name (Memgraph / Neo4j); the query passes through as Cypher. | +| `mappedGraphSource.label` | native | Optional backend label (on an edge, backend relationship type) exposed under this element's `label`. Queries use the exposed name, MemGQL rewrites it for the backend, and returned nodes and relationships carry the exposed name. Two elements cannot share a backend name. Not renamed: strings a query computes from names (`labels(n)`, `type(r)`), the extra labels of a multi-label vertex, and property keys. | +| `mappedContentSource` | either | Optional second source for some of the vertex's properties; see [Content in another connector](#content-in-another-connector). | +| `attributes` | either | The properties the vertex exposes; see [Attributes](#attributes). | + +#### Content in another connector + +A vertex can keep its identity, edges and a few frequently read properties in +one connector while its bulky properties (long text, documents, payloads) live +in a table in **another** connector, keyed by the same value. Queries still see +one vertex with every property. + +```json +{ + "label": "Ticket", + "mappedGraphSource": { "connector": "mg" }, + "attributes": [ + { "name": "ticketId" }, + { "name": "priority" } + ], + "mappedContentSource": { + "connector": "pg", + "table": "ticket_content", + "key": "ticketId", + "keyColumn": "ticket_id", + "attributes": [ + { "name": "subject" }, + { "name": "description", "column": "body" } + ] + } +} +``` + +```gql +MATCH (t:Ticket)-[:ASSIGNED_TO]->(a:Agent) +WHERE t.subject CONTAINS 'login' +RETURN t.ticketId, t.description, a.name; +``` + +The content table is read only when a query filters on or returns one of its +properties. MemGQL then joins it in by `key`, fetching the matching rows in one +batch: a filter on a content property is evaluated in the content connector +first, and a plain projection is read after the traversal for exactly the +vertices it found. A query that touches no content property never reaches the +content connector. + +| `mappedContentSource` key | Required | Description | +|---------------------------|----------|-------------| +| `connector` | yes | A **relational** connector, different from the vertex's own. | +| `schema` | no | Middle namespace of the content table. | +| `table` | yes | The content table. | +| `key` | yes | A **property** declared in the vertex's own `attributes`; its value finds the content row. | +| `keyColumn` | no | The **column** in the content table that holds the `key` value. Defaults to `key`. | +| `attributes` | yes | The properties read from the content table, declared like [Attributes](#attributes); `column` and `path` refer to the content table. | + +**Rules and limits** (each is refused with a message saying what to write +instead): + +- Each property has exactly one home: a name may not appear both in the + vertex's `attributes` and in the content `attributes`, and the `key` is + declared only on the vertex. +- The content connector must be relational and differ from the vertex's + connector. Content in the same connector is just more attributes. +- A query that reads content properties must be a single `MATCH โ€ฆ RETURN`: no + `WITH`, `UNWIND`, `CALL`, `UNION` or `RETURN *`, and `OPTIONAL MATCH` may not + bind the vertex whose content is read. ### Edges @@ -333,6 +461,28 @@ entry of a `labels` list). } ``` +| Field | Description | +|-------|-------------| +| `label` | The relationship type. | +| `from` / `to` | The identifying **labels** of the source and target vertex. | +| `mappedTableSource` / `mappedGraphSource` / `mappedJoinSource` | Where the edge lives; exactly one is set. | +| `attributes` | Edge properties; see [Attributes](#attributes). Not allowed on a `mappedJoinSource` edge. | + +A native relationship type is often used between several label pairs. Declare +it once per pair; this is the one case where an edge label repeats within a +graph (a table-backed edge label stays unique): + +```json +{ "label": "inGroup", "from": "CAT:Product", "to": "CAT:Group", "mappedGraphSource": { "connector": "mg" } }, +{ "label": "inGroup", "from": "CAT:Product", "to": "CAT:Note", "mappedGraphSource": { "connector": "mg" } } +``` + +The declared ends also constrain a pattern end you leave bare: +`` MATCH (p:`CAT:Product`)-[:inGroup]->(g) `` reaches the backend as +`(g:CAT:Group)` with one pair declared and as `WHERE g:CAT:Group OR g:CAT:Note` +with two, never as an unconstrained `(g)` that would return nodes of labels +the graph does not declare. + A relational edge adds two more `metaFields` on top of `id`: | `metaFields` key | Description | @@ -408,8 +558,7 @@ instead): - Across connectors: the edge must be traversed **in a direction** (`-[:R]->` or `<-[:R]-`, not `-[:R]-`); no variable-length traversal, no `OPTIONAL MATCH`, one `MATCH` with one path pattern per query; every node in - the pattern needs a label, so each side can be routed; aggregating *across* - the join (`count()` over both sides) is not supported yet. + the pattern needs a label, so each side can be routed. - Within one relational connector the edge is pushed down as a single `JOIN` on the two key columns, so those limits do not apply: aggregates work, and an undirected step is accepted when the two endpoint labels differ. A @@ -659,3 +808,110 @@ the runtime DDL that builds the same catalog live. which graph defines each label. A query with no `USE` routes by the labels and declared properties it mentions; anything ambiguous returns an actionable error rather than a silent guess. + +## Glossary + +Every name a schema file can contain, grouped by where it appears. + +### Top level + +| Name | Meaning | +|------|---------| +| `connectors` | The backends to connect to, shared by all graphs. Alias: `catalogs`. | +| `graphs` | Named graphs, each with its own `vertices` and `edges`. | +| `vertices` / `edges` | At the top level instead of `graphs`: one implicit graph named `default`. | +| `_comment` | Free-text note, allowed at the top level and on a graph. | + +### Connector (`connectors[]`) + +| Name | Meaning | +|------|---------| +| `name` | The id sources use to point at this connector. | +| `type` | Backend kind, for example `postgres` or `memgraph`. | +| `connection` | How to reach the backend. | +| `connection.uri` / `path` | Connection string, or a file location for file-backed backends. | +| `connection.user` / `password` | Credentials. | +| `connection.database` / `schema` / `catalog` | The **physical** namespace the connection defaults to. | +| `connection.warehouse` / `role` / `private_key_path` / `token` | Snowflake session context and authentication. | +| `connection.tenant_id` / `client_id` / `client_secret` | Microsoft Fabric service-principal authentication. | +| `connection.language` | Default query language (`gql` / `cypher`) for graphs on this connector. | +| `connection.sslRootCert` / `trustServerCertificate` | TLS trust settings. | +| `connection.maxInListSize` | Most join-key values sent in one cross-backend `IN` list. | + +### Graph (`graphs[]`) + +| Name | Meaning | +|------|---------| +| `name` | The graph name, as in `USE `. | +| `vertices` / `edges` | The graph's elements. | +| `accessMode` | `readOnly` or `readWrite` (default). | +| `discover` | Native connectors whose labels and relationship types are added automatically. | + +### Vertex + +| Name | Meaning | +|------|---------| +| `label` | The exposed node label; unique within a graph. | +| `labels` | Several labels; the first identifies the mapping, the rest may be shared. | +| `mappedTableSource` / `mappedGraphSource` | Where the vertex lives; exactly one is set. | +| `mappedContentSource` | Optional table in another connector for some of its properties. | +| `attributes` | The properties read from the vertex's own source. | + +### Edge + +| Name | Meaning | +|------|---------| +| `label` | The relationship type. | +| `from` / `to` | The **labels** of the source and target vertex. | +| `mappedTableSource` / `mappedGraphSource` / `mappedJoinSource` | Where the edge lives; exactly one is set. | +| `attributes` | Edge properties; not allowed with `mappedJoinSource`. | + +### Sources + +| Name | Meaning | +|------|---------| +| `mappedTableSource` | A table in a relational connector. | +| โ†ณ `connector` | Which connector. Alias: `catalog`. | +| โ†ณ `schema` / `database` | Per-element override of the physical namespace. | +| โ†ณ `table` | Table, view or collection. | +| โ†ณ `metaFields.id` | **Column(s)** identifying a row; a list for a composite key. | +| โ†ณ `metaFields.from` / `to` | Edges: **column(s)** holding the source and target vertex's `id`. | +| `mappedGraphSource` | A label or relationship type in a native graph connector. | +| โ†ณ `connector` | Which connector. Alias: `catalog`. | +| โ†ณ `label` | The backend's own name, when it differs from the exposed one. | +| `mappedContentSource` | Some of a vertex's properties, read from a table in another connector. | +| โ†ณ `connector` / `schema` / `table` | Where the content table is. | +| โ†ณ `key` | The vertex **property** the content row is found by. | +| โ†ณ `keyColumn` | The content-table **column** holding the `key` value. | +| โ†ณ `attributes` | Properties read from the content table. | +| `mappedJoinSource` | An edge that exists where two property values are equal. | +| โ†ณ `fromKey` / `toKey` | **Properties** on the `from` and `to` vertex. | + +### Attribute (`attributes[]`) + +| Name | Meaning | +|------|---------| +| `name` | The exposed property name. | +| `type` | Declared type, default `String`; `Json` makes the column descendable. | +| `column` | The source column; defaults to `name`. | +| `path` | Location of the value inside a JSON `column`. | + +### Names with more than one meaning + +| Name | Where | Means | +|------|-------|-------| +| `catalog` | top level | alias for `connectors` | +| | in a source block | alias for `connector` | +| | in `connection` | the backend's physical catalog | +| `from` / `to` | on an edge | vertex **labels** | +| | in `metaFields` | **columns** | +| `label` | on an element | the exposed name | +| | in `mappedGraphSource` | the backend's name | +| `schema` / `database` | in `connection` | the connection-wide default | +| | in a source block | an override for that element | +| `path` | in `connection` | a file location | +| | on an attribute | a location inside a JSON column | + +Key fields name either a **column** or a **property**: `metaFields.*` and +`keyColumn` take columns of the source table; `key`, `fromKey` and `toKey` +take property names, which resolve through the element's `attributes`.