Serve saved activities and objects of every type over ActivityPub

CHANGES.md

@@ -8,6 +8,16 @@ To be released.
 
 ### DrFed
 
+ -  Activities and objects saved by the common write operations are now
+    served over ActivityPub as they were saved, with their extension
+    properties and without blind recipients, also inside extension values.
+    Published activities of every type appear in the actor's outbox beside
+    existing `Create` activities. Earlier activities keep the versions they
+    embedded after an edit.  A deleted object is answered with a `Tombstone`,
+    and its `Delete` is the only activity about it that remains retrievable.  A
+    Public activity that embeds content addressed to particular recipients,
+    even inside an extension value, is not served. Existing IRIs do not change.
+    [[#153], [#177] by ChanHaeng Lee\]
  -  Added immutable document histories for local object creation and accepted
     incoming Create activities. Database upgrades retain existing object and
     activity documents, including those of deleted objects. Back up and stop
@@ -28,8 +38,10 @@ To be released.
 [#135]: https://github.com/fedify-dev/drfed/issues/135
 [#137]: https://github.com/fedify-dev/drfed/pull/137
 [#150]: https://github.com/fedify-dev/drfed/issues/150
+[#153]: https://github.com/fedify-dev/drfed/issues/153
 [#164]: https://github.com/fedify-dev/drfed/pull/164
 [#167]: https://github.com/fedify-dev/drfed/pull/167
+[#177]: https://github.com/fedify-dev/drfed/pull/177
 
 ### GraphQL APIs
 
@@ -73,6 +85,16 @@ To be released.
     revision ID is stale, and object histories can end with a whiteout deletion
     marker.
     [[#150], [#164] by Hong Minhee\]
+ -  Added the `@drfed/federation/serving` subpath.  It exports
+    `publicDocument()`, the serialization federation reads use for saved
+    documents, `parseVocab()`, which parses such a document into a Fedify
+    vocabulary object, and `loadActivitySnapshot()`.
+    [[#153], [#177] by ChanHaeng Lee\]
+ -  Added `extensions` to the nodes `describeDocument()` in
+    `@drfed/models/document` returns: the JSON-LD node objects in their
+    extension values, with the standard properties their keys stand for,
+    their types and their IRI references.
+    [[#153], [#177] by ChanHaeng Lee\]
  -  Added `validateDocument()` in `@drfed/models/document` for parsed
     ActivityStreams documents. It returns structured errors and compatibility
     warnings with original nested field and list-entry paths, preserves custom

CONTRIBUTING.md

@@ -663,11 +663,17 @@ An inbound delivery that was verified and answered with 2xx is linked to
 `activities` by the activity IRI it recorded, through `linkInboundActivity`,
 never by what a listener reports.
 
+The dispatchers serve resources without legacy typed rows from their current
+revisions through *src/serving/*, exported as `@drfed/federation/serving`.
+The activity dispatchers, outbox pages and outbox counter share the predicates
+in *src/visibility.ts*; change them together.  See [federation read internals].
+
 See [inbox persistence internals] for acceptance and idempotence, and
 [activity delivery internals] for completion reports, receipts and queue
 observations.
 
 [local write internals]: https://drfed.org/internals/local-writes
+[federation read internals]: https://drfed.org/internals/federation-reads
 [inbox persistence internals]: https://drfed.org/internals/inbox-persistence
 [activity delivery internals]: https://drfed.org/internals/activity-deliveries
 

changes.d/drfed/serve-saved-activities.md

@@ -0,0 +1,15 @@
+---
+links:
+  '#153': https://github.com/fedify-dev/drfed/issues/153
+  '#177': https://github.com/fedify-dev/drfed/pull/177
+---
+ -  Activities and objects saved by the common write operations are now
+    served over ActivityPub as they were saved, with their extension
+    properties and without blind recipients, also inside extension values.
+    Published activities of every type appear in the actor's outbox beside
+    existing `Create` activities. Earlier activities keep the versions they
+    embedded after an edit.  A deleted object is answered with a `Tombstone`,
+    and its `Delete` is the only activity about it that remains retrievable.  A
+    Public activity that embeds content addressed to particular recipients,
+    even inside an extension value, is not served. Existing IRIs do not change.
+    [[#153], [#177] by ChanHaeng Lee]

changes.d/javascript/federation-serving.md

@@ -0,0 +1,16 @@
+---
+links:
+  '#153': https://github.com/fedify-dev/drfed/issues/153
+  '#177': https://github.com/fedify-dev/drfed/pull/177
+---
+ -  Added the `@drfed/federation/serving` subpath.  It exports
+    `publicDocument()`, the serialization federation reads use for saved
+    documents, `parseVocab()`, which parses such a document into a Fedify
+    vocabulary object, and `loadActivitySnapshot()`.
+    [[#153], [#177] by ChanHaeng Lee]
+
+ -  Added `extensions` to the nodes `describeDocument()` in
+    `@drfed/models/document` returns: the JSON-LD node objects in their
+    extension values, with the standard properties their keys stand for,
+    their types and their IRI references.
+    [[#153], [#177] by ChanHaeng Lee]

docs/.vitepress/config.mts

@@ -18,6 +18,16 @@ import { defineConfig } from "vitepress";
 
 import { forEngine } from "../../scripts/images/engines.mts";
 
+// A ghcr.io/fedify-dev/drfed image tag in the documentation.
+const IMAGE_REFERENCE = /\bghcr\.io\/fedify-dev\/drfed:\d[0-9A-Za-z.-]*/gu;
+
+// Star-orbit symbol from https://github.com/hackers-pub/visual-identity
+// (exports/starorbit-transparent-17x12.svg), © 2025 Bak Eunji,
+// CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0/>.
+// Modified: recolored to currentColor and stripped of IDs and clip paths.
+const HACKERS_PUB_ICON =
+  '<svg xmlns="http://www.w3.org/2000/svg" width="17.995798" height="12.67316" viewBox="0 0 17.995798 12.67316"><title>hackers.pub</title><desc>Star-orbit symbol by Bak Eunji (hackers-pub/visual-identity), CC BY-SA 4.0, recolored to currentColor</desc><g transform="translate(-20112.718,-703.56027)"><g><path d="M 0,0 C -2.542,-0.369 -5.396,-0.247 -5.483,0.591 -5.569,1.429 -2.68,2.344 -0.147,2.771 2.499,3.216 5.155,3.372 5.575,2.487 5.993,1.607 2.738,0.398 0,0 M -0.07,3.07 C -3.963,2.432 -6.437,1.398 -6.522,0.163 -6.59,-0.819 -4.481,-1.731 0.28,-1.046 c 5.038,0.724 7.075,2.5 6.635,3.479 -0.45,1 -2.977,1.293 -6.985,0.637" style="fill:currentColor;fill-opacity:1;fill-rule:nonzero;stroke:none" transform="matrix(1.3333333,0,0,-1.3333333,20121.416,714.52093)" /><path d="m 0,0 c -0.909,0.834 -0.652,1.825 -1.553,1.825 -0.876,0 -0.62,-0.905 -1.551,-1.825 -0.623,-0.669 -2.434,-0.14 -2.434,-1.214 0,-0.924 1.364,-0.764 2.306,-1.659 0.745,-0.708 0.711,-2.045 1.679,-2.045 0.884,0 0.901,1.516 1.584,2.199 0.957,0.888 2.255,0.578 2.255,1.505 C 2.286,-0.225 0.8,-0.652 0,0" style="fill:currentColor;fill-opacity:1;fill-rule:nonzero;stroke:none" transform="matrix(1.3333333,0,0,-1.3333333,20123.704,705.9936)" /></g></g></svg>';
+
 // A link to the other deployment (the release site from canary and the other
 // way around), set at build time.  Neither variable set means no extra link;
 // setting only one of them is a misconfigured deployment and fails the build.
@@ -48,23 +58,15 @@ if (
     `DRFED_IMAGE_VERSION must be a published version; got ${JSON.stringify(DRFED_IMAGE_VERSION)}.`,
   );
 }
-const imageReference = /\bghcr\.io\/fedify-dev\/drfed:\d[0-9A-Za-z.-]*/gu;
 function publishedImage(text: string): string {
   return DRFED_IMAGE_VERSION === undefined
     ? text
     : text.replaceAll(
-        imageReference,
+        IMAGE_REFERENCE,
         `ghcr.io/fedify-dev/drfed:${DRFED_IMAGE_VERSION}`,
       );
 }
 
-// Star-orbit symbol from https://github.com/hackers-pub/visual-identity
-// (exports/starorbit-transparent-17x12.svg), © 2025 Bak Eunji,
-// CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0/>.
-// Modified: recolored to currentColor and stripped of IDs and clip paths.
-const HACKERS_PUB_ICON =
-  '<svg xmlns="http://www.w3.org/2000/svg" width="17.995798" height="12.67316" viewBox="0 0 17.995798 12.67316"><title>hackers.pub</title><desc>Star-orbit symbol by Bak Eunji (hackers-pub/visual-identity), CC BY-SA 4.0, recolored to currentColor</desc><g transform="translate(-20112.718,-703.56027)"><g><path d="M 0,0 C -2.542,-0.369 -5.396,-0.247 -5.483,0.591 -5.569,1.429 -2.68,2.344 -0.147,2.771 2.499,3.216 5.155,3.372 5.575,2.487 5.993,1.607 2.738,0.398 0,0 M -0.07,3.07 C -3.963,2.432 -6.437,1.398 -6.522,0.163 -6.59,-0.819 -4.481,-1.731 0.28,-1.046 c 5.038,0.724 7.075,2.5 6.635,3.479 -0.45,1 -2.977,1.293 -6.985,0.637" style="fill:currentColor;fill-opacity:1;fill-rule:nonzero;stroke:none" transform="matrix(1.3333333,0,0,-1.3333333,20121.416,714.52093)" /><path d="m 0,0 c -0.909,0.834 -0.652,1.825 -1.553,1.825 -0.876,0 -0.62,-0.905 -1.551,-1.825 -0.623,-0.669 -2.434,-0.14 -2.434,-1.214 0,-0.924 1.364,-0.764 2.306,-1.659 0.745,-0.708 0.711,-2.045 1.679,-2.045 0.884,0 0.901,1.516 1.584,2.199 0.957,0.888 2.255,0.578 2.255,1.505 C 2.286,-0.225 0.8,-0.652 0,0" style="fill:currentColor;fill-opacity:1;fill-rule:nonzero;stroke:none" transform="matrix(1.3333333,0,0,-1.3333333,20123.704,705.9936)" /></g></g></svg>';
-
 export default defineConfig({
   lang: "en-US",
   title: "DrFed",
@@ -165,6 +167,7 @@ export default defineConfig({
           { text: "Runtime lifecycle", link: "/internals/runtime-lifecycle" },
           { text: "Inbox persistence", link: "/internals/inbox-persistence" },
           { text: "Local writes", link: "/internals/local-writes" },
+          { text: "Federation reads", link: "/internals/federation-reads" },
           {
             text: "Activity deliveries",
             link: "/internals/activity-deliveries",

docs/internals/federation-reads.md

@@ -0,0 +1,166 @@
+Federation reads
+================
+
+The federation package serves local resources through the object dispatchers
+in *packages/federation/src/object-dispatchers.ts* and the outbox dispatcher in
+*packages/federation/src/collection.ts*.  Both read through
+*packages/federation/src/serving/*, and share the SQL predicates in
+*packages/federation/src/visibility.ts*, so that a resource is served the same
+way wherever it is read.
+
+
+Routes
+------
+
+| Route                   | Serves                                                    |
+| ----------------------- | --------------------------------------------------------- |
+| `/users/{actor}/{id}`   | Objects attributed to the actor, including a `Question`   |
+| `/ap/creates/{id}`      | `Create` activities                                       |
+| `/ap/activities/{id}`   | Every other activity, including a `Question` saved as one |
+| `/users/{actor}/outbox` | The actor's published activities                          |
+
+The common write operations allocate an IRI from the route of the resource's
+type, so a route answers 404 for a resource allocated to another one.  No IRI
+changes: legacy `Create` activities stay at `/ap/creates/{id}`.
+
+
+Sources
+-------
+
+A `Note` or `Article` created by the `createObject` mutation, and its `Create`,
+have legacy typed rows and are serialized from them by `toObject()` and
+`toCreate()`, as before, also after an edit keeps the rows in step.
+
+Every other resource, saved by the [common write operations](local-writes.md),
+is served from the document of its current revision:
+
+ -  The document is served as saved, with its context, extension properties,
+    IRI references and embedded values.  An activity keeps the version it
+    embedded when it was saved, so editing an object does not change an
+    earlier activity's response.  Nothing is dereferenced.
+ -  `bto` and `bcc` are removed from the root, from every embedded
+    document, and from every JSON-LD node object in extension values, which
+    a JSON-LD processor reads with the context it inherits.  They are found
+    by meaning through `describeDocument()`, so aliases of them are removed
+    as well; in extension values, whatever the alias's container.  JSON
+    literals stay as saved, however their keywords are spelled.
+ -  A document saved without `@context` states the ActivityStreams context,
+    which the validator read it with and which Activity Streams 2.0 asks
+    documents to include.
+ -  The document must name its own resource in `id`.  Fedify does not parse
+    it: the dispatchers hand Fedify a vocabulary object whose JSON-LD is the
+    document, so a document with values Fedify 2.4.3 cannot read, such as a
+    `Link` in `audience` or `object`, is served as saved too.
+ -  A document the write path does not save, one the validator cannot read
+    or one naming another resource, can only come from a change made outside
+    it.  It is logged under `["drfed", "federation", "serving"]` and left
+    out: its resource answers 404, and outbox pages and the counter skip it.
+    Each revision is logged once per process, however often it is read,
+    while it is among the last 4,096 logged.
+
+
+Conditions
+----------
+
+Every read needs the request's authority to be the instance of a live local
+actor, and Public `to` or `cc` addressing in the resource's current
+`addressing` index.  Moreover:
+
+ -  An activity must be in the outbox of such an actor.  A saved activity that
+    was not published is not served, nor is a received one.
+ -  A legacy `Create` is not served once its object is deleted.  A saved
+    document is not served when the `object` of its root or of an embedded
+    activity refers to a deleted local object, unless that activity is a
+    `Delete`: the deletion hides the `Like` and other activities about the
+    object, and an `Undo` of such a `Like`, and leaves its `Delete`
+    retrievable, also inside an `Announce` that embeds it.  A local object is
+    deleted when its typed row is, or when its current revision is a local
+    whiteout.
+ -  A saved document that embeds a deleted local object, at any depth, is
+    not served, so the `Create` and `Update` activities that carried its
+    content, and an `Undo` of a `Like` embedding it, are hidden.
+ -  A saved document that embeds, at any depth, a document addressed to
+    recipients but not to the public in its `to` or `cc` is not served,
+    since its content is theirs.  Its recipients are read as the
+    `addressing` index records them: references as resolved, embedded
+    documents by `id`, and `Link` values by `href`.  An embedded document
+    without recipients of its own goes with the document that embeds it, as
+    the `Like` in an `Undo` usually does.
+ -  These rules read the node objects in extension values as embedded
+    documents, with their values through `@set` and `@list` objects and
+    their recipients by `href` when a type is `Link`, and by `id`
+    otherwise.  A node object whose `object` refers to a deleted
+    local object, by a reference or a `href`, hides the document unless its
+    type is `Delete`.  A saved document is not served when a node object in
+    an extension value is one `describeDocument()` leaves uninterpreted: one
+    with a `@context` of its own, which the validator does not read, or with
+    a value object, a language map, a string its key does not coerce to an
+    IRI, or a relative reference where a reference belongs, more than one
+    `id` or `href`, properties nested by `@nest` or reversed by `@reverse`,
+    a value or a list object with keys other than keywords, or with keys a
+    context of its own may alias, or a value that is an object but no JSON
+    literal, since its recipients and references cannot then be told.  A
+    blank node identifier is read as an `id` that names no resource, but
+    not as a reference, since it may name a node object defined apart.  A
+    node object whose `id` is used elsewhere in the document, as another
+    node object's `id` or as a reference outside its own properties, is one
+    node with what uses it, and is not read apart either; a `Link` whose
+    `href` is its own `id` is read.  Language maps are read as strings, and
+    an object in one, which is no valid JSON-LD, is not read; a list under
+    a key with a `@language` container is no language map, and its node
+    objects are read.
+ -  Embedded documents that share an `id` are one document, which none of
+    them tells, so a saved document with two is not served either.  An
+    `object` reference names the `href` of a `Link` the document embeds
+    elsewhere with that `id` as well.  Such documents are logged once per
+    revision as a warning.
+ -  An object must be attributed to the actor of its route by its last
+    snapshot.  A deleted object is answered with `410 Gone` and a `Tombstone`
+    of its `id` and `deleted` time only.
+
+What a saved document embeds and refers to is read from the document, through
+`describeDocument()`; every other condition is SQL.
+
+
+Outbox
+------
+
+Outbox pages hold legacy and saved activities in one order: ascending
+`position` of the outbox's `collection_items`, where the newest item has the
+smallest position.  An item without a position is not listed, and its activity
+is not served either.  The cursor is the position of the last item of the
+previous page.
+
+The saved activities their documents do not allow are left out after rows are
+selected, so a page reads further batches until it is full, each twice as
+large as the one before, up to 1,000 items.  It promises a next page only when
+a served activity follows, so only the last page is short and no page is empty
+while another follows.  A page after many hidden activities therefore reads
+all of them.
+
+The counter counts what the pages hold.  It reads the document of every saved
+activity in the outbox on each request, 100 at a time, so its cost grows with
+the outbox.  Deleted objects are looked up 1,000 IRIs at a time, below the
+number of parameters PostgreSQL binds in one statement.
+
+
+Outgoing payloads
+-----------------
+
+`@drfed/federation/serving` exports the same serialization for outgoing
+payloads:
+
+ -  `publicDocument(document)` returns a copy without blind recipients and
+    with a stated context, or `null` for a document the validator cannot
+    read offline or whose blind recipients cannot be found, as in a node
+    object with a `@context` of its own in an extension value.
+ -  `parseVocab(ctx, document)` parses a document into a vocabulary object
+    that keeps it as its JSON-LD.  It throws for a document Fedify cannot
+    read, such as one with a `Link` in `audience` or `object`.
+ -  `loadActivitySnapshot(db, id)` loads a saved activity's current document,
+    blind recipients included, and its revision ID, applying no condition
+    above.
+
+An object derived from a parsed one, as by `clone()`, loses that JSON-LD and is
+serialized from the properties Fedify knows, without extensions.  Delivery is
+not part of these reads.