docs(api): add API conventions, migrations, and README

[jacomago]

Document API conventions including RSQL query syntax, bulk operations, jobs/batch definitions, and migration notes under dev-docs/api/.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[anderslindho]

Overall the v1 direction here is solid: plural nouns, correct method semantics, RFC 9457 errors, split request/response schemas, jobs for async work, RSQL replacing the old boolean-DSL magic chars. All good moves, agree with merging that as the baseline.

Where I think this needs another pass is the batch/bulk section, specifically re how it's meant to actually serve RecCeiver.

batch vs reconcile

:batch's mixed post/put/patch/delete shape still needs the producer to know what already exists in CF (to bucket into post vs put) and what to delete (no query-scoped delete, no server-side reconcile, per the doc's own wording). That's the same read-before-write pattern we're trying to get RecCeiver away from. The doc even says it outright: "a producer that does not track what it holds builds the operations by diffing its desired set against a lean GET /channels...". If RecCeiver still has to GET-then-diff to build the batch, that's the same read, just done before submitting instead of after.

Proposal: add an actual reconcile op. Producer submits "this is my desired/observed set for scope X, generation N," CF upserts and tombstones anything in-scope not mentioned. That's the real one-way primitive. Once that exists I think the mixed-method :batch shape can go away entirely, reconcile covers what it was trying to do for RecCeiver, and I don't think anyone else needs mixed-method batch (open Q below).

Reconcile doesn't get to skip jobs/per-item results just because it's declarative. Still large-N (thousands of channels per IOC per our own README numbers), still partially-failable, and a failed item must not get tombstoned (it didn't disappear, it just failed validation), so reconcile needs per-item results plus sync-under-threshold/async-over-threshold, same machinery batch has today, just wrapped around a "desired state" body instead of an op-list.

generic (non-reconcile) bulk, leaving this open

Once reconcile absorbs RecCeiver: is there still a real need for generic bulk create/update/delete (admin tooling, migration scripts, whatever)? If yes, and it's small-N/interactive (our own actor table already has admin/curator as "occasional, deliberate"), I don't think that needs jobs or per-item results at all. Validate-then-write, atomic, single response and status. Simpler for callers than today's 200-with-embedded-422s. If there's actually a consumer doing large/failure-prone homogeneous bulk writes outside RecCeiver, that path needs per-item handling too, same as reconcile, just without tombstoning. Need to know if that consumer exists before deciding.

200 vs 207 vs job, the doc contradicts itself here

Hard rules say "never 200 with an error body." Async section says "never... a 207 Multi-Status." Sync :batch's own example is a 200 with embedded per-item 422s. All three can't be true at once. If the intent is ":batch is a command, 200 = command ran, per-item statuses are data not errors," fine, but that needs to be written as an explicit carve-out to the hard rule, not left implicit.

"mirrors ES _bulk", ground this in domain not backend

Non-atomicity and the 200-status choice for batch are both justified by "follows Elasticsearch _bulk," but the doc is explicit elsewhere (opaque cursor) that engine behavior shouldn't leak into the contract. Same conclusion probably still holds, just ground it properly: large batches need forward progress, one bad item shouldn't sink 5k good ones, and no backend gives free cross-doc atomicity at this scale anyway. That reasoning would survive a hypothetical backend swap. The ES justification wouldn't.

native concepts needed for reconcile

For tombstoning to be safe I think we need, minimum:

• an opaque source_id/scope key, producer-supplied, CF doesn't interpret it (same idea as the opaque pagination cursor already in here)
• a generation/sequence guard per scope, so a delayed or reordered reconcile submission can't tombstone state a newer one already established
• first-class lifecycle fields on the channel (state, last_seen_at, tombstoned_at) instead of inferring lifecycle from arbitrary metadata, makes purge policy simpler to query, lets GET /channels default-exclude tombstoned channels, and gives an audit trail without extra work

Don't think we need IOC / RecCeiver / session to become their own CF resources though, scope just needs to be whatever unit appears/disappears together. RecCeiver owns getting that scoping right. CF just needs the opaque key plus generation guard.

one more thing this forces us to decide now, not later: can one channel have multiple hosts?

Separately from all of the above, PVs can be hosted by more than one IOC at once. Today CF pretends this can't happen, so when more than one recceiver/IOC reports the same name, whichever write lands last just silently clobbers the previous metadata. No visibility that it happened.

This lands directly on the source_id-scoped reconcile design above, not just future work. As sketched, if source A and source B both claim channel X, B's reconcile submission would silently overwrite A's metadata for X, and A's next reconcile would silently overwrite it back. Same bug we're trying to kill, just now inside the new primitive instead of the old one. So reconcile can't be specced without deciding: is an upsert a per-source claim on a channel, or a global clobber?

I think existence and metadata split cleanly here. Existence can aggregate now: channel exists if ≥1 source claims it, tombstone only when the last claim drops, that's just reference counting, solvable today. Metadata conflict is harder: two sources reporting different properties/tags for the same name doesn't aggregate cleanly, and I don't think we need to solve that now (whether we warn, reject, or just surface both isn't decided). But the identity question, keyed by name alone vs. some (name, ioc_id)-style composite, exact keying TBD, needs to at least be flagged as open now, because retrofitting single-owner -> multi-claim later is a lot more expensive than leaving room for it in the reconcile primitive from the start.

[jacomago]

batch vs reconcile

I can swap to reconcile.

native concepts needed for reconcile

For tombstoning to be safe I think we need, minimum:

I still don't understand tombstoning, is this a feature inside of channelfinder? Or is this a client is pushing some extra metadata?

Can you give an example of using just the normal update a single channel example?

one more thing this forces us to decide now, not later: can one channel have multiple hosts?

urgl, I don't particularly want to tackle this as it requires doing an elastic migration. The channel name is the key in the channels 'table'. Adding a source_id (which I think would have to be recceiver:ioc_id) Would mean adding a migration to elastic which we've never done before. The support for migrations also kind of ... is not nearly as good as flyway is for postgres.

Which completely changes the scope of this PR. Adding support for a source_id, which I think is pretty easily added on top instead, we modify the Channel Response and Request object with an optional field source_id. This would not break a v1 api. I think it does not need to be added now for this reason.

[simon-ess]

Typo: shouldn't this be domain-addressable?

[simon-ess]

Should this perhaps be optional? Given that we seem to be moving towards all services requiring authentication here at ESS, should we not consider this configurable?

[simon-ess]

This is a bad example. There are no kebabs here.

[simon-ess]

There is some ambiguity here: What if I have a property named tag?

[jacomago]

Nice catch.

[simon-ess]

If we go this direction (I share @anderslindho 's concerns) shouldn't this return the entire object instead of just a 201?

[simon-ess]

What is the use case for this instead of allowing

POST /channels:batch
{
   [ { "name": "SR:C01", "properties": { "cell": "1" }, "tags": ["active"] } ]
}

vs

PUT /channels:batch
{
  [ { "name": "SR:C02", "properties": { "cell": "2" }, "tags": [] } ]
}

etc.? Why do we need to create, update, update, and delete all at once?

[jacomago]

Well, my thought was that recceiver would be the one deleting channels, but this is really the crux between 'resolve' vs 'batch'. Does the recceiver calculate the sync or does channelfinder.

[simon-ess]

Do we need to reap these? I think of awx where the jobs are there in perpetuity.

That said, maybe we don't care to revisit job logs years after the fact 🙃

[jacomago]

Process would run after every single ioc restart. So thats a lot of jobs.

[simon-ess]

Eh... it's not like we will run out of integers. And looking at other utilties, we have typically one or more jobs per IOC start, stop, deploy, undeploy, which is of the same order....

[simon-ess]

Depends how long you keep the jobs for!

[simon-ess]

I still don't understand tombstoning, is this a feature inside of channelfinder? Or is this a client is pushing some extra metadata?

From what I read (e.g. https://en.wikipedia.org/wiki/Tombstone_(data_store)) it seems not obviously relevant in our case as we do not have a distributed database to deal with, but perhaps it is the correct approach regardless?

[anderslindho]

Well, my thought was that recceiver would be the one deleting channels, but this is really the crux between 'resolve' [sic] vs 'batch'. Does the recceiver calculate the sync or does channelfinder.

ChannelFinder - recCeiver always just pushes what it has. Else it needs state/db. Good that you're up for swapping to reconcile. @simon-ess's "why do we need to create, update, update, and delete all at once?" on the batch example is the same question from the other end I think.

I still don't understand tombstoning, is this a feature inside of channelfinder? Or is this a client is pushing some extra metadata?

Can you give an example of using just the normal update a single channel example?

(SR) From what I read (e.g. https://en.wikipedia.org/wiki/Tombstone_(data_store)) it seems not obviously relevant in our case as we do not have a distributed database to deal with, but perhaps it is the correct approach regardless?

TBH I am not 100% sure if it is the correct term, but I think it covers it. What I mean by it is that CF alone is responsible for changing channels from active to inactive. So my intention is basically soft delete + retention/purge policy, and the important part is who owns that (CF).

To answer the either/or directly: today it is a client pushing extra metadata. That's pvStatus. Client decides a channel went away, writes that decision into CF as a property. That's the pattern I want gone.

Example

Say SOME:AI stops being reported.

Today: recCeiver reads CF, works out that SOME:AI is no longer in its set, and writes the deactivation itself (pvStatus -> Inactive, or a delete). CF is a dumb store, the client holds the logic.

Proposed: recCeiver sends its current set for its scope, SOME:AI isn't in it, CF notices it's in-scope-but-absent and does the equivalent of PATCH /channels/SOME:AI {"state": "inactive"} on its own. Record stays, GET /channels excludes it by default, it comes back active on the next upsert that mentions it. recCeiver never reads and never computes a diff.

one more thing this forces us to decide now, not later: can one channel have multiple hosts?

urgl, I don't particularly want to tackle this as it requires doing an elastic migration. [...]

Which completely changes the scope of this PR. Adding support for a source_id, which I think is pretty easily added on top instead, we modify the Channel Response and Request object with an optional field source_id. This would not break a v1 api. I think it does not need to be added now for this reason.

I think we should separate API design from business logic design. What I think is important is that the API already now accommodates for this design. How exactly that is implemented and/or how dbs are migrated to support it is a separate issue.

Not asking for the migration, and definitely not in this PR. But "add it on top later" only holds for the field, not for the semantics. Schema-wise you're right, optional source_id is additive. Semantically no: whether an upsert is a per-source claim or a global clobber is observable behavior clients build on. Change it later and the same request does something different.

And it's not really future work, it's load-bearing for reconcile being correct. With a single source_id, a channel hosted by two IOCs gets deactivated by whichever one drops it first, even though the other still has it. Live PVs flapping in and out of GET /channels. The old failure was two writes clobbering each other; the new one is CF soft-deleting things on its own bad inference. Worse, because now it's our logic doing it.

Doesn't mean solving it now. But reconcile's rule needs to be "deactivate when nothing claims it" rather than "deactivate when the recorded source dropped it", and v1 can implement that with one claim per channel as a documented limitation. Free to write down now, expensive once clients have built on one-source-per-channel being the actual rule.

[shroffk]

one more thing this forces us to decide now, not later: can one channel have multiple hosts?

Separately from all of the above, PVs can be hosted by more than one IOC at once. Today CF pretends this can't happen, so when more than one recceiver/IOC reports the same name, whichever write lands last just silently clobbers the previous metadata. No visibility that it happened.

During the meeting we agreed that I will be adding a "preprocessor" SPI similar to the "processor" one that exists.

So each site can add their own validation for channels... one of these can be a validator that detects if existing/active channels are there in channelfinder which have different hosts, it can then log or notify the appropriate entities to resolve the issue.

I don't think we should allow multiple properties with the same name.

[tynanford]

Overall I really like this. RSQL queries would be great.

I'm also thinking more about the duplicate PV names. I agree with @anderslindho that it is good to think about even if we don't address/support it fully in v1. I'm not saying this is practical but if PV name is the key and a PV can have multiple iocid, hostname, etc., then a channel in CF could be represented like this:

{
    "name": "SR12C:SQSF1:Setpoint",
    "properties": {
      "Acc": "SR",
      "Family": "SQSF",
      "Field": "Setpoint",
      "Sector": "12",
      "Device": "1",
      "Position": "187.277343",
      "hostName": "ioc-host-a.example.local",
      "iocName": "skewquads2",
      "iocid": "192.168.20.11:45334",
      "iocIP": "192.168.20.11",
      "time": "2026-07-15 09:12:00.123456",
      "caPort": "45502",
      "recordType": "ao",
      "recordDesc": "Current setpoint",
      "archive": "Slow",
      "Engineer": "jdoe",
      "Location": "SoftIOC"
    },
    "pvStatus": "Active",
    "conflict": {
      "duplicateActiveSources": true,
      "activeSourceIds": [
        "recceiver-1.example.local:192.168.20.11:45334",
        "recceiver-2.example.local:192.168.20.12:52110"
      ],
      "note": "flattened properties above reflect one of these sources arbitrarily; do not treat as authoritative until resolved"
    },
    "tags": ["MML", "PS"],
    "sources": [
      {
        "source_id": "recceiver-1.example.local:192.168.20.11:45334",
        "pvStatus": "Active",
        "last_seen": "2026-07-15T09:12:00Z",
        "properties": {
          "recceiverID": "recceiver-1.example.local",
          "iocName": "skewquads2",
          "hostName": "ioc-host-a.example.local",
          "iocid": "192.168.20.11:45334",
          "iocIP": "192.168.20.11",
          "time": "2026-07-15 09:12:00.123456",
          "caPort": "45502",
          "recordType": "ao",
          "recordDesc": "Current setpoint",
          "archive": "Slow",
          "Engineer": "jdoe",
          "Location": "SoftIOC"
        }
      },
      {
        "source_id": "recceiver-2.example.local:192.168.20.12:52110",
        "pvStatus": "Active",
        "last_seen": "2026-07-15T09:10:45Z",
        "properties": {
          "recceiverID": "recceiver-2.example.local",
          "iocName": "skewquads2-test",
          "hostName": "ioc-host-b.example.local",
          "iocid": "192.168.20.12:52110",
          "iocIP": "192.168.20.12",
          "time": "2026-07-15 09:10:45.987654",
          "caPort": "45519",
          "recordType": "ao",
          "recordDesc": "Current setpoint (test IOC)",
          "archive": "Slow",
          "Engineer": "jdoe",
          "Location": "SoftIOC"
        }
      },
      {
        "source_id": "recceiver-1.example.local:192.168.20.13:38821",
        "pvStatus": "Inactive",
        "last_seen": "2026-06-20T14:22:03Z",
        "properties": {
          "recceiverID": "recceiver-1.example.local",
          "iocName": "skewquads2",
          "hostName": "ioc-host-old.example.local",
          "iocid": "192.168.20.13:38821",
          "iocIP": "192.168.20.13",
          "time": "2026-06-20 14:22:03.456789",
          "caPort": "45506",
          "recordType": "ao",
          "recordDesc": "Current setpoint",
          "archive": "Slow",
          "Engineer": "jdoe",
          "Location": "SoftIOC"
        }
      }
    ]
  }

Each source in this case is an IOC connection to a specific recceiver instance and includes all the properties that IOC reported. The top level properties is the flattened view for normal clients and includes properties manually added by non-recceiver clients. The top level pvStatus is an OR of all the source pvStatus fields and there is a conflict flag in case there are multiple active sources.

There are some problems with this idea (including an IOC restart adding a new source every time) but throwing it out there to brainstorm.

[tynanford]

oh that json blob reminded me of an old complaint about CF here: all the property values are strings so things like position in the example above are hard to sort on unless you make all positions the same number of characters. We added leading zeros in those cases. Not sure if it can be addressed or makes sense but did come up here