Skip to content

feat: switch uploads to the reva upload coordinator - #3839

Open
butonic wants to merge 13 commits into
workpagagesfrom
port-upload-coordinator
Open

butonic wants to merge 13 commits into
workpagagesfrom
port-upload-coordinator

Conversation

@butonic

@butonic butonic commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Switches opencloud to the reva upload coordinator: uploads are initiated, received, postprocessed and committed by pkg/upload instead of the storage drivers.

  • reva is bumped to main (036856689), which now carries the coordinator and the other WP1 storage changes since Merge Workpackages back to main reva#957 merged workpackages into main.
  • The ocm storage and data providers stage uploads in <storage root>/uploads, and the storage-users dataprovider gets the mount id.
  • storage-users passes its mount id to the drivers, and postprocessing sets the resource id on its result, so the coordinator ignores postprocessing results for other storages.
  • opencloud storage-users uploads sessions lists the sessions through the coordinator, since the drivers no longer keep them.
  • New --orphaned filter for uploads sessions: it lists the sessions whose node metadata can no longer be read, and --clean removes them and gives their quota back.

Assisted by Claude Code (claude-opus-5-5): it rebased the branch onto main, bumped reva to workpackages, and drafted this description.

@codacy-production

codacy-production Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Not up to standards ⛔

🔴 Issues 1 medium

Alerts:
⚠ 1 issue (≤ 0 issues of at least minor severity)

Results:
1 new issue

Category Results
Security 1 medium

View in Codacy

🟢 Metrics 8 complexity

Metric Results
Complexity 8

View in Codacy

🟢 Coverage ∅ diff coverage · +0.01% coverage variation

Metric Results
Coverage variation ✅ +0.01% coverage variation (-1.00%)
Diff coverage ∅ diff coverage

View coverage diff in Codacy

Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (9c7ecc1) 93830 27062 28.84%
Head commit (18fc08c) 93867 (+37) 27079 (+17) 28.85% (+0.01%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#3839) 0 0 ∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@butonic butonic added the Type:Maintenance E.g. technical debt, packaging, etc. label Oct 27, 2026
@butonic
butonic force-pushed the port-upload-coordinator branch from f1e3f7d to c47e107 Compare November 6, 2026 14:16
@butonic butonic changed the title port the upload coordinator feat: switch uploads to the reva upload coordinator Nov 6, 2026
LarsJurgensen and others added 11 commits November 12, 2026 11:11
Co-authored-by: Firas Frikha <firas.frikha@kiteworks.com>
Pins reva to the workpackages branch (2d4db7d27), which carries the
upload coordinator and the other WP1 storage changes on top of reva
main. Once workpackages lands in reva main this becomes a regular bump
to main.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
Signed-off-by: Firas Frikha <firas.frikha@kiteworks.com>
Upload sessions whose node metadata can no longer be read can never
finish postprocessing, stay in processing state indefinitely and keep
consuming the quota of their space.

Wire up the Orphaned session filter so operators can list the affected
sessions and remove them with --clean:

    opencloud storage-users uploads sessions --orphaned
    opencloud storage-users uploads sessions --orphaned --clean

Signed-off-by: Julian Koberg <julian.koberg@kiteworks.com>
The driver no longer consumes postprocessing events: the upload
coordinator does, in the "dcfs" group, and the driver only consumes
DeleteRevision in "dcfs-revisions". The command still disables async
uploads so it does not join that group.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
…on PostprocessingFinished

The storage drops postprocessing events that belong to another storage,
which needs its mount id and the resource id on the event.
Postprocessing results are dropped when their storage id does not match
the driver's mount id. Only the decomposed driver was given a mount_id,
so under decomposeds3 there was nothing to compare against.

Also pass it in DecomposedNoEvents and DecomposedS3NoEvents.
The upload coordinator reads the mount id from the driver config to drop
postprocessing results that belong to another storage. The dataprovider
does not add it to the driver config the way the storageprovider does,
so without it the coordinator accepts every result for posix.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
The remote instance still sees the share with the role it was created
with, so the upload coordinator refuses the upload there instead of the
owner's instance failing it with 500.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
Picks up opencloud-eu/reva#978: a failed blob
commit now logs the space and node id, and the last attempt logs that
we give up. This runs the rest of the `owncloud/reva#733` port through
the opencloud CI.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
Picks up the space manager checks from
opencloud-eu/reva#990 and
opencloud-eu/reva#993.

go-chi and go-jose move up because reva requires the newer versions.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
butonic force-pushed the port-upload-coordinator branch from 6b43871 to 0da2e89 Compare November 12, 2026 10:16
@butonic
butonic changed the base branch from main to workpagages November 12, 2026 10:16
@butonic
butonic marked this pull request as ready for review November 12, 2026 15:50
@butonic butonic self-assigned this Nov 12, 2026
@butonic
butonic requested a review from dragonchaser November 12, 2026 15:51

@pbleser-oc pbleser-oc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Since the CI only runs on posix, I ran the upload features locally on the decomposed driver, with async uploads and ClamAV: apiSpaces/uploadSpaces, apiSpaces/tusUpload, coreApiWebdavUploadTUS/uploadFile and checksums, and cliCommands/uploadSessions (including the antivirus scenarios) all pass. So the mount id matches there too, and the uploads command works through the coordinator.

One thing that might be worth a line in the description: storage-users now needs a local upload directory for every driver. eos, eoshome, eosgrpc and owncloudsql have no root in their configuration, so with this change the storageprovider and the dataprovider refuse to start with "cannot determine the upload directory, set upload_directory". These drivers aren't maintained any more, so I don't think it needs a fix, but whoever still uses one of them would run into it on the update.

Brings in opencloud-eu/reva#1001, which merges
reva main into workpackages: the typed storage results, the fix for
synchronous uploads stuck in processing (moved to the upload
coordinator), the main-queue replica fix and the posix id cache fail
fast.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

@micbar micbar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We fix the codacy issue later, go jose v3 needs to be removed.

workpackages was merged into reva main with
opencloud-eu/reva#957, so we no longer need to
pin the workpackages branch. The code is the same as 7ff1d6fb1, only
the version changes.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Type:Maintenance E.g. technical debt, packaging, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants