perf(sftp): make downloads substantially faster - #55
perf(sftp): make downloads substantially faster#55devin-ai-integration[bot] wants to merge 2 commits into
Conversation
…ism and cheaper buffering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
de sftp download is nog steeds niks sneller geworden |
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Twee dingen:
Daarom staat er nu een meettool in deze branch, Die print de volledige DCFS-download plus de ruwe CDN-snelheid met 1, 4, 8 en 16 parallelle verbindingen op dezelfde attachment. Daaruit volgt direct waar de grens ligt:
Stuur me die getallen (plus de bestandsgrootte en wat je nu via SFTP haalt), dan optimaliseer ik op basis van meting in plaats van gokwerk. |
het downloaden via sftp gaat nog steeds best traag.
het uploaden via sftp gaat aan een goede snelheid.
root@debian:/opt/dcfs# docker run --pull=always -it -p 1900:1900 -p 2121:2121 -p 2022:2022 -p 4445:4445 -v "/opt/dcfs:/home/dcfs/.dcfs" ghcr.io/vulcanosoftware/dcfs:latest webdav blijkt het niet echt meer te doen. 
dit is mijn internetsnelheid |
|
Rei.Fs.SimpleWebException: De externe server heeft een fout geretourneerd: (500) Interne serverfout. de webdav werkt nog steeds niet. |
1 participant
Summary
Downloads over SFTP were bottlenecked by per-request Discord REST round trips, a fixed and low amount of CDN parallelism, and O(n²) buffer copying in the SFTP read path. Four independent fixes:
1. Attachment URL cache (
discord_bot.py) — everydownload_file()didchannel.fetch_message(...)before touching the CDN, so an N-way parallel range download paid N REST round trips per part. URLs are now cached per message and concurrent misses for the same message share one lookup:TTL is 15 min, clamped to the
?ex=<hex unix>expiry embedded in Discord's signed URL (minus a 60s margin), so a cached URL is never served after it stops working. Cache is capped at 4096 entries.2. More CDN parallelism —
download_file_parallelused a hardcodedn = 4; it now scales one request per 2 MiB of range,4..8. Message parts fetched concurrently went from 3 to 6 (matching thedownload_max_concurrent_partsconfig default, which the repository default contradicted).3. SFTP read buffer (
handler.py) — the buffer was abytearraythat was re-sliced on every prune (self._read_buf = self._read_buf[discard:]), copying up to 4 MiB of remaining data repeatedly during a sequential read. It's now adeque[(abs_offset, chunk)]; a read copies only the requested range (and returns a slice of a single chunk with no copy when it fits), and pruning is apopleft.4. SSH transport — asyncssh defaults to ChaCha20-Poly1305; AES-GCM uses the CPU's AES instructions and benchmarks ~5x faster locally (~7.1 GB/s vs ~1.5 GB/s), so GCM/CTR are moved to the front of the offer list (all other algorithms keep their order, so no client loses compatibility). SSH compression is disabled — content is already compressed/encrypted, so it only burns CPU.
Also replaced the CDN
total=120stimeout withsock_read=60s: a total deadline aborts otherwise healthy long-running range downloads on slow links, while the read timeout still catches a hung connection.Testing
ruffclean;mypyclean except a pre-existing error indcfs/auth/bearer.py(untouched).encryption_algs/compression_algsand negotiates AES-GCM.Link to Devin session: https://app.devin.ai/sessions/bebf5d7564264685b19844fbef1e95eb
Open in Devin Desktop: https://app.devin.ai/desktop/session/bebf5d7564264685b19844fbef1e95eb?variant=devin
Requested by: @VulcanoSoftware