[Cruslan]

Summary

This pull request introduces critical stability fixes to librsx, cgcomp, and related GCM headers in PSL1GHT. These patches resolve severe hardware deadlocks, memory corruptions, pointer arithmetic overflows, and shader compilation failures that break continuous drawing and streaming workloads.

These issues directly surfaced during the porting and bare-metal deployment of PS3-Moonlight and PS3-Movian.

Important

Bare-Metal Hardware Verification:
Both the reproduced failure states and their corresponding fixes have been thoroughly tested and validated on physical PlayStation 3 retail hardware running HEN as well as RPCS3. These fixes are proven in sustained real-world production workloads.

---

Technical Problem Breakdown & Fix Details

1. librsx: Wrap User Callback in __get_opd32 to Prevent Fatal Crash on Buffer Wrap

• Commit: b28d153
• Affected File: ppu/librsx/init.c
• Root Cause:
  In the 64-bit PowerPC ELF ABI (ppc64), C function pointers do not point directly to executable machine code; they point to a 24-byte Official Procedure Descriptor (OPD) consisting of [function_entry_address, toc_pointer, environment_pointer].
  However, in ppu/include/rsx/rsx_function_macros.h, the low-level GCM assembly callback dispatcher (rsxContextCallback) is designed with 32-bit word instructions (lwz 0, 0(%3); lwz 2, 4(%3); mtctr 0; bctrl), expecting a 32-bit OPD layout (opd32).
  Passing a standard 64-bit function pointer directly to rsxSetContextCallback causes the dispatcher to read high 32-bit word halves as function entry and TOC pointers. The moment the RSX command buffer wraps around, the dispatcher executes a jump to corrupted memory, immediately producing a fatal PPU trap / panic.
• Fix:
  rsxSetContextCallback now wraps user callback handlers through __get_opd32(), guaranteeing that a properly formatted 32-bit OPD entry point and TOC descriptor are stored into context->callback.

---

2. librsx & cgcomp: Fix Pointer Arithmetic Overflow, Vertex Constant Offsets, and Texture Sampling Flag

• Commit: c4fb5c7
• Affected Files: ppu/librsx/init.c, ppu/librsx/commands_impl.h, ppu/librsx/buffer_impl.h, ppu/include/rsx/rsx_function_macros.h, tools/cgcomp/source/compilerfp.cpp, ppu/include/codec/vdec.h
• Root Cause & Fix Details:
  1. Pointer Arithmetic 4x Scaling Bug in rsxSetupContextData (init.c):
     The command buffer initialization previously performed pointer arithmetic by adding a byte length directly to a typed uint32_t* pointer. In standard C/C++, adding length to uint32_t* scales the offset by sizeof(uint32_t) == 4. For instance, a 1 MB buffer resulted in setting context->current 4 MB ahead, pointing far outside the allocated buffer boundaries and immediately causing heap/memory corruption.
     Fix: Cast the base address to uintptr_t / raw byte offsets before adding the byte length.
  2. cgcomp Texture Sampler Control Bit (allowTex / 0x8000):
     When compiling Cg fragment shaders that perform texture sampling (such as YUV-to-RGB color space conversion in video playback and streaming), the RSX microcode requires bit 15 (0x8000) to be set in the fragment program instruction word. Without this flag, texture coordinate interpolation and texture unit fetches fail or output black/garbage frames.
     Fix: Explicitly set allowTex (bit 15) in compilerfp.cpp whenever the fragment shader references texture samplers.
  3. Vertex Program Constants Multi-Vector Burst Offset (commands_impl.h):
     In SetVertexProgramConstants, the second burst packet was reusing the base pointer value instead of advancing by 16 floats (&value[16]). When uploading matrix blocks or multi-vector arrays exceeding 16 constants, the first chunk was repeatedly overwritten while subsequent vectors were dropped.
     Fix: Properly advance the source data pointer to &value[16] for the second transfer chunk.
  4. Packet Boundary Out-of-Bounds in SetTimeStamp (commands_impl.h):
     SetTimeStamp was writing to current[1] instead of [2], corrupting the command stream and misaligning subsequent GPU commands.
     Fix: Corrected packet indexing to respect packet payload sizing.
  5. Assembly Calling Convention Register Preservation (rsx_function_macros.h):
     The inline callback macro was clobbering caller-saved registers during 32-bit OPD dispatch, causing mysterious crashes upon returning from user callbacks.
     Fix: Guarded register preservation across the assembly invocation.
  6. Missing Video Decoder Definitions (vdec.h):
     Added missing standard VDEC codec constants (MPEG2, AVC/H.264, VC-1, DivX) required for hardware video decoding integration.

---

3. librsx: Bounded Spin Timeout and Hardware Memory Barrier in ResetCommandBuffer

• Commit: 20241e3
• Affected File: ppu/librsx/buffer_impl.h
• Root Cause Analysis:
  ResetCommandBuffer had an unbounded loop waiting for the GPU read pointer to reach the offset:

  while(ctrl->get != offset) usleep(30);

  Under sustained load (such as fast video frame rendering in Movian or 60 FPS streaming in Moonlight), if the RSX FIFO pipeline stalls or an interrupt timing difference occurs, ctrl->get may never match offset. This caused the PPU thread to enter an infinite spinlock, hard-freezing the entire console and requiring a physical power-cord disconnection.
• Fix:
  • Implemented an upper-bounded spin counter (spins < 100000, ~3 second timeout) to gracefully break out of deadlocks.
  • Added an explicit hardware memory barrier (__sync()) before updating ctrl->put = offset to ensure all previous cache flushes and memory transactions are committed to the RSX memory controller.
  • Synchronized host write pointers properly using gcmAddressToOffset(context->begin).

---

Real-World Project Impact

Project	Failure Mode Without Patches	Behavior With Patches
PS3-Moonlight	Instant crash on buffer wrap (corrupted OPD callback), missing texture rendering (Cg sampler bit 0x8000), and hard PS3 freezes on frame burst stalls.	Stable 60 FPS 1080p/720p sustained streaming for hours with zero hangs or dropped frames.
PS3-Movian (Movian Next)	Command buffer allocation overflows (4x scaling bug), corrupted OSD/subtitle vertex matrices, and random hard freezes during seeking/playback.	Flawless hardware-accelerated video decoding, correct subtitle rendering, and crash-free UI transitions.

[CapnRon]

Independent confirmation on RPCS3 with the current ps3toolchain (GCC 13.2 PPU).

Before this PR (PSL1GHT master f649a08fd): the stock samples/graphics/rsxtest, and an app derived from it, froze in RPCS3 about 1 second after rendering started. The RSX thread died with Dead FIFO commands queue state has been detected, preceded by repeated FIFO error: possible desync event (last cmd = 0x3e202104). The FIFO dump showed an old-style JUMP (0x20000000 | 0x1e202104) whose target is an address-like value rather than a command buffer offset.

Things that made no difference:

• RPCS3 FIFO accuracy, Driver Wake-Up Delay, FIFO reordering, Vulkan vs OpenGL
• -O0 vs -O2
• the command buffer size requested in rsxInit (RPCS3's effective buffer seems to be about 128 KB, which would explain a wrap at roughly 1 s)
• PSL1GHT at the last pre-June-2026 commit (eca3f990a): same-size binaries, same freeze, so this is not a recent PSL1GHT regression

After applying this PR on top of master: the stock rsxtest renders the sphere and donut and keeps animating (checked for 50+ s, frames changing, no RSX errors), and the derived bouncing-ball app also runs stably.

My reading, not verified by disassembly: this matches item 5. In the old rsxContextCallback asm, context and count are plain "r" operands that are not bound to r3/r4, so what the callback receives depends on the compiler's register allocation. GCC 13 does not leave them in r3/r4, which would produce a garbage callback argument and a garbage jump when the ring wraps. Binding them explicitly, as this PR does, fixes it.

Not tested on real hardware (my PS3 is offline right now).

Environment: ps3toolchain master, GCC 13.2.0 PPU (powerpc64-ps3-elf), binutils 2.42, Ubuntu 24.04, RPCS3 v0.0.43-20291 (Vulkan), firmware 4.76, booting the .self.

[Cruslan]

Those are real issues to had to be fixed. I am doubtful about my PR is still awaiting.

[TheMrIron2]

For future reference and maintainers: the OP was removed from the community for misconduct. The followup "independent confirmation" is not tested on a real PS3 by its own admission.

Both reports also fail the AI policy introduced in dd4e8c9. This PR predates the official policy, but the concern is self-evident; PRs with no evidence of human involvement are assumed to not be understood by their original poster. Therefore, we can't confirm if these bugs are real.

There may, however, be a genuine issue referenced in this PR and that is worth acknowledging. However, we cannot accept the precedent as it stands.

[Cruslan]

So admitting pr rejection from "arbitrary" reasons and personal hate?

[bucanero]

Independent confirmation on RPCS3 with the current ps3toolchain (GCC 13.2 PPU).

Before this PR (PSL1GHT master f649a08fd): the stock samples/graphics/rsxtest, and an app derived from it, froze in RPCS3 about 1 second after rendering started. The RSX thread died with Dead FIFO commands queue state has been detected, preceded by repeated FIFO error: possible desync event (last cmd = 0x3e202104). The FIFO dump showed an old-style JUMP (0x20000000 | 0x1e202104) whose target is an address-like value rather than a command buffer offset.

Please, there's no value or makes any sense to "confirm on RPCS3" , unless you can test and validate things on real hardware, there's no further evidence that it will work on a real PS3 system.

PSL1GHT's aim will always be to allow developers to build on real hardware. You can use tools and emulators, but unless things are thoroughly tested on a PS3, we won't take the code as valid.

[Cruslan]

Independent confirmation on RPCS3 with the current ps3toolchain (GCC 13.2 PPU).
Before this PR (PSL1GHT master f649a08fd): the stock samples/graphics/rsxtest, and an app derived from it, froze in RPCS3 about 1 second after rendering started. The RSX thread died with Dead FIFO commands queue state has been detected, preceded by repeated FIFO error: possible desync event (last cmd = 0x3e202104). The FIFO dump showed an old-style JUMP (0x20000000 | 0x1e202104) whose target is an address-like value rather than a command buffer offset.

Please, there's no value or makes any sense to "confirm on RPCS3" , unless you can test and validate things on real hardware, there's no further evidence that it will work on a real PS3 system.

PSL1GHT's aim will always be to allow developers to build on real hardware. You can use tools and emulators, but unless things are thoroughly tested on a PS3, we won't take the code as valid.

As i said they are tested on both of my Moonlight and Movian-Next projects. Both are tested on real hardware. :/ PSL1GHT's RSX code is broken, i found three of them and they needed to be fixed.

[TheMrIron2]

@Cruslan - the current assumption is that these are likely to be RPCS3 bugs, and the current PR is incompatible with our AI policy.
If you can produce visual evidence of these bugs occurring on hardware and a PR description that has been human edited, I'd be happy to open it for review. It's possible some or all of these are PSL1GHT defects, but we have currently no way of knowing concretely; we don't have any code that tests these bugs nor any hardware confirmation.

[bucanero]

Since other users reported the example samples/graphics/rsxtest is failing, I think the best way to address this and include the fix is to:

• create a RSX-related fix so samples/graphics/rsxtest works.
• test the sample on real hardware
• open a PR and explain the fix , add proof of the rsxtest sample working on hardware

That way we can verify and merge the changes to the repo.
I don't want to take other projects like Movian as reference for this fix because they're out of scope for the SDK repo. The codebase might already include a ton of other things that we can really review. If we use the rsxtest as validation, everything stays within PSL1GHT and we can quickly confirm if the change fixes the problem or not.

I'm open to review PRs , no arbitrary / personal issues here.