Add GroupTransform::ParseFromBuffer to parse a LUT from memory

[hmms]

Summary

Adds GroupTransform::ParseFromBuffer(const char * buffer, size_t bufferSize), a static factory that parses a LUT directly from an in-memory buffer rather than requiring a file on disk.

Closes #2076

Implementation

Reuses the existing LUT-reading pipeline rather than duplicating parser code. A small internal BufferConfigIOProxy (anonymous namespace in src/OpenColorIO/transforms/GroupTransform.cpp) serves the buffer as LUT data through Config::CreateRaw()->createEditableCopy()->setConfigIOProxy(...), then a FileTransform is resolved through Config::getProcessor(), and processor->createGroupTransform() returns the result.

Because OCIO's internal file caches are keyed by filename/hash strings and a buffer has no filename, the buffer's own content is hashed (CacheIDHash, XXH3-128) and used to form a synthetic filename ("ParseFromBuffer:" + hash, namespaced so it can never collide with a real file's resolved path) and the ConfigIOProxy::getFastLutFileHash value, so two different buffers can never collide in the cache, and re-parsing identical content is a cache hit. Null/empty buffers and unparseable content both throw a clear Exception.

Caching behavior worth flagging explicitly: like OCIO's existing file cache, entries created via ParseFromBuffer are never evicted, only released via ClearAllCaches(). This is fine for the typical case (a bounded set of files/buffers), but an application feeding many distinct, dynamically-generated buffers through this API (e.g. procedural LUTs per-frame) will grow the cache unboundedly unless it calls ClearAllCaches() itself. Happy to discuss whether this API should default to not caching, or take an opt-in/opt-out flag, if that's a concern, didn't want to guess at the right policy and expand scope without your input.

Python bindings

Added GroupTransform.ParseFromBuffer to the Python bindings (src/bindings/python/transforms/PyGroupTransform.cpp), with a corresponding test (tests/python/GroupTransformTest.py::test_parse_from_buffer). Verified with a real OCIO_BUILD_PYTHON=ON build (Python 3.14, pybind11 3.0.4): module imports, ParseFromBuffer is callable and correctly round-trips both text LUT content and produces the expected Lut1DTransform, raises on empty/malformed input, full tests/python suite (400 tests) passes with zero regressions.

Test plan

• OCIO_ADD_TEST(GroupTransform, parse_from_buffer) in tests/cpu/transforms/GroupTransform_tests.cpp covering:
  • a SPI1D fixture parsed from memory (length/value assertions)
  • a CLF fixture parsed as a second distinct buffer in the same process (guards against cache collisions between buffers, verified this test actually catches a collision by temporarily breaking the cache-key logic and confirming it fails)
  • re-parsing the same buffer twice (cache hit path)
  • null/zero-size buffer throwing
  • malformed buffer content throwing rather than crashing
• Full CPU test suite: 1161/1161 passed, 0 failed, 0 skipped, rebuilt clean with zero warnings in the changed files.
• GroupTransformTest.py::test_parse_from_buffer and the full tests/python suite (400 tests): all passing against a real Python build, not just compile-checked.

Notes for maintainers

• Only SPI1D and CLF formats were exercised in tests; other LUT formats should follow the same code path but are untested per-format.
• The method naming/signature (ParseFromBuffer(const char*, size_t) as a static factory on GroupTransform) is a reasonable judgment call but not the literal signature sketched in the issue, happy to adjust if you'd prefer something else.
• See the cache-growth note above, open to feedback on whether that needs an API-level answer in this PR or can stay as documented behavior.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: hmms / name: Muralidhar M Shenoy (da59b5c, e0599b5)

[doug-walker]

@hmms, thanks you did a good job here! This was a tricky one to take on as your first contribution to the project though. :)

I requested some changes below. Since you're a newcomer, I tried to propose some code to get you started in the right direction.

Apologies for the delayed review.

[doug-walker]

This must not return the same thing every time. It would trigger an infinite recursion if someone loads a CTF file containing a Reference element. Here's a proposed fix:

// ConfigIOProxy implementation that provides the contents of a single in-memory LUT file
// buffer, under one synthetic file name.  Any other file name is reported as missing.
class BufferConfigIOProxy : public ConfigIOProxy
{
public:
    BufferConfigIOProxy(const char * buffer,
                        size_t bufferSize,
                        const std::string & hash,
                        const std::string & syntheticFileName)
        : m_buffer(reinterpret_cast<const uint8_t *>(buffer),
                   reinterpret_cast<const uint8_t *>(buffer) + bufferSize)
        , m_hash(hash)
        , m_syntheticFileName(syntheticFileName)
    {
    }

    std::vector<uint8_t> getLutData(const char * filepath) const override
    {
        // Return an empty hash to avoid infinite recursion if m_buffer
        // contains a CTF with a Reference element.
        return isBufferPath(filepath) ? m_buffer : std::vector<uint8_t>();
    }

    std::string getConfigData() const override
    {
        // Unused, the config itself is not provided through the proxy.
        return "";
    }

    std::string getFastLutFileHash(const char * filepath) const override
    {
        // An empty hash makes FileExists false, to avoid infinite recursion if m_buffer
        // contains a CTF with a Reference element.
        return isBufferPath(filepath) ? m_hash : "";
    }

private:
    // The proxy is given the *resolved* path, i.e. the synthetic name joined with the
    // config's working directory.  ParseFromBuffer builds its config from a stream, which
    // leaves the working directory empty, so the resolved path is the synthetic name
    // verbatim.
    bool isBufferPath(const char * filepath) const
    {
        return filepath && m_syntheticFileName == filepath;
    }

    std::vector<uint8_t> m_buffer;
    std::string m_hash;
    std::string m_syntheticFileName;
};

[doug-walker]

This would be very inefficient without a file extension to hint which format parser to try first. Otherwise the buffer may need to be copied twice for each of the 20 formats. There should be a third parameter for the function (which is allowed to be empty) because callers will often know the format of the buffer. Here is proposed code:

GroupTransformRcPtr GroupTransform::ParseFromBuffer(const char * buffer,
                                                    size_t bufferSize,
                                                    const char * formatHint)
{
    if (!buffer || bufferSize == 0)
    {
        throw Exception("GroupTransform::ParseFromBuffer: buffer is null or empty.");
    }

    // Hash the buffer contents. The hash is used both to form the synthetic file name given
    // to the FileTransform and as the fast LUT file hash returned by the ConfigIOProxy, so
    // that the global file caches (which are keyed on these strings) never confuse the
    // contents of two different buffers.
    const std::string contentHash = CacheIDHash(buffer, bufferSize);

    // Prefix the hash to form a synthetic file name so that the global file cache entries
    // created here can never collide with those of a real file whose resolved path happens
    // to match the bare hash string.
    std::string syntheticFileName = "ParseFromBuffer:" + contentHash;

    if (formatHint && *formatHint)
    {
        // Append the hint as a file name extension, so that the readers registered for that
        // extension are tried first rather than trying every reader in turn.
        const char * ext = (*formatHint == '.') ? formatHint + 1 : formatHint;
        if (*ext)
        {
            syntheticFileName += ".";
            syntheticFileName += StringUtils::Lower(ext);
        }
    }

    ConfigIOProxyRcPtr ciop = std::make_shared<BufferConfigIOProxy>(buffer,
                                                                    bufferSize,
                                                                    contentHash,
                                                                    syntheticFileName);

    ConfigRcPtr config = Config::CreateRaw()->createEditableCopy();
    config->setConfigIOProxy(ciop);

    FileTransformRcPtr fileTransform = FileTransform::Create();
    fileTransform->setSrc(syntheticFileName.c_str());

    try
    {
        ConstProcessorRcPtr processor = config->getProcessor(fileTransform);
        return processor->createGroupTransform();
    }
    // Note that a CTF containing a Reference element will trigger this. The message should
    // contain the path that the Reference wants to load.
    catch (Exception & e)
    {
        std::ostringstream os;
        os << "GroupTransform::ParseFromBuffer: Error parsing LUT from buffer: " << e.what();
        throw Exception(os.str().c_str());
    }
}

[doug-walker]

Please add the following test that the error is about a missing file rather than triggering the recursion limit:

    // A buffer that refers to an external file must report that file as missing, since only
    // the contents of the buffer itself are available. In particular the reference must not
    // resolve back to the buffer, which would report a recursion instead.
    {
        const std::string content = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
                                    "<ProcessList id=\"ref\" version=\"1.7\">\n"
                                    "   <Reference inBitDepth=\"32f\" outBitDepth=\"32f\" "
                                    "path=\"lut1d_1.spi1d\"/>\n"
                                    "</ProcessList>\n";

        OCIO_CHECK_THROW_WHAT(OCIO::GroupTransform::ParseFromBuffer(content.c_str(),
                                                                    content.size(),
                                                                    "ctf"),
                              OCIO::Exception,
                              "could not be located");
    }

[doug-walker]

This should not take a string argument. For binary formats such as ICC profiles, I think that could trigger a UTF-8 conversion. Please try the following:

        .def_static("ParseFromBuffer", [](const py::buffer & buffer,
                                          const std::string & formatHint)
            {
                // Note that py::buffer accepts any bytes-like object (bytes, bytearray,
                // memoryview, ...) but not str, which would need to be encoded first and so
                // must not be silently transcoded here.
                py::buffer_info info = buffer.request();
                checkCContiguousArray(info);
                return GroupTransform::ParseFromBuffer(
                    static_cast<const char *>(info.ptr),
                    static_cast<size_t>(info.size * info.itemsize),
                    formatHint.c_str());
            },
             "buffer"_a,
             "formatHint"_a = "",
             DOC(GroupTransform, ParseFromBuffer, 2))