[jwrosewell]

What was wrong

The resource cleanup feature of the pipeline specification says Element Data and Flow Elements that hold native (C/C++) resources must have a cleanup mechanism, that cleaning up Flow Data cleans up its Element Data, and that the Pipeline cleans up its Flow Elements. The required examples must show that every Flow Data is disposed of.

None of this existed in Python. There was no way to release the native results of an on-premise engine, its native engine, or a pipeline. Reference counting did not free them either, because a FlowData refers to itself through its evidence and a pipeline refers to itself through its elements, so they went only when the cycle collector happened to run. The native results hold the engine's data set, so a data set stayed in memory after its pipeline had gone, for as long as any of its results were still waiting for the collector. The automatic update and file watcher timers of the data file update service could not be stopped at all, and each held its pipeline and engine until the process ended.

The fix in the library

• FlowData.close() closes the element data that has a close method of its own and leaves the rest alone, so cached element data keeps working, as the specification requires. FlowData is a context manager.
• Pipeline.close() closes its flow elements, unless the pipeline is built with auto_close_elements set to False because its elements are shared with another pipeline, which is the option the specification asks for when elements can be in several pipelines. Pipeline is a context manager, and PipelineBuilder takes the setting as auto_close_elements.
• FlowElement.close() does nothing. Engine.close() stops the checks and updates of the engine's data file through the new DataFileUpdateService.unregister_data_file(), which cancels the pending timers so that nothing holds the engine.
• The device detection and IP intelligence on-premise results (SwigData) release their native results when closed, and reading a property afterwards raises an exception that says so. The on-premise engines release the native engine when closed, and processing afterwards raises an exception that says so.
• ShareUsage.close() shuts down its thread pool.
• PipelineBuilder.build_from_configuration() closes the elements it has already built when a later one fails to build, or when the pipeline cannot be made from them, and then raises the error that stopped the build. A failure to close one is logged.

Nothing changes for code that never calls close. No native code and no SWIG interface changed.

The bugs in the tests and examples

Every one of these created an on-premise pipeline or FlowData and released it on no path, including the exception paths. Line numbers are from rc before this change.

Tests, 7 files:

• device-detection/fiftyone_devicedetection_onpremise/tests/test_devicedetection.py: 32 tests built a pipeline that was never closed (the builds end at lines 63 to 640), and 23 of them also processed a FlowData that was never closed (lines 114 to 642). The test at line 73 also left an automatic update timer holding the engine for the rest of the run. The unused helper at line 654 created a FlowData for each of 20,000 User-Agents without closing any.
• device-detection/fiftyone_devicedetection_onpremise/tests/test_automatic_data_file_updates.py: 11 tests each built a pipeline and processed a FlowData, closing neither (lines 70 to 257), and line 328 opened the archive without closing it.
• device-detection/fiftyone_devicedetection_onpremise/tests/test_properties.py: the module's pipeline (line 43) and 3 FlowData (lines 52, 77 and 112).
• device-detection/fiftyone_devicedetection_onpremise/tests/test_swig.py: the module's pipeline (line 40).
• device-detection/fiftyone_devicedetection_examples/tests/test_onpremisegettingstartedweb.py: the pipeline of the web example the tests build (line 40).
• ip-intelligence/fiftyone_ipintelligence_onpremise/tests/test_ipintelligence.py: the pipeline built in setUpClass (line 48), the FlowData of the 5 tests that use the process helper (line 53), and the pipeline and FlowData at lines 211 and 215.
• ip-intelligence/fiftyone_ipintelligence_examples/tests/test_examples.py: the pipeline and FlowData at lines 67 and 73.

Examples, 22 files, each with a pipeline and FlowData that were never closed:

• Device detection, under device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/: gettingstarted_console.py (lines 69 and 86), match_metrics.py (90 and 94), metadata_console.py (78), offlineprocessing.py (100, and one FlowData per record at 143), performance.py (93, and one FlowData per User-Agent at 66), failuretomatch.py (34, 39 and 67), datafileupdate_console.py (two pipelines at 236 and 297, the second held for good by its automatic update and file watcher timers), shareusage.py (131 and 160), gettingstarted_web/app.py (98, and one FlowData per request at 115, 140 and 168) and useragentclienthints_web/app.py (72, and one FlowData per request at 87).
• device-detection/fiftyone_devicedetection/examples/gettingstarted.py (33, 41 and 67) and device-detection/fiftyone_devicedetection_examples/performance-tests/test_performance.py (74, and one FlowData per detection at 51).
• IP intelligence, under ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/: gettingstarted_console.py (92 and 107), compare_console.py (97 and 121), countries_console.py (72 and 81), metadata_console.py (48), mixed_console.py (141 and 148), offlineprocessing.py (66 and 87), performance_console.py (a pipeline per worker at 118 and 159, and one FlowData per record at 87), datafileupdate_console.py (176, a pipeline with automatic updates and a file watcher that was not even kept in a variable) and gettingstarted_web/app.py (63, and one FlowData per request at 92).
• ip-intelligence/fiftyone_ipintelligence/examples/gettingstarted.py (33, 41 and 67).

Tests now close what they create with addCleanup, or with tearDownClass and tearDownModule for pipelines that several tests share. Examples use with statements, with a comment saying what closing releases, so that they show the lifetime the specification describes. New tests check that closing a FlowData releases its results, that closing a pipeline releases its engine, that closing a pipeline cancels the engine's next automatic update, and how FlowData.close(), Pipeline.close() and unregister_data_file() behave. A test of build_from_configuration() builds a configuration whose second element fails, and checks that the first was closed and that the original error was raised.

The cloud tests and examples, and the tests of the pipeline packages, are unchanged, because their FlowData hold no native resources and so have nothing to release.

The tests now use the pipeline code in the change

The tox environments of the device detection and IP intelligence on-premise and examples packages installed those packages from the checkout, but took the pipeline packages they require from PyPI, so their tests ran against the last released pipeline code rather than the code in the same change, and could not see close. The device detection integration tests also run the web examples under the Selenium contract tests from a virtual environment of their own, which took the pipeline packages from PyPI too. Both now install the pipeline packages from the checkout. The device detection packages in that virtual environment still come from PyPI, as before.

The packages require the pipeline versions with close

The device detection and IP intelligence on-premise packages now require fiftyone_pipeline_core>=4.5.42 and fiftyone_pipeline_engines>=4.5.42, and the two examples packages require fiftyone_pipeline_core>=4.5.42, so updating one of them also brings pipeline packages that have close. The on-premise packages need the engines minimum as well, because the close of their engines calls Engine.close(), which stops the data file timers that would otherwise keep the engine alive. The examples packages are never published, so their requirement only records what they need.

Each package here is versioned and tagged on its own, as <package>/<version>, and the release plan in ci/packages.ps1 releases a package that changed since its last tag at the next patch. The newest tags are fiftyone_pipeline_core/4.5.41 and fiftyone_pipeline_engines/4.5.41, and this change touches both packages, so the release that includes it publishes both at 4.5.42. Run at the head of this pull request, Get-ReleasePlan gives 4.5.42 for both, each "changed since 4.5.41".

The number holds only if this change is in the next release of those two packages. rc already changes fiftyone_pipeline_core since 4.5.41, through its js_templates submodule, so if rc is released before this change is merged, 4.5.42 goes out without close and both minimums must move to 4.5.43. The publish prints its plan under "This publish releases:" before it builds anything.

A package's setup.py reads its version from a version.txt that the publish writes, so a package installed from a checkout is 0.0.0, which is below the new minimums. pip would then find no pipeline package that meets them, or replace the checkout's with a release from PyPI and test that instead. The new ci/write-package-versions.ps1 gives each published package the version this commit would release it at, or its last release when this commit would not release it, and the root ci/setup-environment.ps1 runs it before anything is installed. fiftyone_pipeline_did keeps the version.txt it commits. Git now ignores version.txt, and AGENTS.md says to run the script before running those tests from a checkout.

The guard

A conftest.py in each of the four test folders that run on-premise engines turns the cycle collector off while each test runs, then fails the test if any native results or native engine it created is still alive. A FlowData or pipeline that is not closed is only freed by the collector, so with the collector off, anything a test or an example leaves open is certain to be found, wherever it was created. Objects created before a test, such as a pipeline built in setUpClass or when a module is imported, are checked after the last test, and the session fails if any is still alive. A test that has already failed is not reported again.

No lint rule can do this. pylint's consider-using-with and ruff's SIM115 only know a fixed list of standard library calls, native memory is not visible to tracemalloc, and the process size is too noisy to measure it by.

How it was checked

rc was merged in at 61bcb09, after rc took the move of the data files to cxx/data. The one conflict was in the IP intelligence getting started web example, which keeps rc's message about finding the data file and still closes the pipeline with a with statement when the server stops.

The full build ran common-ci's pull request build of this pull request with dryrun, so nothing was merged, and the other builds ran it with pull request 0 and dryrun, on a throwaway branch per run that has since been deleted.

• Run 41881062372 is the full pull request build of 61bcb09, the head of this pull request, on all 15 Ubuntu, Windows and macOS configurations, Python 3.10 to 3.14, and passed. In its first attempt the macOS job with Python 3.12 stopped at GitHub's two hour limit in the share usage tests of fiftyone_pipeline_engines_fiftyone, which do not call close. The rerun of that job passed in 24 minutes, about the time the other macOS jobs took.
• Run 41870130689 built 2dcce71, the head before rc was merged in, and passed on all 10 Ubuntu and Windows configurations, Python 3.10 to 3.14, with 184 pipeline core tests passing on each.
• Run 41870254820 built this change with build_from_configuration() as it was before, on Ubuntu with Python 3.13. The new test failed with "AssertionError: 1 != 0", because nothing closed the first element, and the other 183 pipeline core tests passed.
• Run 41864800372 built 19a4dcc, which adds the minimum versions, and passed on all 10 Ubuntu and Windows configurations. In every job ci/write-package-versions.ps1 gave fiftyone_pipeline_core and fiftyone_pipeline_engines 4.5.42, and pip installed the checkout's pipeline packages at those versions.
• Run 41830192826 built 29f7655, the head before the minimum versions were added, and passed on all 10 Ubuntu and Windows configurations, Python 3.10 to 3.14. The macOS configurations were left out of it.
• Run 41827494827 built this change with one FlowData left open in test_basic_get_onpremise, and the guard failed that test at teardown with "1 native object(s) still alive (ResultsHashSwig)".
• Run 41829264165 built it with the web example's pipeline left open by its test, and the device detection examples session failed after its 24 tests passed with "After the last test, 1 native object(s) still alive (EngineHashSwig)". In the same run an IP intelligence example left with its FlowData open failed test_onpremise_gettingstarted_console with "4 native object(s) still alive (ResultsIpiSwig, ResultsIpiSwig, ResultsIpiSwig, ResultsIpiSwig)".
• Run 41821813095 and run 41827516054 failed before the change to the test environments, the first because the tox environments took fiftyone_pipeline_core 4.5.41 from PyPI, which has no close, and the second because the Selenium environment did.

Release note

• FlowData and Pipeline now have close() and work in a with statement. Closing a FlowData releases the native results its element data holds. Closing a pipeline closes its flow elements, unless it was built with auto_close_elements set to false.
• The on-premise Device Detection and IP Intelligence packages now require fiftyone_pipeline_core>=4.5.42 and fiftyone_pipeline_engines>=4.5.42, the versions this release publishes with close().