diff --git a/.gitignore b/.gitignore index 7c20e4e1..21cc57bb 100644 --- a/.gitignore +++ b/.gitignore @@ -14,3 +14,7 @@ __pycache__/ .venv/ build/ dist/ + +# The version each package's setup.py reads, written by the publish and by +# ci/write-package-versions.ps1. +version.txt diff --git a/.gitmodules b/.gitmodules index f0d05919..af600379 100644 --- a/.gitmodules +++ b/.gitmodules @@ -4,9 +4,6 @@ [submodule "pipeline/owid-python"] path = pipeline/owid-python url = https://github.com/SWAN-community/owid-python.git -[submodule "ip-intelligence/ip-intelligence-data"] - path = ip-intelligence/ip-intelligence-data - url = ../ip-intelligence-data [submodule "cxx"] path = cxx url = ../51degrees-cxx diff --git a/AGENTS.md b/AGENTS.md index 8dececdf..81f3a40b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,8 +1,256 @@ # Working in this repository -Notes on what is not obvious from the code. Keep them short and current. The +If you have been asked to get an example running, follow +[Run the examples](#run-the-examples). It gives every step in order, from a +fresh clone to the output. The sections after it are notes on what is not +obvious from the code, for changing it. Keep them short and current. The layout and how to get the code are in the [README](README.md). +## Run the examples + +Device Detection's on-premise `gettingstarted_console` is the quickest +example to show, because its data file is about 35 MB. IP Intelligence's +on-premise `gettingstarted_console` runs on the free Asn data file, which is +about 6 MB, and shows registration values once it is given the Lite file, +which is a download of about 230 MB. Run each command from the repository root +unless the step says otherwise. + +The on-premise engines are C and C++ code that is compiled when the package is +installed. PyPI has only source distributions of +`fiftyone-devicedetection-onpremise` and `fiftyone-ipintelligence-onpremise`, +so a compiler is needed whether the packages come from PyPI or from this +checkout. The steps below build them from this checkout, as the product +READMEs do. + +### Prerequisites + +- Git. +- Python 3.10 or later, as the README says. CI tests 3.10 to 3.14 on Ubuntu, + Windows and macOS (`ci/options.json`). +- A C11 and C++17 compiler. On Windows that is Visual Studio 2019 or later, or + the [C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/), + with Platform Toolset `v142` and Windows SDK `10.0.18362.0` or later. On + Linux the Device Detection README gives + `sudo apt-get install g++ make libatomic1`. +- For the data files, `curl`, `gzip` and `md5sum` (`md5` on macOS) on Linux + and macOS, where the examples download with the shell script, and Windows + PowerShell 5.1 or PowerShell 7 on Windows, where they use the PowerShell + script. +- Network access to GitHub, to PyPI for the packages the examples depend on, + and to `distributor.51degrees.com` for the data files. + +Clone the repository with its submodules. + +``` +git clone --recurse-submodules https://github.com/51Degrees/51degrees-python.git +cd 51degrees-python +``` + +In a clone made without `--recurse-submodules`, fetch the submodules. + +``` +git submodule update --init --recursive +``` + +### Data files + +The examples use the free data files, which need no licence key. They are not +stored in this repository. The `cxx` submodule holds the +[data](https://github.com/51Degrees/data) repository as `cxx/data`, which has a +link to each file, and a script downloads them into `cxx/data/device-detection` +and `cxx/data/ip-intelligence`. + +On Linux and macOS, run the shell script. + +``` +sh cxx/data/get-data-files.sh +``` + +On Windows, run the PowerShell script. Windows PowerShell can refuse to run a +script file under the policy Windows sets by default, so run it the way the +examples do. + +``` +powershell -NoProfile -ExecutionPolicy Bypass -File cxx/data/get-data-files.ps1 +``` + +Where PowerShell 7 is installed, `pwsh cxx/data/get-data-files.ps1` does the +same on any system. Either script downloads every free file, the Asn file +included, and keeps a file that is already there. To download one file, give +its name, as in `sh cxx/data/get-data-files.sh 51Degrees-LiteV41.ipi`. An +example run with no argument downloads the file it needs itself, when the file +is missing. + +#### Check the files + +- `cxx/data/device-detection/51Degrees-LiteV4.1.hash` is more than 10 MB. +- `cxx/data/ip-intelligence/51Degrees-IPIV4AsnIpiV41.ipi` is more than 1 MB. +- `cxx/data/ip-intelligence/51Degrees-LiteV41.ipi` is more than 100 MB. + +### Build + +Install into a virtual environment, as CI does for the examples +(`device-detection/ci/run-integration-tests.ps1`). On Linux and macOS, make and +activate one with these commands. + +``` +python3 -m venv .venv +. .venv/bin/activate +``` + +On Windows, in PowerShell, run `python -m venv .venv` and then +`.venv\Scripts\Activate.ps1`. + +Then install the Device Detection packages from source, with the commands the +Device Detection README gives, and the examples package. `setuptools` is +added because `python setup.py` needs it and Python 3.12 and later no longer +install it (`ip-intelligence/ci/setup-environment.ps1`). + +``` +cd device-detection +python -m pip install -r requirements.txt setuptools +python -m pip install -e fiftyone_devicedetection_shared/ +python -m pip install -e fiftyone_devicedetection_cloud/ +cd fiftyone_devicedetection_onpremise/ +python setup.py build_clib build_ext +cd ../ +python -m pip install -e fiftyone_devicedetection_onpremise/ +python -m pip install -e fiftyone_devicedetection/ +python -m pip install -e fiftyone_devicedetection_examples/ +cd .. +``` + +Then install the IP Intelligence packages, with the commands the IP +Intelligence README gives. + +``` +cd ip-intelligence +pip install -e fiftyone_ipintelligence_shared +pip install -e fiftyone_ipintelligence_cloud +pip install -e fiftyone_ipintelligence_onpremise +pip install -e fiftyone_ipintelligence +pip install -e fiftyone_ipintelligence_examples +cd .. +``` + +The packages these need from elsewhere, such as the Pipeline API packages, +come from PyPI. CI installs the ones in `pipeline/` over them +(`ci/setup-environment.ps1`), which running an example does not need. + +### Run + +Run the Device Detection example from the repository root with no argument. +It looks for `cxx/data/device-detection/51Degrees-LiteV4.1.hash` in the folder +it runs in and each folder above it, and runs the download script when the +file is missing, with Windows PowerShell on Windows and `sh` on Linux and +macOS (`find_file` in +`device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/example_utils.py`). + +``` +python -m fiftyone_devicedetection_examples.onpremise.gettingstarted_console +``` + +The IP Intelligence example with no argument looks for the Asn file in the +folder it runs in and in its `assets` and `cxx/data/ip-intelligence` folders, +then does the same in each of the seven folders above it, and downloads the +file into `cxx/data/ip-intelligence` when it is missing. To see registration +values, download the Lite file first, as "Data files" says, and give its path +as the first argument. + +``` +python -m fiftyone_ipintelligence_examples.onpremise.gettingstarted_console +python -m fiftyone_ipintelligence_examples.onpremise.gettingstarted_console cxx/data/ip-intelligence/51Degrees-LiteV41.ipi +``` + +Either example uses a path given as the first argument as it is, so a file +named that way is never downloaded. + +### Correct output + +When an example downloads its data file, it first prints +` is missing, so it is being downloaded by running` and, on the next +line, the command, then the script's `Downloading` and `Saved` lines. + +The pipeline logger writes through Python's `logging`, so its lines go to +standard error, each as `INFO:root:` or `WARNING:root:` followed by a +dictionary that holds the time, the level and the message. + +The Device Detection example logs `Using a 'Lite' data file created '' +from location ''`, a warning that it is using the Lite file, and possibly +a warning that the file is more than 30 days old. Both warnings are expected. +Then for each of five sets of evidence it prints `Input values:` and the +evidence, then `Results:` with `Mobile Device`, `Platform Name`, +`Platform Version`, `Browser Name` and `Browser Version`. The first set is a +User-Agent from a Samsung Android phone. + +The IP Intelligence example logs `Using a '' data file created +''`, then for each of four IP addresses, starting with `116.154.188.222`, +prints `Input values:` with `query.client-ip`, then `Results:` with a line for +each of 16 properties, starting with `RegisteredName`, `RegisteredOwner` and +`RegisteredCountry`. A property the data file lacks reads +`This property is not present in the data file.` With the Asn file that is +every registration and location property, as the examples' README says. The +Lite file has RegisteredName, RegisteredOwner and RegisteredCountry, so with it +those three lines are read from the file. + +The paid data files have many more properties, as the +[pricing page](https://51degrees.com/pricing?utm_source=github&utm_medium=docs&utm_campaign=51degrees-python&utm_content=agents.md&utm_term=correct-output) +describes. + +### Cloud examples + +The cloud examples need a resource key, which a person creates in the +configurator, so ask for one if none is set. A key with the properties the +Device Detection cloud example uses can be made +[here](https://configure.51degrees.com/1QWJwHxl?utm_source=github&utm_medium=docs&utm_campaign=51degrees-python&utm_content=agents.md&utm_term=cloud-examples), +which is the configuration the example's own message names, and a key for IP +Intelligence in the +[configurator](https://configure.51degrees.com/?utm_source=github&utm_medium=docs&utm_campaign=51degrees-python&utm_content=agents.md&utm_term=cloud-examples). + +The Device Detection cloud example takes the key as its first argument, or +from `_51DEGREES_RESOURCE_KEY`, or from the older name `resource_key`, which +the Device Detection README uses. The IP Intelligence cloud example takes it +as its first argument or from `resource_key`. + +``` +resource_key= python -m fiftyone_devicedetection_examples.cloud.gettingstarted_console +resource_key= python -m fiftyone_ipintelligence_examples.cloud.gettingstarted_console +``` + +In PowerShell, set `$env:resource_key = ''` first. Without a key, each +example logs an error that names the variable and the configurator. + +### Pitfalls + +- **A data file that is not there.** `cxx/data` holds a `.link` for each file + rather than the file. An example run with no argument downloads a missing + file, so it needs the network and the tools under "Prerequisites". Check + the file sizes as above. +- **Running outside the repository, or without the submodules.** The Device + Detection example then finds no `cxx/data`, searches for up to 5 seconds + and, when it finds nothing, logs `Failed to find a device detection data + file` with the advice to run `git submodule update --init --recursive`. Run + it from the repository root, in a clone made with Git, because a zip + download of the repository has no submodules. +- **No compiler, or no `setuptools`.** `python setup.py build_clib build_ext` + and the on-premise installs compile C and C++, and on Python 3.12 and later + `python setup.py` stops with `No module named 'setuptools'` until it is + installed. +- **Long paths on Windows.** CI sets `TEMP` to `C:\tmp` on Windows to keep + within the MSVC path length limit + (`device-detection/ci/run-integration-tests.ps1`), so build from a short + folder path. +- **Scripts blocked in Windows PowerShell.** Running `Activate.ps1`, like the + Device Detection README's `setup.ps1`, needs PowerShell to allow scripts, + which that README does with `Set-ExecutionPolicy -Scope CurrentUser Bypass`. +- **Pipenv.** The Device Detection README suggests `pipenv install`, but there + is no `Pipfile` in the repository. Use the virtual environment above. +- **What CI does not test.** CI runs the examples' tests, which call each + example's code rather than its `main`. The Device Detection tests use the + Lite file that `device-detection/ci/fetch-assets.ps1` links in from CI's + assets, and the IP Intelligence example tests use the Asn file and throw + their output away. + ## Products keep their own CI scripts Each product folder still has the `ci/` scripts it had as its own repository, @@ -37,12 +285,47 @@ installed the pipeline packages from PyPI. In CI the root `ci/setup-environment.ps1` installs them from `pipeline/` over those, so a pipeline change is tested with every product in the same run. +A package's `setup.py` reads its version from a `version.txt`, which only +`pipeline/fiftyone_pipeline_did` commits, so any other package installed from +a checkout is 0.0.0. That is below the minimum versions of the pipeline +packages that the on-premise and examples packages require, so +`ci/write-package-versions.ps1` gives each published package the version this +commit would release it at. The root `ci/setup-environment.ps1` runs it +first. Run `pwsh ci/write-package-versions.ps1` before running those +packages' tests from a checkout. + ## Submodules -The C/C++ engines and the data files are submodules, listed in the root -`.gitmodules` at their paths under each product folder. The IP Intelligence -on-premise package must compile its own copy of the common C library with -large data file support; the Device Detection one without it. +The C/C++ engines are the `cxx` submodule, listed in the root `.gitmodules`. +The data files come through cxx's own `data` submodule, at +`cxx/data/device-detection` and `cxx/data/ip-intelligence`, where each free +file is a `.link` that `cxx/data/get-data-files.ps1` or +`cxx/data/get-data-files.sh` downloads. Each product's `ci/fetch-assets.ps1` +gets the files its tests need, and the examples' `example_utils.py` runs the +script for a file that is missing. The IP Intelligence on-premise package must +compile its own copy of the common C library with large data file support; +the Device Detection one without it. + +A package's `ExtraPaths` in `ci/packages.json` cannot be a folder inside +`cxx`, because git reports any change there as a change to `cxx` itself. A +cxx update therefore releases no package, even when it brings new translations +in `cxx/data/ip-intelligence`. + +## Releases + +Pull requests go to `rc`, not `main`, and stay drafts until they have been +reviewed, because the nightly merges any pull request into `rc` that is not a +draft and has no review requested. When a release is wanted, one pull request +merges `rc` into `main`, and the "Only rc into main" check fails a pull request +into `main` from any other branch. + +## Releases + +Pull requests go to `rc`, not `main`, and stay drafts until they have been +reviewed, because the nightly merges any pull request into `rc` that is not a +draft and has no review requested. When a release is wanted, one pull request +merges `rc` into `main`, and the "Only rc into main" check fails a pull request +into `main` from any other branch. ## History diff --git a/README.md b/README.md index 3110be9c..04fb4d05 100644 --- a/README.md +++ b/README.md @@ -47,13 +47,36 @@ Clone with the submodules: git clone --recurse-submodules https://github.com/51Degrees/51degrees-python.git ``` +The data files the examples and tests use are not stored in a repository. The +[data](https://github.com/51Degrees/data) repository, which is the `cxx/data` +submodule, holds a `.link` file for each free one, where the file goes, and +two scripts that download them. From the root of this repository, run + +``` +sh cxx/data/get-data-files.sh +``` + +or, in PowerShell, + +``` +./cxx/data/get-data-files.ps1 +``` + +An example whose file is missing runs the script for that file itself. + +Git leaves the folder of a removed submodule on disk. A clone made before the +data files moved to `cxx/data` still has an `ip-intelligence/ip-intelligence-data` +folder, and keeps `cxx/device-detection-data` and `cxx/ip-intelligence-data` +once cxx removes them. Delete those folders, so that nothing finds an old copy +of a data file in them. + Each product's README says what else it needs, such as a C/C++ compiler for the on-premise engines and the data files. ## Supported Python versions The packages support Python 3.10 and later. The -[tested versions](https://51degrees.com/documentation/_info__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=tested-versions) +[tested versions](https://51degrees.com/documentation/_product_summaries__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=tested-versions) page lists the versions each release is tested against. ## Versions diff --git a/ci/packages.json b/ci/packages.json index 873df967..deb06ce9 100644 --- a/ci/packages.json +++ b/ci/packages.json @@ -140,7 +140,6 @@ "Name": "fiftyone_ipintelligence_onpremise", "Product": "ip-intelligence", "Path": "ip-intelligence/fiftyone_ipintelligence_onpremise", - "ExtraPaths": ["ip-intelligence/ip-intelligence-data"], "DependsOn": [ "fiftyone_ipintelligence_shared", "fiftyone_pipeline_engines_fiftyone" @@ -153,7 +152,6 @@ "Name": "fiftyone_ipintelligence_translation", "Product": "ip-intelligence", "Path": "ip-intelligence/fiftyone_ipintelligence_translation", - "ExtraPaths": ["ip-intelligence/ip-intelligence-data"], "DependsOn": [ "fiftyone_pipeline_core", "fiftyone_pipeline_translation" diff --git a/ci/setup-environment.ps1 b/ci/setup-environment.ps1 index 9994e2ef..932f5d81 100644 --- a/ci/setup-environment.ps1 +++ b/ci/setup-environment.ps1 @@ -12,6 +12,10 @@ if ($env:GITHUB_JOB -eq "PreBuild") { exit 0 } +# Before anything installs a package from this checkout, so that each one +# installs at the version this commit would release it at. +& (Join-Path $PSScriptRoot 'write-package-versions.ps1') -RepoName $RepoName + Invoke-ProductStep 'setup-environment' $RepoName $PSBoundParameters # The device detection, IP intelligence and location scripts install the diff --git a/ci/write-package-versions.ps1 b/ci/write-package-versions.ps1 new file mode 100644 index 00000000..44b23f25 --- /dev/null +++ b/ci/write-package-versions.ps1 @@ -0,0 +1,28 @@ +param ( + [string]$RepoName = '.' +) +$ErrorActionPreference = "Stop" +. (Join-Path $PSScriptRoot 'packages.ps1') + +# Writes the version.txt that each published package's setup.py reads. Without +# one a package installed from this checkout is 0.0.0, which is below the +# versions the packages here require of each other, so pip refuses it, or +# replaces it with a release from PyPI. Each package gets the version this +# commit would release it at, as the publish gives it when it builds the +# package, or its last release when this commit would not release it. +$plan = @(Get-ReleasePlan $RepoName) +$published = @(Get-Packages $RepoName | Where-Object { $_.Publish -eq $true }) +foreach ($package in $published) { + # A version.txt committed with the package is left as it is. + if (git -C $RepoName ls-files -- "$($package.Path)/version.txt") { + continue + } + $release = $plan | Where-Object { $_.Name -eq $package.Name } + if ($null -ne $release) { + $version = $release.Version + } else { + $version = "$((Get-ReleasedVersions $RepoName $package.Name)[-1])" + } + $version | Out-File (Join-Path $RepoName $package.Path 'version.txt') + Write-Output "$($package.Name) $version" +} diff --git a/cxx b/cxx index 1bec7b57..e866f6d3 160000 --- a/cxx +++ b/cxx @@ -1 +1 @@ -Subproject commit 1bec7b570575e2e180b52435599bef9b03e35ad8 +Subproject commit e866f6d3cea936587790f87967eff7863778b30e diff --git a/device-detection/ci/README.md b/device-detection/ci/README.md index 0db0f74d..76a328d6 100644 --- a/device-detection/ci/README.md +++ b/device-detection/ci/README.md @@ -4,7 +4,7 @@ This API complies with the `common-ci` approach. The following secrets are required: * `ACCESS_TOKEN` - GitHub [access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#about-personal-access-tokens) for cloning repos, creating PRs, etc. * Example: `github_pat_l0ng_r4nd0m_s7r1ng` -* `SUPER_RESOURCE_KEY` - [resource key](https://51degrees.com/documentation/_info__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=ci-readme.md&utm_term=api-specific-ci-cd-approach) for integration tests +* `SUPER_RESOURCE_KEY` - [resource key](https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=ci-readme.md&utm_term=api-specific-ci-cd-approach) for integration tests * Example: `R4nd0m-S7r1ng` * `DEVICE_DETECTION_KEY` - [license key](https://51degrees.com/pricing?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=ci-readme.md&utm_term=api-specific-ci-cd-approach) for downloading assets (TAC hashes file and TAC CSV data file) * Example: `V3RYL0NGR4ND0M57R1NG` diff --git a/device-detection/ci/fetch-assets.ps1 b/device-detection/ci/fetch-assets.ps1 index c551f8f9..45cb2875 100644 --- a/device-detection/ci/fetch-assets.ps1 +++ b/device-detection/ci/fetch-assets.ps1 @@ -11,7 +11,8 @@ if ($env:GITHUB_JOB -eq "PreBuild") { exit 0 } -$deviceDetectionData = "$PSScriptRoot/../../cxx/device-detection-data" +$data = "$PSScriptRoot/../../cxx/data" +$deviceDetectionData = "$data/device-detection" $assets = '51Degrees-LiteV4.1.hash', '20000 Evidence Records.yml', '20000 User Agents.csv' if ($DeviceDetection) { @@ -29,3 +30,12 @@ foreach ($asset in $assets) { default {New-Item -ItemType SymbolicLink -Force -Target "$PWD/assets/$_" -Path "$deviceDetectionData/$_"} } } + +# Downloads any other file the data repository links to in the folder. The +# script keeps a file that is already there, so the assets linked in above +# are not downloaded again. +$global:LASTEXITCODE = 0 +& "$data/get-data-files.ps1" device-detection +if ($LASTEXITCODE -ne 0) { + throw "Not every data file in $deviceDetectionData was downloaded" +} diff --git a/device-detection/ci/run-integration-tests.ps1 b/device-detection/ci/run-integration-tests.ps1 index 9aa2cf2f..f1483528 100644 --- a/device-detection/ci/run-integration-tests.ps1 +++ b/device-detection/ci/run-integration-tests.ps1 @@ -67,7 +67,12 @@ Push-Location $examplesDir try { python3 -m venv .virtualenv if ($LASTEXITCODE -ne 0) { throw "failed to create the examples virtual environment" } - & $py -m pip install -e . + # The pipeline packages come from this checkout rather than a package + # index, so the examples run with the pipeline code they are released with. + $pipeline = 'fiftyone_pipeline_core', 'fiftyone_pipeline_engines', + 'fiftyone_pipeline_engines_fiftyone', 'fiftyone_pipeline_cloudrequestengine' + $local = $pipeline | ForEach-Object { '-e'; "../../pipeline/$_" } + & $py -m pip install -e . @local if ($LASTEXITCODE -ne 0) { throw "failed to install the examples package" } } finally { Pop-Location } diff --git a/device-detection/fiftyone_devicedetection/examples/gettingstarted.py b/device-detection/fiftyone_devicedetection/examples/gettingstarted.py index 994d736f..b439c3d3 100644 --- a/device-detection/fiftyone_devicedetection/examples/gettingstarted.py +++ b/device-detection/fiftyone_devicedetection/examples/gettingstarted.py @@ -24,7 +24,7 @@ # First create the device detection pipeline with the desired settings. -data_file = "../../cxx/device-detection-data/51Degrees-LiteV4.1.hash" +data_file = "../../cxx/data/device-detection/51Degrees-LiteV4.1.hash" resource_key = "!!YOUR_RESOURCE_KEY!!" # Uncommment the line below 'On-Premise' and comment the line below 'Cloud' to @@ -35,46 +35,49 @@ # Cloud #pipeline = DeviceDetectionPipelineBuilder({"resource_key": resource_key}).build() -# We create a FlowData object from the pipeline -# this is used to add evidence to and then process +# The pipeline is closed at the end of the with block, which releases the +# native data it holds. +with pipeline: + # We create a FlowData object from the pipeline + # this is used to add evidence to and then process. It is closed at the + # end of its with block, which releases the results of the detection. + with pipeline.create_flowdata() as flowdata1: -flowdata1 = pipeline.create_flowdata() + # Here we add a User-Agent of a desktop as evidence -# Here we add a User-Agent of a desktop as evidence + desktop_ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36" -desktop_ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36" + flowdata1.evidence.add("header.user-agent", desktop_ua) -flowdata1.evidence.add("header.user-agent", desktop_ua) + # Now we process the FlowData -# Now we process the FlowData + flowdata1.process() -flowdata1.process() + # To check whether the User-Agent is a mobile device we look at the ismobile property + # inside the Device Detection Engine -# To check whether the User-Agent is a mobile device we look at the ismobile property -# inside the Device Detection Engine + # first we check if this has a meaningful result -# first we check if this has a meaningful result + print("Is User-Agent " + desktop_ua + " a mobile? ") + if flowdata1.device.ismobile.has_value(): + print(flowdata1.device.ismobile.value()) + else: + # Output why the value isn't meaningful + print(flowdata1.device.ismobile.no_value_message()) -print("Is User-Agent " + desktop_ua + " a mobile? ") -if flowdata1.device.ismobile.has_value(): - print(flowdata1.device.ismobile.value()) -else: - # Output why the value isn't meaningful - print(flowdata1.device.ismobile.no_value_message()) + # Now we do the same with a new User-Agent, this time from an iphone -# Now we do the same with a new User-Agent, this time from an iphone + with pipeline.create_flowdata() as flowdata2: -flowdata2 = pipeline.create_flowdata() + iphone_ua = "Mozilla/5.0 (iPhone; CPU iPhone OS 11_2 like Mac OS X) AppleWebKit/604.4.7 (KHTML, like Gecko) Mobile/15C114" -iphone_ua = "Mozilla/5.0 (iPhone; CPU iPhone OS 11_2 like Mac OS X) AppleWebKit/604.4.7 (KHTML, like Gecko) Mobile/15C114" + flowdata2.evidence.add("header.user-agent", iphone_ua) -flowdata2.evidence.add("header.user-agent", iphone_ua) + flowdata2.process() -flowdata2.process() - -print("Is User-Agent " + iphone_ua + " a mobile device?: ") -if flowdata2.device.ismobile.has_value(): - print(flowdata2.device.ismobile.value()) -else: - # Output why the value isn't meaningful - print(flowdata2.device.ismobile.no_value_message()) + print("Is User-Agent " + iphone_ua + " a mobile device?: ") + if flowdata2.device.ismobile.has_value(): + print(flowdata2.device.ismobile.value()) + else: + # Output why the value isn't meaningful + print(flowdata2.device.ismobile.no_value_message()) diff --git a/device-detection/fiftyone_devicedetection/readme.md b/device-detection/fiftyone_devicedetection/readme.md index 7cd26f83..72ffdfb4 100644 --- a/device-detection/fiftyone_devicedetection/readme.md +++ b/device-detection/fiftyone_devicedetection/readme.md @@ -1,8 +1,8 @@ # 51Degrees Device Detection Engines - Device Detection -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection-readme.md&utm_term=51degrees-device-detection-engines-device-detection "THE Fastest and Most Accurate Device Detection") **v4 Device Detection Python** +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection&utm_term=51degrees-device-detection-engines-device-detection "THE Fastest and Most Accurate Device Detection") **v4 Device Detection Python** -[Developer Documentation](https://51degrees.com/device-detection-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection-readme.md&utm_term=51degrees-device-detection-engines-device-detection "Developer Documentation") | [Available Properties](https://51degrees.com/resources/property-dictionary?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection-readme.md&utm_term=51degrees-device-detection-engines-device-detection "View all available properties and values") +[Developer Documentation](https://51degrees.com/device-detection-python/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection&utm_term=51degrees-device-detection-engines-device-detection "Developer Documentation") | [Available Properties](https://51degrees.com/resources/property-dictionary?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection&utm_term=51degrees-device-detection-engines-device-detection "View all available properties and values") ## Introduction @@ -25,3 +25,11 @@ The Pipeline is a generic web request intelligence and data processing solution To run the tests use: `python -m unittest discover -s tests -p test*.py -b` + +## Find out more + +- [Device Detection](https://51degrees.com/device-detection?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection&utm_term=find-out-more-device-detection) +- [Device Detection documentation](https://51degrees.com/documentation/_device_detection__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection&utm_term=find-out-more-documentation) +- [Getting started](https://51degrees.com/documentation/_device_detection__quickstart.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection&utm_term=find-out-more-getting-started) +- [Pricing](https://51degrees.com/pricing?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection&utm_term=find-out-more-pricing) +- https://github.com/51Degrees/51degrees-python diff --git a/device-detection/fiftyone_devicedetection/setup.py b/device-detection/fiftyone_devicedetection/setup.py index 99b1c73d..e3bd7af0 100644 --- a/device-detection/fiftyone_devicedetection/setup.py +++ b/device-detection/fiftyone_devicedetection/setup.py @@ -41,7 +41,11 @@ def read(file_name): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/device-detection?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_device_detection__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("Device Detection and User-Agent parser for Python that detects the device, browser, operating system, and crawlers from the User-Agent and User-Agent Client Hints, on-premise with a data file or in the cloud. An alternative to UAParser.js, DeviceAtlas, and WURFL."), keywords=[ "51degrees", diff --git a/device-detection/fiftyone_devicedetection_cloud/readme.md b/device-detection/fiftyone_devicedetection_cloud/readme.md index dead810b..10fe2fe4 100644 --- a/device-detection/fiftyone_devicedetection_cloud/readme.md +++ b/device-detection/fiftyone_devicedetection_cloud/readme.md @@ -1,8 +1,8 @@ # 51Degrees Device Detection Engines - Cloud -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud-readme.md&utm_term=51degrees-device-detection-engines-cloud "THE Fastest and Most Accurate Device Detection") **v4 Device Detection Python** +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud&utm_term=51degrees-device-detection-engines-cloud "THE Fastest and Most Accurate Device Detection") **v4 Device Detection Python** -[Developer Documentation](https://51degrees.com/device-detection-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud-readme.md&utm_term=51degrees-device-detection-engines-cloud "Developer Documentation") | [Available Properties](https://51degrees.com/resources/property-dictionary?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud-readme.md&utm_term=51degrees-device-detection-engines-cloud "View all available properties and values") +[Developer Documentation](https://51degrees.com/device-detection-python/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud&utm_term=51degrees-device-detection-engines-cloud "Developer Documentation") | [Available Properties](https://51degrees.com/resources/property-dictionary?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud&utm_term=51degrees-device-detection-engines-cloud "View all available properties and values") ## Introduction @@ -12,7 +12,7 @@ You can confirm this is working with the following micro-example. -* Create a resource key for free with the 51Degrees [configurator](https://configure.51degrees.com/np5M4nlF?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud-readme.md&utm_term=from-pypi). This defines the properties you want to access. +* Create a resource key for free with the 51Degrees [configurator](https://configure.51degrees.com/np5M4nlF?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud&utm_term=from-pypi). This defines the properties you want to access. * On the 'implement' page of the configurator, copy the resource key and replace YOUR_RESOURCE_KEY in the example below. Save this as exampledd.py * Run the example with `python exampledd.py` * Feel free to try different user-agents and property values. @@ -26,7 +26,7 @@ fd.process() print(fd.device.ismobile.value()) ``` -For more in-depth examples, check out the [examples](https://51degrees.com/device-detection-python/examples.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud-readme.md&utm_term=from-pypi) page in the documentation. +For more in-depth examples, check out the [examples](https://51degrees.com/device-detection-python/examples.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud&utm_term=from-pypi) page in the documentation. ### From GitHub @@ -40,4 +40,10 @@ To run the tests use: `python -m unittest discover -s tests -p test*.py -b` +## Find out more +- [Device Detection](https://51degrees.com/device-detection?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud&utm_term=find-out-more-device-detection) +- [Device Detection documentation](https://51degrees.com/documentation/_device_detection__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud&utm_term=find-out-more-documentation) +- [Getting started](https://51degrees.com/documentation/_device_detection__quickstart.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud&utm_term=find-out-more-getting-started) +- [Pricing](https://51degrees.com/pricing?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud&utm_term=find-out-more-pricing) +- https://github.com/51Degrees/51degrees-python diff --git a/device-detection/fiftyone_devicedetection_cloud/setup.py b/device-detection/fiftyone_devicedetection_cloud/setup.py index 0d721623..7318548d 100644 --- a/device-detection/fiftyone_devicedetection_cloud/setup.py +++ b/device-detection/fiftyone_devicedetection_cloud/setup.py @@ -41,7 +41,11 @@ def read(file_name): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/device-detection?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_device_detection__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_cloud&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("Device Detection and User-Agent parser for Python that detects the device, browser, operating system, and crawlers from the User-Agent and User-Agent Client Hints using the 51Degrees cloud service. An alternative to UAParser.js, DeviceAtlas, and WURFL."), keywords=[ "51degrees", diff --git a/device-detection/fiftyone_devicedetection_examples/performance-tests/test_performance.py b/device-detection/fiftyone_devicedetection_examples/performance-tests/test_performance.py index b7064b14..a8de1b36 100644 --- a/device-detection/fiftyone_devicedetection_examples/performance-tests/test_performance.py +++ b/device-detection/fiftyone_devicedetection_examples/performance-tests/test_performance.py @@ -48,16 +48,18 @@ def detect(app, pipeline, user_agents): # in flowdata.process(). This mock-request doesn't set it, so we set it manually request.remote_addr = "127.0.0.1" - flowdata = pipeline.create_flowdata() + # The FlowData is closed when the function returns, which releases + # the results of the detection straight away. + with pipeline.create_flowdata() as flowdata: - # Add any information from the request (headers, cookies and additional - # client side provided information) - flowdata.evidence.add_from_dict(webevidence(request)) + # Add any information from the request (headers, cookies and additional + # client side provided information) + flowdata.evidence.add_from_dict(webevidence(request)) - # Process the flowdata - flowdata.process() + # Process the flowdata + flowdata.process() - return getValueHelper(flowdata, "device", "ismobile") + return getValueHelper(flowdata, "device", "ismobile") def benchmark(data_file, user_agents_file, output=False): app = Flask(__name__) @@ -73,13 +75,16 @@ def benchmark(data_file, user_agents_file, output=False): settings = { "add_javascript_builder": False }, ).build() - with open(user_agents_file) as f: - user_agents_list = [ua.rstrip() for ua in f] - user_agents = iter(user_agents_list) + # The pipeline is closed once the detections have been timed, which + # releases the native data it holds. + with pipeline: + with open(user_agents_file) as f: + user_agents_list = [ua.rstrip() for ua in f] + user_agents = iter(user_agents_list) - print(f"Benchmarking with {len(user_agents_list)} user agents", file=sys.stderr) - environment = {"detect": detect, "app": app, "pipeline": pipeline, "user_agents": user_agents} - time = timeit("detect(app, pipeline, user_agents)", globals=environment, number=len(user_agents_list)) + print(f"Benchmarking with {len(user_agents_list)} user agents", file=sys.stderr) + environment = {"detect": detect, "app": app, "pipeline": pipeline, "user_agents": user_agents} + time = timeit("detect(app, pipeline, user_agents)", globals=environment, number=len(user_agents_list)) if output: json.dump({ diff --git a/device-detection/fiftyone_devicedetection_examples/setup.py b/device-detection/fiftyone_devicedetection_examples/setup.py index 7f6dd0e9..e5ab4dbf 100644 --- a/device-detection/fiftyone_devicedetection_examples/setup.py +++ b/device-detection/fiftyone_devicedetection_examples/setup.py @@ -41,7 +41,11 @@ def read(file_name): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/device-detection?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_device_detection__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("The 51Degrees Pipeline API is a generic web request intelligence and data processing solution with the ability to add a range of 51Degrees and/or custom plug ins (Engines). " "This project contains examples of using 51Degrees Device Detection engines with the Pipeline API"), long_description=read("readme.md"), @@ -61,7 +65,9 @@ def read(file_name): # named here so that is visible to anyone reading them. # fiftyone_devicedetection is still required because the on-premise # examples in this same package use the on-premise engine. - install_requires=["fiftyone_devicedetection_cloud", "fiftyone_devicedetection", "flask", "flask-unittest", "json5", "ruamel.yaml"], + # fiftyone_pipeline_core 4.5.42 is the first version with close(), which + # the examples use. + install_requires=["fiftyone_devicedetection_cloud", "fiftyone_devicedetection", "fiftyone_pipeline_core>=4.5.42", "flask", "flask-unittest", "json5", "ruamel.yaml"], license="EUPL-1.2", classifiers=[ "Development Status :: 5 - Production/Stable", diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/configurator_console.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/configurator_console.py index b985afe9..9e6d0b5f 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/configurator_console.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/configurator_console.py @@ -31,7 +31,7 @@ # It shows how to call the cloud with the newly created key and how to access the values # of the selected properties. # -# See [Getting Started](https://51degrees.com/documentation/_examples__device_detection__getting_started__console__cloud.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-configurator_console.py&utm_term=top) +# See [Getting Started](https://51degrees.com/documentation/_device_detection__examples__getting_started__console__cloud.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-configurator_console.py&utm_term=top) # for a fuller example. # # Required PyPi Dependencies: @@ -96,7 +96,7 @@ def main(argv): f"'{ExampleUtils.RESOURCE_KEY_ENV_VAR}'. The 51Degrees " + "cloud service is accessed using a 'ResourceKey'. " + "For more detail see " + - "https://51degrees.com/documentation/_info__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-configurator_console.py&utm_term=resource-key-required. " + + "https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-configurator_console.py&utm_term=resource-key-required. " + "A resource key with the properties required by this " + "example can be created for free at " + "https://configure.51degrees.com/1QWJwHxl?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-configurator_console.py&utm_term=resource-key-required. " + diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/gettingstarted_console.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/gettingstarted_console.py index caa7c7b8..3a92915b 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/gettingstarted_console.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/gettingstarted_console.py @@ -50,7 +50,7 @@ def run(self, config, logger, output): # In this example, we use the PipelineBuilder and configure it from a file. # For more information about builders in general see the documentation at - # https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-gettingstarted_console.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-gettingstarted_console.py&utm_term=run # Create the pipeline using the service provider and the configured options. pipeline = PipelineBuilder().add_logger(logger).build_from_configuration(config) @@ -174,7 +174,7 @@ def main(argv): f"'{ExampleUtils.RESOURCE_KEY_ENV_VAR}'. The 51Degrees cloud " + "service is accessed using a 'ResourceKey'. For more information " + "see " + - "https://51degrees.com/documentation/_info__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-gettingstarted_console.py&utm_term=resource-key-required. " + + "https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-gettingstarted_console.py&utm_term=resource-key-required. " + "A resource key with the properties required by this example can be " + "created for free at https://configure.51degrees.com/1QWJwHxl?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-gettingstarted_console.py&utm_term=resource-key-required. " + "Once complete, populate the config file or environment variable " + diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/gettingstarted_web/app.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/gettingstarted_web/app.py index ba86890b..8260065b 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/gettingstarted_web/app.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/gettingstarted_web/app.py @@ -58,7 +58,7 @@ # ``` # # Results can also be accessed in client-side code by using the `fod` object. See the -# [JavaScriptBuilderElementBuilder](https://51degrees.com/pipeline-python/4.3/classpipeline-python_1_1fiftyone__pipeline__core_1_1fiftyone__pipeline__core_1_1javascriptbuilde778a9036818b19ab55d981a40be4a4d7.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-gettingstarted_web-app.py&utm_term=top) +# [JavascriptBuilderElement](https://51degrees.com/documentation/apis/pipeline-python/classfiftyone__pipeline__core_1_1javascriptbuilder_1_1_javascript_builder_element.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-gettingstarted_web-app.py&utm_term=top) # for details on available settings such as changing the `fod` name. # ```{js} # window.onload = function () { @@ -230,7 +230,7 @@ def main(argv): f"'{ExampleUtils.RESOURCE_KEY_ENV_VAR}'. The 51Degrees " + "cloud service is accessed using a 'ResourceKey'. " + "For more detail see " + - "https://51degrees.com/documentation/_info__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-gettingstarted_web-app.py&utm_term=resource-key-required. " + + "https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-gettingstarted_web-app.py&utm_term=resource-key-required. " + "A resource key with the properties required by this " + "example can be created for free at " + "https://configure.51degrees.com/g3gMZdPY?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-gettingstarted_web-app.py&utm_term=resource-key-required. " + diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/metadata_console.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/metadata_console.py index 74b0df56..d8b06656 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/metadata_console.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/metadata_console.py @@ -120,7 +120,7 @@ def main(argv): f"'{ExampleUtils.RESOURCE_KEY_ENV_VAR}'. The 51Degrees " + "cloud service is accessed using a 'ResourceKey'. " + "For more detail see " + - "https://51degrees.com/documentation/_info__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-metadata_console.py&utm_term=resource-key-required. " + + "https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-metadata_console.py&utm_term=resource-key-required. " + "A resource key with the properties required by this " + "example can be created for free at " + "https://configure.51degrees.com/1QWJwHxl?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-metadata_console.py&utm_term=resource-key-required. " + diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/nativemodellookup_console.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/nativemodellookup_console.py index 4d11f6f5..98b7212e 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/nativemodellookup_console.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/nativemodellookup_console.py @@ -61,7 +61,7 @@ def run(self, resource_key, logger, output, cloudEndPoint = ""): # This example creates the pipeline and engines in code. For a demonstration # of how to do this using a configuration file instead, see the TacLookup example. # For more information about builders in general see the documentation at - # https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-nativemodellookup_console.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-nativemodellookup_console.py&utm_term=run cloudRequestEngineSettings = { "resource_key": resource_key } # If a cloud endpoint has been provided then set the @@ -128,7 +128,7 @@ def main(argv): f"environment variable '{ExampleUtils.RESOURCE_KEY_ENV_VAR}'. " + "The 51Degrees cloud service is accessed using a 'ResourceKey'. " + "For more information " + - "see https://51degrees.com/documentation/_info__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-nativemodellookup_console.py&utm_term=resource-key-required. " + + "see https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-nativemodellookup_console.py&utm_term=resource-key-required. " + "Native model lookup is not available as a free service. This means that " + "you will first need a license key, which can be purchased from our " + "pricing page: https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-nativemodellookup_console.py&utm_term=resource-key-required. Once this is done, a resource " + diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/taclookup_console.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/taclookup_console.py index 4563bd26..63355916 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/taclookup_console.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/taclookup_console.py @@ -62,7 +62,7 @@ def run(self, config, logger, output): # For a demonstration of how to do this in code instead, see the # NativeModelLookup example. # For more information about builders in general see the documentation at - # https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-taclookup_console.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-taclookup_console.py&utm_term=run # Create the pipeline using the service provider and the configured options. pipeline = PipelineBuilder().add_logger(logger).build_from_configuration(config) @@ -129,7 +129,7 @@ def main(argv): f"the environment variable '{ExampleUtils.RESOURCE_KEY_ENV_VAR}'. " + "The 51Degrees cloud service is accessed using a 'ResourceKey'. " + "For more information see " + - "https://51degrees.com/documentation/_info__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-taclookup_console.py&utm_term=resource-key-required. " + + "https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-taclookup_console.py&utm_term=resource-key-required. " + "TAC lookup is not available as a free service. This means " + "that you will first need a license key, which can be purchased " + "from our pricing page: https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-taclookup_console.py&utm_term=resource-key-required. Once this is " + diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/useragentclienthints_web/app.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/useragentclienthints_web/app.py index 730153df..728f0594 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/useragentclienthints_web/app.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/cloud/useragentclienthints_web/app.py @@ -136,7 +136,7 @@ def main(argv): f"'{ExampleUtils.RESOURCE_KEY_ENV_VAR}'. The 51Degrees " + "cloud service is accessed using a 'ResourceKey'. " + "For more detail see " + - "https://51degrees.com/documentation/_info__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-useragentclienthints_web-app.py&utm_term=resource-key-required. " + + "https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-useragentclienthints_web-app.py&utm_term=resource-key-required. " + "A resource key with the properties required by this " + "example can be created for free at " + "https://configure.51degrees.com/?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-cloud-useragentclienthints_web-app.py&utm_term=resource-key-required. " + diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/example_utils.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/example_utils.py index d36c35ea..05260f95 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/example_utils.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/example_utils.py @@ -23,6 +23,8 @@ from datetime import datetime, timedelta from threading import Event, Thread import os +import shlex +import subprocess from pathlib import Path @@ -51,6 +53,15 @@ class ExampleUtils: # file named in its configuration. DATA_FILE_ENV_VAR = "51DEGREES_DD_PATH" + # The data repository's device detection folder, from the root of a + # checkout of this repository. Each file the data repository links to + # belongs in this folder, beside its .link file. + DATA_FOLDER = os.path.join("cxx", "data", "device-detection") + + # Sent with each download, so the downloads these examples make can be + # counted. + DOWNLOAD_SOURCE = "examples-51degrees-python" + @staticmethod def get_resource_key_from_config(config): key = "" @@ -183,12 +194,29 @@ def __get_env_variable(name): else: return "" - # Find the specified filename within the specified directory. - # If no directory is specified, the working directory is used. - # If the file cannot be found, the algorithm will move to the - # parent directory and repeat the process. + # Find the specified file. A path to a file that exists is returned as + # it is. Next, the data repository's folder is checked in the working + # directory and in each directory above it, and a file the data + # repository links to is downloaded there when it is missing. Otherwise + # the working directory is searched for a file with the same name, then + # each parent directory in turn. None is returned when nothing is found. @staticmethod def find_file(file_name): + path = Path(file_name).absolute() + if path.is_file(): + return str(path.resolve()) + + # The data repository's copy is preferred to a file with the same + # name elsewhere in the tree, which may be out of date or a Git LFS + # pointer. + current = Path(os.getcwd()).absolute() + for root in [current, *current.parents]: + candidate = root / ExampleUtils.DATA_FOLDER / path.name + if candidate.is_file(): + return str(candidate) + if candidate.with_name(candidate.name + ".link").is_file(): + return ExampleUtils.__download(str(candidate)) + stop_event = Event() result = [None] # Start the file system search as a separate task. @@ -235,6 +263,39 @@ def __find_file(file_name, stop_event, result, search_dir=None): if search_dir.parent != search_dir and not stop_event.is_set(): ExampleUtils.__find_file(file_name, stop_event, result, search_dir.parent) + # Download a file the data repository links to, by running the data + # repository's own script for that file, and return its path, or None + # when it could not be downloaded. The data repository is the folder + # above the one that holds the file's .link. + @staticmethod + def __download(path): + name = os.path.basename(path) + data = os.path.dirname(os.path.dirname(path)) + if os.name == "nt": + command = [ + "powershell", "-NoProfile", "-ExecutionPolicy", "Bypass", + "-File", os.path.join(data, "get-data-files.ps1"), name, + "-Source", ExampleUtils.DOWNLOAD_SOURCE] + text = subprocess.list2cmdline(command) + else: + command = [ + "sh", os.path.join(data, "get-data-files.sh"), + "-s", ExampleUtils.DOWNLOAD_SOURCE, name] + text = shlex.join(command) + + # Flushed first, because the script writes to the same output. + print(f"{name} is missing, so it is being downloaded by running\n" + f"{text}", flush=True) + try: + subprocess.run(command, check=False) + except OSError: + pass + if os.path.isfile(path): + return path + print(f"{name} could not be downloaded. Run this command to try " + f"again.\n{text}") + return None + @staticmethod def get_data_file_date(engine): date = engine.engine.getPublishedTime() @@ -272,9 +333,10 @@ def check_data_file(pipeline, logger): f"that is more than '{ExampleUtils.DATA_FILE_AGE_WARNING}' days " + "old. A more recent data file may be needed to " + "correctly detect the latest devices, browsers, " + - "etc. The latest lite data file is available from " + - "the device-detection-data repository on GitHub " + - "https://github.com/51Degrees/device-detection-data. " + + "etc. Get the latest lite data file by running " + + "cxx/data/get-data-files.sh -u 51Degrees-LiteV4.1.hash, " + + "or cxx/data/get-data-files.ps1 51Degrees-LiteV4.1.hash " + + "-Update on Windows. " + "Find out about the Enterprise data file, which " + "includes automatic updates, on our pricing page: " + "https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-example_utils.py&utm_term=data-file-age-warning") diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/datafileupdate_console.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/datafileupdate_console.py index bf9ff7a3..4fd126fc 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/datafileupdate_console.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/datafileupdate_console.py @@ -53,9 +53,8 @@ # # ## Lite Data File # Lite data files (free-to-use, limited capabilities, no license key required) are created roughly -# once a month and cannot be updated using auto-update, they may be downloaded from -# [Github](https://github.com/51Degrees/device-detection-data) and are included with -# source distributions of this software. +# once a month and cannot be updated using auto-update, they may be downloaded with the +# scripts in the [data](https://github.com/51Degrees/data) repository on GitHub. # # # Update on Start-Up # You can configure the pipeline builder to download an Enterprise data file on start-up. @@ -235,15 +234,18 @@ def run(self, data_file, license_key, interactive, logger, output): auto_update = False, licence_keys = "").add_logger(logger).build() - # and output the results - ExampleUtils.check_data_file(pipeline, logger) - if (ExampleUtils.get_data_file_tier(pipeline.get_element("device")) == "Lite"): - logger.log("error", - "Will not download an 'Enterprise' data file over the top of " - "a 'Lite' data file, please supply another location.") - raise Exception("File supplied has wrong data tier") - logger.log("info", "Existing data file will be replaced with downloaded data file") - logger.log("info", f"Existing data file will be copied to {copy_data_file_name}") + # The pipeline is closed at the end of the with block, which + # releases the native data it holds before the file is replaced. + with pipeline: + # and output the results + ExampleUtils.check_data_file(pipeline, logger) + if (ExampleUtils.get_data_file_tier(pipeline.get_element("device")) == "Lite"): + logger.log("error", + "Will not download an 'Enterprise' data file over the top of " + "a 'Lite' data file, please supply another location.") + raise Exception("File supplied has wrong data tier") + logger.log("info", "Existing data file will be replaced with downloaded data file") + logger.log("info", f"Existing data file will be copied to {copy_data_file_name}") # do we really want to do this if (interactive): @@ -296,41 +298,44 @@ def run(self, data_file, license_key, interactive, logger, output): data_update_product_type="V4Enterprise", ).add_logger(logger).build() - - # thread blocks till update checking is complete - or if there is an - # exception we don't get this far - update_event.wait() - output(f"Update on start-up complete - status - {update_event.status}") - - if update_event.status == UpdateStatus.AUTO_UPDATE_SUCCESS: - - output("Modifying downloaded file to trigger reload - please wait for that" - " to complete") - - # wait for the dataUpdateService to notify us that it has updated - update_event.clear() - - # it's the same file but changing the file metadata will trigger reload, - # demonstrating that if you download a new file and replace the - # existing one, then it will be loaded - now = datetime.now().timestamp() - try: - os.utime(data_file, (now, now)) - except: - raise Exception("Could not modify file time, abandoning " - "example") - - if update_event.wait(120): - output(f"Update on file modification complete, status: {update_event.status}") - else: - output("Update on file modification timed out") - else: - logger.log("error", "Auto update was not successful, abandoning example") - error_message = f"Auto update failed: {update_event.status}" - if update_event.status == UpdateStatus.AUTO_UPDATE_ERR_429_TOO_MANY_ATTEMPTS: - output(error_message) + # The pipeline is closed at the end of the with block, which stops + # the automatic updates and the file system watcher, and releases + # the native data it holds. + with pipeline: + # thread blocks till update checking is complete - or if there is an + # exception we don't get this far + update_event.wait() + output(f"Update on start-up complete - status - {update_event.status}") + + if update_event.status == UpdateStatus.AUTO_UPDATE_SUCCESS: + + output("Modifying downloaded file to trigger reload - please wait for that" + " to complete") + + # wait for the dataUpdateService to notify us that it has updated + update_event.clear() + + # it's the same file but changing the file metadata will trigger reload, + # demonstrating that if you download a new file and replace the + # existing one, then it will be loaded + now = datetime.now().timestamp() + try: + os.utime(data_file, (now, now)) + except: + raise Exception("Could not modify file time, abandoning " + "example") + + if update_event.wait(120): + output(f"Update on file modification complete, status: {update_event.status}") + else: + output("Update on file modification timed out") else: - raise Exception(error_message) + logger.log("error", "Auto update was not successful, abandoning example") + error_message = f"Auto update failed: {update_event.status}" + if update_event.status == UpdateStatus.AUTO_UPDATE_ERR_429_TOO_MANY_ATTEMPTS: + output(error_message) + else: + raise Exception(error_message) output("Finished Example") diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/failuretomatch.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/failuretomatch.py index f640b1d8..9db313bd 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/failuretomatch.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/failuretomatch.py @@ -33,46 +33,49 @@ performance_profile = 'MaxPerformance', auto_update=False).build() -# We create a FlowData object from the pipeline -# this is used to add evidence to and then process +# The pipeline is closed at the end of the with block, which releases the +# native data it holds. +with pipeline: + # We create a FlowData object from the pipeline + # this is used to add evidence to and then process. It is closed at the + # end of its with block, which releases the results of the detection. + with pipeline.create_flowdata() as flowdata1: -flowdata1 = pipeline.create_flowdata() + # Here we add a User-Agent of an iphone as evidence -# Here we add a User-Agent of an iphone as evidence + iphone_ua = "Mozilla/5.0 (iPhone; CPU iPhone OS 11_2 like Mac OS X) AppleWebKit/604.4.7 (KHTML, like Gecko) Mobile/15C114" -iphone_ua = "Mozilla/5.0 (iPhone; CPU iPhone OS 11_2 like Mac OS X) AppleWebKit/604.4.7 (KHTML, like Gecko) Mobile/15C114" + flowdata1.evidence.add("header.user-agent", iphone_ua) -flowdata1.evidence.add("header.user-agent", iphone_ua) + # Now we process the FlowData -# Now we process the FlowData + flowdata1.process() -flowdata1.process() + # To check whether the User-Agent is a mobile device we look at the ismobile property + # inside the Device Detection Engine -# To check whether the User-Agent is a mobile device we look at the ismobile property -# inside the Device Detection Engine + # first we check if this has a meaningful result -# first we check if this has a meaningful result + print("Is User-Agent " + iphone_ua + " a mobile device?: ") + if flowdata1.device.ismobile.has_value(): + print(flowdata1.device.ismobile.value()) + else: + # Output why the value isn't meaningful + print(flowdata1.device.ismobile.no_value_message()) -print("Is User-Agent " + iphone_ua + " a mobile device?: ") -if flowdata1.device.ismobile.has_value(): - print(flowdata1.device.ismobile.value()) -else: - # Output why the value isn't meaningful - print(flowdata1.device.ismobile.no_value_message()) + # Now we do the same with a new User-Agent, this time a corrupted one -# Now we do the same with a new User-Agent, this time a corrupted one + badUA = "--" -badUA = "--" + with pipeline.create_flowdata() as flowdata2: -flowdata2 = pipeline.create_flowdata() + flowdata2.evidence.add("header.user-agent", badUA) -flowdata2.evidence.add("header.user-agent", badUA) + flowdata2.process() -flowdata2.process() - -print("Is User-Agent " + badUA + " a mobile device?: ") -if flowdata2.device.ismobile.has_value(): - print(flowdata2.device.ismobile.value()) -else: - # Output why the value isn't meaningful - print(flowdata2.device.ismobile.no_value_message()) + print("Is User-Agent " + badUA + " a mobile device?: ") + if flowdata2.device.ismobile.has_value(): + print(flowdata2.device.ismobile.value()) + else: + # Output why the value isn't meaningful + print(flowdata2.device.ismobile.no_value_message()) diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_console.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_console.py index 9c695f43..3675fee2 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_console.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_console.py @@ -51,15 +51,15 @@ def run(self, data_file, logger, output): # In this example, we use the DeviceDetectionPipelineBuilder # and configure it in code. For more information about # pipelines in general see the documentation at - # https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-gettingstarted_console.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-gettingstarted_console.py&utm_term=run pipeline = DeviceDetectionPipelineBuilder( data_file_path = data_file, # We use the low memory profile as its performance is # sufficient for this example. See the documentation for # more detail on this and other configuration options: # https://51degrees.com/documentation/_device_detection__features__performance_options.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-gettingstarted_console.py&utm_term=run - # https://51degrees.com/documentation/_features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-gettingstarted_console.py&utm_term=run - # https://51degrees.com/documentation/_features__usage_sharing.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-gettingstarted_console.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-gettingstarted_console.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__features__usage_sharing.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-gettingstarted_console.py&utm_term=run performance_profile = "MaxPerformance", # inhibit sharing usage for this test, usually this # should be set "true" @@ -68,11 +68,16 @@ def run(self, data_file, logger, output): auto_update = False, licence_keys = "").add_logger(logger).build() - ExampleUtils.check_data_file(pipeline, logger) + # The pipeline is closed at the end of the with block, which + # releases the native data it holds. An application normally builds + # one pipeline when it starts, uses it for every detection, and + # closes it when it stops. + with pipeline: + ExampleUtils.check_data_file(pipeline, logger) - # carry out some sample detections - for values in EVIDENCE_VALUES: - self.analyseEvidence(values, pipeline, output) + # carry out some sample detections + for values in EVIDENCE_VALUES: + self.analyseEvidence(values, pipeline, output) def analyseEvidence(self, evidence, pipeline, output): @@ -83,42 +88,45 @@ def analyseEvidence(self, evidence, pipeline, output): # and usually consists of a number of HTTP Header field # values, in this case represented by a dictionary of header # name/value entries. - data = pipeline.create_flowdata() + # Closing the FlowData, which the with statement does at the end of + # the block, releases the results of the detection straight away, + # rather than when the garbage collector next runs. + with pipeline.create_flowdata() as data: - message = [] + message = [] - # List the evidence - message.append("Input values:\n") - for key in evidence: - message.append(f"\t{key}: {evidence[key]}\n") + # List the evidence + message.append("Input values:\n") + for key in evidence: + message.append(f"\t{key}: {evidence[key]}\n") - output("".join(message)) - - # Add the evidence values to the flow data - data.evidence.add_from_dict(evidence) - - # Process the flow data. - data.process() - - message = [] - message.append("Results:\n") - - # Now that it's been processed, the flow data will have - # been populated with the result. In this case, we want - # information about the device, which we can get by - # asking for a result matching named "device" - device = data.device - - # Display the results of the detection, which are called - # device properties. See the property dictionary at - # https://51degrees.com/developers/property-dictionary?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-gettingstarted_console.py&utm_term=analyseevidence - # for details of all available properties. - self.outputValue("Mobile Device", device.ismobile, message) - self.outputValue("Platform Name", device.platformname, message) - self.outputValue("Platform Version", device.platformversion, message) - self.outputValue("Browser Name", device.browsername, message) - self.outputValue("Browser Version", device.browserversion, message) - output("".join(message)) + output("".join(message)) + + # Add the evidence values to the flow data + data.evidence.add_from_dict(evidence) + + # Process the flow data. + data.process() + + message = [] + message.append("Results:\n") + + # Now that it's been processed, the flow data will have + # been populated with the result. In this case, we want + # information about the device, which we can get by + # asking for a result matching named "device" + device = data.device + + # Display the results of the detection, which are called + # device properties. See the property dictionary at + # https://51degrees.com/developers/property-dictionary?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-gettingstarted_console.py&utm_term=analyseevidence + # for details of all available properties. + self.outputValue("Mobile Device", device.ismobile, message) + self.outputValue("Platform Name", device.platformname, message) + self.outputValue("Platform Version", device.platformversion, message) + self.outputValue("Browser Name", device.browsername, message) + self.outputValue("Browser Version", device.browserversion, message) + output("".join(message)) def outputValue(self, name, value, message): # Individual result values have a wrapper called @@ -152,9 +160,9 @@ def main(argv): else: logger.log("error", "Failed to find a device detection " + - "data file. Make sure the device-detection-data " + - "submodule has been updated by running " + - "`git submodule update --recursive`.") + "data file. Make sure the data submodule is there, " + + "by running `git submodule update --init --recursive`, " + + "and run this example again to download the file.") if __name__ == "__main__": main(sys.argv[1:]) diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/app.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/app.py index c2ec0e2b..a8e1e3ce 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/app.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/app.py @@ -58,7 +58,7 @@ # ``` # # Results can also be accessed in client-side code by using the `fod` object. See the -# [JavaScriptBuilderElementBuilder](https://51degrees.com/pipeline-python/4.3/classpipeline-python_1_1fiftyone__pipeline__core_1_1fiftyone__pipeline__core_1_1javascriptbuilde778a9036818b19ab55d981a40be4a4d7.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-gettingstarted_web-app.py&utm_term=top) +# [JavascriptBuilderElement](https://51degrees.com/documentation/apis/pipeline-python/classfiftyone__pipeline__core_1_1javascriptbuilder_1_1_javascript_builder_element.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-gettingstarted_web-app.py&utm_term=top) # for details on available settings such as changing the `fod` name. # ```{js} # window.onload = function () { @@ -102,7 +102,10 @@ def run(self): # The port can be set with the PORT environment variable, as it can # for the cloud example, so the example can run next to other services. - GettingStartedWeb.app.run(port=int(os.environ.get("PORT", 5001))) + # The pipeline lives as long as the app, and is closed when the server + # stops, which releases the native data it holds. + with GettingStartedWeb.pipeline: + GettingStartedWeb.app.run(port=int(os.environ.get("PORT", 5001))) # First we make a JSON route that will be called from the client side and will return # a JSON encoded property database using any additional evidence provided by the client @@ -111,21 +114,23 @@ def run(self): @app.route('/json', methods=['POST']) def jsonroute(): - # Create the flowdata object for the JSON route - flowdata = GettingStartedWeb.pipeline.create_flowdata() + # Create the flowdata object for the JSON route. It is closed when the + # route returns, which releases the results of the detection straight + # away rather than when the garbage collector next runs. + with GettingStartedWeb.pipeline.create_flowdata() as flowdata: - # Add any information from the request (headers, cookies and additional - # client side provided information) + # Add any information from the request (headers, cookies and additional + # client side provided information) - flowdata.evidence.add_from_dict(webevidence(request)) + flowdata.evidence.add_from_dict(webevidence(request)) - # Process the flowdata + # Process the flowdata - flowdata.process() + flowdata.process() - # Return the JSON from the JSONBundler engine + # Return the JSON from the JSONBundler engine - return json.dumps(flowdata.jsonbundler.json) + return json.dumps(flowdata.jsonbundler.json) # Next we serve the client-side JavaScript from its own route. The other # Pipeline APIs ship a web integration that intercepts '/51Degrees.core.js' @@ -136,26 +141,27 @@ def jsonroute(): @app.route('/51Degrees.core.js') def core_js(): - # Create the flowdata object for the JavaScript route - flowdata = GettingStartedWeb.pipeline.create_flowdata() + # Create the flowdata object for the JavaScript route. It is closed + # when the route returns. + with GettingStartedWeb.pipeline.create_flowdata() as flowdata: - # Add any information from the request (headers, cookies and additional - # client side provided information). Query parameters are included, so - # the per-request 'fod-js-enable-cookies' override is picked up here and - # applied by the JavaScriptBuilder engine. + # Add any information from the request (headers, cookies and additional + # client side provided information). Query parameters are included, so + # the per-request 'fod-js-enable-cookies' override is picked up here and + # applied by the JavaScriptBuilder engine. - flowdata.evidence.add_from_dict(webevidence(request)) + flowdata.evidence.add_from_dict(webevidence(request)) - # Process the flowdata + # Process the flowdata - flowdata.process() + flowdata.process() - # Return the JavaScript from the JavaScriptBuilder engine + # Return the JavaScript from the JavaScriptBuilder engine - response = make_response(flowdata.javascriptbuilder.javascript) - response.headers["Content-Type"] = "application/x-javascript" + response = make_response(flowdata.javascriptbuilder.javascript) + response.headers["Content-Type"] = "application/x-javascript" - return response + return response # In the main route we dynamically update the screen's device property display # using the above JSON route @@ -164,36 +170,37 @@ def core_js(): @app.route('/') def server(): - # Create the flowdata object for the JSON route - flowdata = GettingStartedWeb.pipeline.create_flowdata() + # Create the flowdata object for the page. It is closed when the route + # returns, once the page has been rendered from it. + with GettingStartedWeb.pipeline.create_flowdata() as flowdata: - # Add any information from the request (headers, cookies and additional - # client side provided information) + # Add any information from the request (headers, cookies and additional + # client side provided information) - flowdata.evidence.add_from_dict(webevidence(request)) + flowdata.evidence.add_from_dict(webevidence(request)) - # Process the flowdata + # Process the flowdata - flowdata.process() + flowdata.process() - response = make_response() + response = make_response() - # Some browsers require that extra HTTP headers are explicitly - # requested. So set whatever headers are required by the browser in - # order to return the evidence needed by the pipeline. - # More info on this can be found at - # https://51degrees.com/blog/user-agent-client-hints?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-gettingstarted_web-app.py&utm_term=server + # Some browsers require that extra HTTP headers are explicitly + # requested. So set whatever headers are required by the browser in + # order to return the evidence needed by the pipeline. + # More info on this can be found at + # https://51degrees.com/blog/user-agent-client-hints?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-gettingstarted_web-app.py&utm_term=server - set_response_header(flowdata, response) + set_response_header(flowdata, response) - # Generate the HTML - response.set_data(render_template( - 'index.html', - data=flowdata, - utils=ExampleUtils, - response=response)) + # Generate the HTML + response.set_data(render_template( + 'index.html', + data=flowdata, + utils=ExampleUtils, + response=response)) - return response + return response # Typically, something like this will not be necessary. @@ -246,8 +253,9 @@ def build_config(): if foundDataFile == False: raise Exception("Failed to find a device detection data file matching " + f"'{dataFile}'. If using the lite file, then make sure the " + - "device-detection-data submodule has been updated by running " + - "`git submodule update --recursive`. Otherwise, ensure that the filename " + + "data submodule is there, by running " + + "`git submodule update --init --recursive`, and run this " + + "example again to download the file. Otherwise, ensure that the filename " + "is correct in config.json, or set the environment variable " + f"'{ExampleUtils.DATA_FILE_ENV_VAR}' to the path of the data file.") diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/templates/index.html b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/templates/index.html index 8106f4cc..d26ab8da 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/templates/index.html +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/templates/index.html @@ -30,8 +30,8 @@
WARNING: This example is using a data file that is more than {{ utils.DATA_FILE_AGE_WARNING }} days old. A more recent data file may be needed to correctly detect the latest devices, - browsers, etc. The latest lite data file is available from the - device-detection-data + browsers, etc. Get the latest lite data file with the scripts in the + data repository on GitHub. Find out about the Enterprise data file, which includes automatic daily updates, on our pricing page.
diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/match_metrics.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/match_metrics.py index f9f254f8..6a223d00 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/match_metrics.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/match_metrics.py @@ -88,83 +88,89 @@ def run(self, data_file, show_descs, logger, output): # We want to show the matching evidence characters as part of this example, so we have to set # this flag to true. update_matched_useragent = True).add_logger(logger).build() + + # The pipeline is closed at the end of the with block, which + # releases the native data it holds. + with pipeline: + ExampleUtils.check_data_file(pipeline, logger) + + # Closing the FlowData, which the with statement does at the end + # of the block, releases the results of the detection straight + # away, rather than when the garbage collector next runs. + with pipeline.create_flowdata() as data: + + # Process a single evidence to retrieve the values + # associated with the user-agent and other evidence such as sec-ch-* for the + # selected properties. + data.evidence.add_from_dict(self.Evidence) + data.process() + + device = data.device + + output("--- Compare evidence with what was matched ---\n") + output("Evidence") + # output the evidence in reverse value length order + for entry in sorted(self.Evidence.items(), key=lambda item: len(item[1]), reverse=True): + output(f" {entry[0]}: {entry[1]}") + # Obtain the matched User-Agents: the matched substrings in the + # User-Agents are separated with underscores - output in forward length order. + output("Matches") + for entry in sorted(device.useragents.value(), key=lambda item: len(item)): + output(f" Matched User-Agent: {entry}") + + output("") + + + output("--- Listing all available properties, by component, by property name ---") + output("For a discussion of what the match properties mean, see: https://51degrees.com/documentation/_device_detection__hash.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-match_metrics.py&utm_term=match-properties-discussion#DeviceDetection_Hash_DataSetProduction_Performance\n") + + # get the properties available from the DeviceDetection engine + # which has the key "device". For the sake of illustration we will + # retrieve it indirectly. + hashEngineElementKey = pipeline.get_element("device").datakey + + # retrieve the available properties from the hash engine. The properties + # available depends on + # a) the use of restricted_properties in the builder (see above) + # which controls which properties will be extracted, and also affects + # the performance of extraction + # b) the tier of data file being used. The Lite data file contains fewer + # than 20 of the >200 available properties + availableProperties = dict(pipeline.get_properties()[hashEngineElementKey]) - ExampleUtils.check_data_file(pipeline, logger) - - data = pipeline.create_flowdata() - - # Process a single evidence to retrieve the values - # associated with the user-agent and other evidence such as sec-ch-* for the - # selected properties. - data.evidence.add_from_dict(self.Evidence) - data.process() - - device = data.device - - output("--- Compare evidence with what was matched ---\n") - output("Evidence") - # output the evidence in reverse value length order - for entry in sorted(self.Evidence.items(), key=lambda item: len(item[1]), reverse=True): - output(f" {entry[0]}: {entry[1]}") - # Obtain the matched User-Agents: the matched substrings in the - # User-Agents are separated with underscores - output in forward length order. - output("Matches") - for entry in sorted(device.useragents.value(), key=lambda item: len(item)): - output(f" Matched User-Agent: {entry}") - - output("") - - - output("--- Listing all available properties, by component, by property name ---") - output("For a discussion of what the match properties mean, see: https://51degrees.com/documentation/_device_detection__hash.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-match_metrics.py&utm_term=match-properties-discussion#DeviceDetection_Hash_DataSetProduction_Performance\n") - - # get the properties available from the DeviceDetection engine - # which has the key "device". For the sake of illustration we will - # retrieve it indirectly. - hashEngineElementKey = pipeline.get_element("device").datakey - - # retrieve the available properties from the hash engine. The properties - # available depends on - # a) the use of restricted_properties in the builder (see above) - # which controls which properties will be extracted, and also affects - # the performance of extraction - # b) the tier of data file being used. The Lite data file contains fewer - # than 20 of the >200 available properties - availableProperties = dict(pipeline.get_properties()[hashEngineElementKey]) - - # create a Map keyed on the component name of the properties available - # components being hardware, browser, OS and Crawler. - def get_component(property): - return property[1]["component"] - - availableProperties = dict(sorted(availableProperties.items(),key=lambda item: get_component(item))) - categoryMap = groupby(availableProperties.items(), get_component) - # iterate the map created above - for component, properties in categoryMap: - output(component) - for propertyKey, property in properties: - propertyName = property["name"] - propertyDescription = property["description"] + # create a Map keyed on the component name of the properties available + # components being hardware, browser, OS and Crawler. + def get_component(property): + return property[1]["component"] + + availableProperties = dict(sorted(availableProperties.items(),key=lambda item: get_component(item))) + categoryMap = groupby(availableProperties.items(), get_component) + # iterate the map created above + for component, properties in categoryMap: + output(component) + for propertyKey, property in properties: + propertyName = property["name"] + propertyDescription = property["description"] - # while we get the available properties and their metadata from the - # pipeline we get the values for the last detection from flowData - value = device[propertyKey] + # while we get the available properties and their metadata from the + # pipeline we get the values for the last detection from flowData + value = device[propertyKey] - # output property names, values and descriptions - # some property values are lists. - if value.has_value() and isinstance(value.value(), list): - output(f" {propertyName}: {len(value.value())} Values") - for item in value.value(): - output(f" : {item}") + # output property names, values and descriptions + # some property values are lists. + if value.has_value() and isinstance(value.value(), list): + output(f" {propertyName}: {len(value.value())} Values") + for item in value.value(): + output(f" : {item}") - else: - output(f" {propertyName}: {value.value()}") + else: + output(f" {propertyName}: {value.value()}") - if (show_descs == True): - output(f" {propertyDescription}") + if (show_descs == True): + output(f" {propertyDescription}") - output() - logger.log("info", "Finished Match Metrics Example") + output() + logger.log("info", "Finished Match Metrics Example") # Evidence values from a windows 11 device using a browser # that supports User-Agent Client Hints. @@ -189,8 +195,9 @@ def main(argv): else: logger.log("error", "Failed to find a device detection data file. Make sure the " + - "device-detection-data submodule has been updated by running " + - "`git submodule update --recursive`.") + "data submodule is there, by running " + + "`git submodule update --init --recursive`, and run this " + + "example again to download the file.") if __name__ == "__main__": main(sys.argv[1:]) \ No newline at end of file diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/metadata_console.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/metadata_console.py index 1e5e0929..5e605158 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/metadata_console.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/metadata_console.py @@ -60,15 +60,15 @@ def run(self, data_file, logger, output): # In this example, we use the DeviceDetectionPipelineBuilder # and configure it in code. For more information about # pipelines in general see the documentation at - # https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-metadata_console.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-metadata_console.py&utm_term=run pipeline = DeviceDetectionPipelineBuilder( data_file_path = data_file, # We use the low memory profile as its performance is # sufficient for this example. See the documentation for # more detail on this and other configuration options: # https://51degrees.com/documentation/_device_detection__features__performance_options.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-metadata_console.py&utm_term=run - # https://51degrees.com/documentation/_features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-metadata_console.py&utm_term=run - # https://51degrees.com/documentation/_features__usage_sharing.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-metadata_console.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-metadata_console.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__features__usage_sharing.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-metadata_console.py&utm_term=run performance_profile = "LowMemory", # inhibit sharing usage for this test, usually this # should be set "true" @@ -76,10 +76,14 @@ def run(self, data_file, logger, output): # Inhibit auto-update of the data file for this example auto_update = False, licence_keys = "").add_logger(logger).build() - self.outputProperties(pipeline.get_element("device"), output) - self.outputEvidenceKeyDetails(pipeline.get_element("device"), output) - ExampleUtils.check_data_file(pipeline, logger) + # The pipeline is closed at the end of the with block, which + # releases the native data it holds. + with pipeline: + self.outputProperties(pipeline.get_element("device"), output) + self.outputEvidenceKeyDetails(pipeline.get_element("device"), output) + + ExampleUtils.check_data_file(pipeline, logger) def outputEvidenceKeyDetails(self, engine, output): @@ -129,8 +133,9 @@ def main(argv): else: logger.log("error", "Failed to find a device detection data file. Make sure the " + - "device-detection-data submodule has been updated by running " + - "`git submodule update --recursive`.") + "data submodule is there, by running " + + "`git submodule update --init --recursive`, and run this " + + "example again to download the file.") if __name__ == "__main__": main(sys.argv[1:]) \ No newline at end of file diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/offlineprocessing.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/offlineprocessing.py index 2487f3e9..500fdba7 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/offlineprocessing.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/offlineprocessing.py @@ -73,15 +73,15 @@ def run(self, data_file, evidence_yaml, logger, output): # In this example, we use the DeviceDetectionPipelineBuilder # and configure it in code. For more information about # pipelines in general see the documentation at - # https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-offlineprocessing.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-offlineprocessing.py&utm_term=run pipeline = DeviceDetectionPipelineBuilder( data_file_path = data_file, # We use the low memory profile as its performance is # sufficient for this example. See the documentation for # more detail on this and other configuration options: # https://51degrees.com/documentation/_device_detection__features__performance_options.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-offlineprocessing.py&utm_term=run - # https://51degrees.com/documentation/_features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-offlineprocessing.py&utm_term=run - # https://51degrees.com/documentation/_features__usage_sharing.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-offlineprocessing.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-offlineprocessing.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__features__usage_sharing.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-offlineprocessing.py&utm_term=run performance_profile = "LowMemory", # inhibit sharing usage for this test, usually this # should be set "true" @@ -92,47 +92,50 @@ def run(self, data_file, evidence_yaml, logger, output): # in order to help us improve detection of new devices/browsers/etc, then # this additional data will need to be collected and included as evidence # to the Pipeline. See - # https://51degrees.com/documentation/_features__usage_sharing.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-offlineprocessing.py&utm_term=run#Low_Level_Usage_Sharing + # https://51degrees.com/documentation/_pipeline_api__features__usage_sharing.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-offlineprocessing.py&utm_term=run#Low_Level_Usage_Sharing # for more details on this. usage_sharing = False, # Inhibit auto-update of the data file for this example auto_update = False, licence_keys = "").add_logger(logger).build() - records = 0 - yaml = YAML() - # Replace tab characters as they are not valid in unquoted YAML values - # and some evidence files contain them in user-agent strings. - content = evidence_yaml.read().replace('\t', ' ') - yaml_data = yaml.load_all(content) + # The pipeline is closed at the end of the with block, which + # releases the native data it holds. + with pipeline: + records = 0 + yaml = YAML() + # Replace tab characters as they are not valid in unquoted YAML values + # and some evidence files contain them in user-agent strings. + content = evidence_yaml.read().replace('\t', ' ') + yaml_data = yaml.load_all(content) - try: - # Keep going as long as we have more document records. - for evidence in yaml_data: - # Output progress. - records = records + 1 - if (records % 100 == 0): - logger.log("info", f"Processed {records} records") + try: + # Keep going as long as we have more document records. + for evidence in yaml_data: + # Output progress. + records = records + 1 + if (records % 100 == 0): + logger.log("info", f"Processed {records} records") - # write the yaml document separator - print("---", file = output) - # Pass the record to the pipeline as evidence so that it can be analyzed - headers = {} - for key in evidence: - # Convert to string - YAML may parse values like ?1 as booleans - # Keys already have header. prefix in the YAML file - headers[key] = str(evidence[key]) + # write the yaml document separator + print("---", file = output) + # Pass the record to the pipeline as evidence so that it can be analyzed + headers = {} + for key in evidence: + # Convert to string - YAML may parse values like ?1 as booleans + # Keys already have header. prefix in the YAML file + headers[key] = str(evidence[key]) - self.analyseEvidence(headers, pipeline, output, yaml) - except BaseException as err: - # We can't read the evidence values, so cant write them to the output. Will just - # have to skip this entry. - logger.log("error", err) + self.analyseEvidence(headers, pipeline, output, yaml) + except BaseException as err: + # We can't read the evidence values, so cant write them to the output. Will just + # have to skip this entry. + logger.log("error", err) - # write the yaml document end marker - print("...", file = output) + # write the yaml document end marker + print("...", file = output) - ExampleUtils.check_data_file(pipeline, logger) + ExampleUtils.check_data_file(pipeline, logger) def analyseEvidence(self, evidence, pipeline, output, yaml): # FlowData is a data structure that is used to convey information required for @@ -140,46 +143,50 @@ def analyseEvidence(self, evidence, pipeline, output, yaml): # Information required for detection is called "evidence" and usually consists # of a number of HTTP Header field values, in this case represented by a # dictionary of header name/value entries. - data = pipeline.create_flowdata() - # Add the evidence values to the flow data - data.evidence.add_from_dict(evidence) - # Process the flow data. - data.process() + # Closing the FlowData, which the with statement does at the end of + # the block, releases the results of the detection straight away. + # Without it each record's results would stay in memory until the + # garbage collector next runs. + with pipeline.create_flowdata() as data: + # Add the evidence values to the flow data + data.evidence.add_from_dict(evidence) + # Process the flow data. + data.process() - device = data.device + device = data.device - values = {} - # Add the evidence values to the output - for key in evidence: - values[key] = evidence[key] - # Now add the values that we want to store against the record. - values["device.ismobile"] = device.ismobile.value() if device.ismobile.has_value() else "Unknown" - values["device.platformname"] = ExampleUtils.get_human_readable(device, "platformname") - values["device.platformversion"] = ExampleUtils.get_human_readable(device, "platformversion") - values["device.browsername"] = ExampleUtils.get_human_readable(device, "browsername") - values["device.browserversion"] = ExampleUtils.get_human_readable(device, "browserversion") - values["device.tac"] = ExampleUtils.get_human_readable(device, "tac") - values["device.devicetype"] = ExampleUtils.get_human_readable(device, "devicetype") - values["device.hardwarevendor"] = ExampleUtils.get_human_readable(device, "hardwarevendor") - values["device.hardwarename"] = ExampleUtils.get_human_readable(device, "hardwarename") - values["device.hardwaremodel"] = ExampleUtils.get_human_readable(device, "hardwaremodel") - values["device.nativedevice"] = ExampleUtils.get_human_readable(device, "nativedevice") - values["device.nativename"] = ExampleUtils.get_human_readable(device, "nativename") - values["device.nativemodel"] = ExampleUtils.get_human_readable(device, "nativemodel") - # DeviceId is a unique identifier for the combination of hardware, operating - # system, browser and crawler that has been detected. - # Our device detection solution uses machine learning to find the optimal - # way to identify devices based on the real-world evidence values that we - # observe each day. - # As this changes over time, the result of detection can potentially change - # as well. By storing the device id, we can use this as a lookup in future - # rather than performing detection with the original evidence again. - # Do this by passing an evidence entry with: - # key = query.51D_ProfileIds - # value = [the device id] - # This is much faster and avoids the potential for getting a different - # result. - values["device.deviceid"] = ExampleUtils.get_human_readable(device, "deviceid") + values = {} + # Add the evidence values to the output + for key in evidence: + values[key] = evidence[key] + # Now add the values that we want to store against the record. + values["device.ismobile"] = device.ismobile.value() if device.ismobile.has_value() else "Unknown" + values["device.platformname"] = ExampleUtils.get_human_readable(device, "platformname") + values["device.platformversion"] = ExampleUtils.get_human_readable(device, "platformversion") + values["device.browsername"] = ExampleUtils.get_human_readable(device, "browsername") + values["device.browserversion"] = ExampleUtils.get_human_readable(device, "browserversion") + values["device.tac"] = ExampleUtils.get_human_readable(device, "tac") + values["device.devicetype"] = ExampleUtils.get_human_readable(device, "devicetype") + values["device.hardwarevendor"] = ExampleUtils.get_human_readable(device, "hardwarevendor") + values["device.hardwarename"] = ExampleUtils.get_human_readable(device, "hardwarename") + values["device.hardwaremodel"] = ExampleUtils.get_human_readable(device, "hardwaremodel") + values["device.nativedevice"] = ExampleUtils.get_human_readable(device, "nativedevice") + values["device.nativename"] = ExampleUtils.get_human_readable(device, "nativename") + values["device.nativemodel"] = ExampleUtils.get_human_readable(device, "nativemodel") + # DeviceId is a unique identifier for the combination of hardware, operating + # system, browser and crawler that has been detected. + # Our device detection solution uses machine learning to find the optimal + # way to identify devices based on the real-world evidence values that we + # observe each day. + # As this changes over time, the result of detection can potentially change + # as well. By storing the device id, we can use this as a lookup in future + # rather than performing detection with the original evidence again. + # Do this by passing an evidence entry with: + # key = query.51D_ProfileIds + # value = [the device id] + # This is much faster and avoids the potential for getting a different + # result. + values["device.deviceid"] = ExampleUtils.get_human_readable(device, "deviceid") yaml.dump(values, output) def main(argv): @@ -211,8 +218,9 @@ def main(argv): else: logger.log("error", "Failed to find a device detection data file. Make sure the " + - "device-detection-data submodule has been updated by running " + - "`git submodule update --recursive`.") + "data submodule is there, by running " + + "`git submodule update --init --recursive`, and run this " + + "example again to download the file.") if __name__ == "__main__": main(sys.argv[1:]) diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/performance.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/performance.py index 665cdabf..d795b548 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/performance.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/performance.py @@ -62,24 +62,26 @@ def process_user_agent(user_agent): - # First we create the flowdata using the global pipeline - flowdata = pipeline.create_flowdata() # pylint: disable=used-before-assignment + # First we create the flowdata using the global pipeline. Closing it, + # which the with statement does when the function returns, releases the + # results of the detection straight away. + with pipeline.create_flowdata() as flowdata: # pylint: disable=used-before-assignment - # Here we add the user agent as evidence - flowdata.evidence.add("header.user-agent", user_agent) + # Here we add the user agent as evidence + flowdata.evidence.add("header.user-agent", user_agent) - # We process the flowdata to get the results - flowdata.process() + # We process the flowdata to get the results + flowdata.process() - # To check whether the User-Agent is a mobile device we look at the ismobile - # property inside the Device Detection Engine + # To check whether the User-Agent is a mobile device we look at the ismobile + # property inside the Device Detection Engine - # first we check if this has a meaningful result + # first we check if this has a meaningful result - if flowdata.device.ismobile.has_value(): - return flowdata.device.ismobile.value() - else: - return None + if flowdata.device.ismobile.has_value(): + return flowdata.device.ismobile.value() + else: + return None def process_user_agent_list(data_file, user_agent_list, list_number, output, skip=False): global pipeline @@ -92,23 +94,26 @@ def process_user_agent_list(data_file, user_agent_list, list_number, output, ski usage_sharing=False, auto_update=False).build() - results = { - "mobile": 0, - "notmobile": 0, - "unknown": 0 - } - for user_agent in user_agent_list: - if skip: - break - result = process_user_agent(user_agent[0]) - if(result == None): - results["unknown"] += 1 - if(result == True): - results["mobile"] += 1 - if(result == False): - results["notmobile"] += 1 - - output.put(results, list_number) + # The pipeline is closed at the end of the with block, which releases + # the native data it holds. + with pipeline: + results = { + "mobile": 0, + "notmobile": 0, + "unknown": 0 + } + for user_agent in user_agent_list: + if skip: + break + result = process_user_agent(user_agent[0]) + if(result == None): + results["unknown"] += 1 + if(result == True): + results["mobile"] += 1 + if(result == False): + results["notmobile"] += 1 + + output.put(results, list_number) # Run the process def run(data_file, skip = False): @@ -145,7 +150,7 @@ def run(data_file, skip = False): if __name__ == "__main__": ap = argparse.ArgumentParser(description='Run detection benchmark.') ap.add_argument('-d', '--data_file', default='', help='Path to data file') - ap.add_argument('-u', '--user_agents_file', default='../../cxx/device-detection-data/20000 User Agents.csv', help='Path to user agents evidence file') + ap.add_argument('-u', '--user_agents_file', default='../../cxx/data/device-detection/20000 User Agents.csv', help='Path to user agents evidence file') ap.add_argument('-j', '--json_output', default='', help='Output results in JSON format') args = ap.parse_args() if args.data_file == "": diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/shareusage.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/shareusage.py index 5d808bf4..51bcff0c 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/shareusage.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/shareusage.py @@ -130,46 +130,51 @@ def run(self, evidence_yaml): # Build a pipeline with only the ShareUsage element pipeline = PipelineBuilder().add(share_usage_element).build() - print("Pipeline built successfully with ShareUsage element") - - # Load YAML evidence - yaml = YAML() - yaml_data = yaml.load_all(evidence_yaml) - - # Process evidence records - count = 0 - for evidence in yaml_data: - if count >= RECORDS_TO_PROCESS: - break - - # Filter to only header.* entries - data = filter_evidence( - {k: str(v) for k, v in evidence.items()}, - "header.") - - # Add a client IP address to the evidence for deduplication purposes - # (see class-level note). In a real web application, this would come - # from the request (e.g., REMOTE_ADDR or X-Forwarded-For header) - data[EVIDENCE_CLIENTIP_KEY] = generate_random_ip() - - # Add the usage-from identifier so 51Degrees knows the source of this data. - # This appears as
YourCompanyName
in the XML. - data[EVIDENCE_USAGE_FROM_KEY] = USAGE_FROM_VALUE - - # Create flow data and process - flow_data = pipeline.create_flowdata() - flow_data.evidence.add_from_dict(data) - flow_data.process() - - count += 1 - if count % 10 == 0: - print(f"Processed {count} records") - - print(f"Finished processing {count} records") - print("Waiting for share usage to complete sending...") - - # Give the background thread time to send any remaining data - time.sleep(5) + # The pipeline is closed at the end of the with block, which waits + # for any usage data being sent and then stops the threads that send + # it. + with pipeline: + print("Pipeline built successfully with ShareUsage element") + + # Load YAML evidence + yaml = YAML() + yaml_data = yaml.load_all(evidence_yaml) + + # Process evidence records + count = 0 + for evidence in yaml_data: + if count >= RECORDS_TO_PROCESS: + break + + # Filter to only header.* entries + data = filter_evidence( + {k: str(v) for k, v in evidence.items()}, + "header.") + + # Add a client IP address to the evidence for deduplication purposes + # (see class-level note). In a real web application, this would come + # from the request (e.g., REMOTE_ADDR or X-Forwarded-For header) + data[EVIDENCE_CLIENTIP_KEY] = generate_random_ip() + + # Add the usage-from identifier so 51Degrees knows the source of this data. + # This appears as
YourCompanyName
in the XML. + data[EVIDENCE_USAGE_FROM_KEY] = USAGE_FROM_VALUE + + # Create flow data and process. The with statement closes it + # once it has been processed. + with pipeline.create_flowdata() as flow_data: + flow_data.evidence.add_from_dict(data) + flow_data.process() + + count += 1 + if count % 10 == 0: + print(f"Processed {count} records") + + print(f"Finished processing {count} records") + print("Waiting for share usage to complete sending...") + + # Give the background thread time to send any remaining data + time.sleep(5) print("Done!") @@ -184,8 +189,9 @@ def main(argv): ShareUsageExample().run(input) else: print("ERROR: Failed to find the evidence file. Make sure the " - "device-detection-data submodule has been updated by running " - "`git submodule update --recursive`.") + "data submodule is there, by running " + "`git submodule update --init --recursive`, and run this " + "example again to download the file.") if __name__ == "__main__": diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/useragentclienthints_web/app.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/useragentclienthints_web/app.py index 6d10a1a7..789c2d59 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/useragentclienthints_web/app.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/useragentclienthints_web/app.py @@ -73,7 +73,10 @@ def build(self, config, logger): return self def run(self): - UserAgentClientHintsWeb.app.run(port=5001) + # The pipeline lives as long as the app, and is closed when the server + # stops, which releases the native data it holds. + with UserAgentClientHintsWeb.pipeline: + UserAgentClientHintsWeb.app.run(port=5001) # The main route performs device detection and renders the results. On the # second request, any additional client hints headers requested via the @@ -83,33 +86,35 @@ def run(self): @app.route('/') def server(): - # Create the flowdata object for the device detection - flowdata = UserAgentClientHintsWeb.pipeline.create_flowdata() + # Create the flowdata object for the device detection. It is closed + # when the route returns, once the page has been rendered from it, + # which releases the results of the detection straight away. + with UserAgentClientHintsWeb.pipeline.create_flowdata() as flowdata: - # Add any information from the request (headers, cookies and any other - # additional information) - flowdata.evidence.add_from_dict(webevidence(request)) + # Add any information from the request (headers, cookies and any other + # additional information) + flowdata.evidence.add_from_dict(webevidence(request)) - # Process the flowdata - flowdata.process() + # Process the flowdata + flowdata.process() - response = make_response() + response = make_response() - # Some browsers require that extra HTTP headers are explicitly - # requested. So set whatever headers are required by the browser in - # order to return the evidence needed by the pipeline. - # More info on this can be found at - # https://51degrees.com/blog/user-agent-client-hints?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-useragentclienthints_web-app.py&utm_term=server - set_response_header(flowdata, response) + # Some browsers require that extra HTTP headers are explicitly + # requested. So set whatever headers are required by the browser in + # order to return the evidence needed by the pipeline. + # More info on this can be found at + # https://51degrees.com/blog/user-agent-client-hints?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-onpremise-useragentclienthints_web-app.py&utm_term=server + set_response_header(flowdata, response) - # Generate the HTML - response.set_data(render_template( - 'index.html', - data=flowdata, - utils=ExampleUtils, - response=response)) + # Generate the HTML + response.set_data(render_template( + 'index.html', + data=flowdata, + utils=ExampleUtils, + response=response)) - return response + return response # Typically, something like this will not be necessary. # The device detection API will accept an absolute or relative path for the data file. @@ -148,8 +153,9 @@ def build_config(): if foundDataFile == False: raise Exception("Failed to find a device detection data file matching " + f"'{dataFile}'. If using the lite file, then make sure the " + - "device-detection-data submodule has been updated by running " + - "`git submodule update --recursive`. Otherwise, ensure that the filename " + + "data submodule is there, by running " + + "`git submodule update --init --recursive`, and run this " + + "example again to download the file. Otherwise, ensure that the filename " + "is correct in config.json.") return config diff --git a/device-detection/fiftyone_devicedetection_examples/tests/conftest.py b/device-detection/fiftyone_devicedetection_examples/tests/conftest.py new file mode 100644 index 00000000..fc53a6d0 --- /dev/null +++ b/device-detection/fiftyone_devicedetection_examples/tests/conftest.py @@ -0,0 +1,119 @@ +# ********************************************************************* +# This Original Work is copyright of 51 Degrees Mobile Experts Limited. +# Copyright 2026 51 Degrees Mobile Experts Limited, Davidson House, +# Forbury Square, Reading, Berkshire, United Kingdom RG1 3EU. +# +# This Original Work is licensed under the European Union Public Licence +# (EUPL) v.1.2 and is subject to its terms as set out below. +# +# If a copy of the EUPL was not distributed with this file, You can obtain +# one at https://opensource.org/licenses/EUPL-1.2. +# +# The 'Compatible Licences' set out in the Appendix to the EUPL (as may be +# amended by the European Commission) shall be deemed incompatible for +# the purposes of the Work and the provisions of the compatibility +# clause in Article 5 of the EUPL shall not apply. +# +# If using the Work as, or as part of, a network application, by +# including the attribution notice(s) required under Article 5 of the EUPL +# in the end user terms of the application under an appropriate heading, +# such notice(s) shall fulfill the requirements of that article. +# ********************************************************************* + +# Fails a test that leaves the native results or engine of an on-premise +# engine alive, because they hold the engine's data in memory. The same +# file is in each test folder that runs on-premise engines. +# +# Reference counting frees a native object as soon as nothing holds it. A +# FlowData refers to itself through its evidence, and a pipeline through +# its elements, so when they are not closed only the cycle collector frees +# what they hold. The collector is off while each test runs, so whatever +# the test leaves open is still alive when it ends. Objects that existed +# before the test, such as a pipeline built in setUpClass, are checked once +# every test has run. + +import gc +import importlib + +import pytest + +# The SWIG modules of the on-premise engines, whichever are installed. +NATIVE_MODULES = [ + "fiftyone_devicedetection_onpremise.DeviceDetectionHashEngineModule", + "fiftyone_ipintelligence_onpremise.IpIntelligenceEngineModule", +] + + +def _native_types(): + types = [] + for name in NATIVE_MODULES: + try: + module = importlib.import_module(name) + except ImportError: + continue + types.append(module.ResultsBaseSwig) + types.append(module.EngineBaseSwig) + return tuple(types) + + +NATIVE_TYPES = _native_types() + +# A failed test is not reported again, because its traceback can hold the +# objects it would have closed. +_failed_tests = set() + +# Objects created when the tests were collected are checked at the end only +# when every test ran, because the teardown that closes them runs with the +# tests. +_some_tests_deselected = False + + +def _alive(): + # The real type is used, because isinstance also asks the object for + # its __class__, which some proxies cannot answer. + return [o for o in gc.get_objects() + if issubclass(type(o), NATIVE_TYPES)] + + +def _describe(names): + return (f"{len(names)} native object(s) still alive " + f"({', '.join(sorted(names))}). Close every FlowData and " + "pipeline the test creates, for example with a with " + "statement, so their native data is released straight away.") + + +def pytest_runtest_logreport(report): + if report.failed: + _failed_tests.add(report.nodeid) + + +def pytest_deselected(items): + global _some_tests_deselected + _some_tests_deselected = True + + +@pytest.fixture(autouse=True) +def native_objects_released(request): + gc.collect() + before = {id(o) for o in _alive()} + gc.disable() + try: + yield + left = [type(o).__name__ for o in _alive() if id(o) not in before] + finally: + gc.enable() + gc.collect() + if left and request.node.nodeid not in _failed_tests: + pytest.fail(_describe(left), pytrace=False) + + +def pytest_sessionfinish(session, exitstatus): + gc.collect() + left = [type(o).__name__ for o in _alive()] + if (left and exitstatus == pytest.ExitCode.OK and + not _some_tests_deselected): + reporter = session.config.pluginmanager.get_plugin("terminalreporter") + if reporter is not None: + reporter.write_line( + "After the last test, " + _describe(left), red=True) + session.exitstatus = pytest.ExitCode.TESTS_FAILED diff --git a/device-detection/fiftyone_devicedetection_examples/tests/test_onpremisegettingstartedweb.py b/device-detection/fiftyone_devicedetection_examples/tests/test_onpremisegettingstartedweb.py index 70374c6a..a997f5c7 100644 --- a/device-detection/fiftyone_devicedetection_examples/tests/test_onpremisegettingstartedweb.py +++ b/device-detection/fiftyone_devicedetection_examples/tests/test_onpremisegettingstartedweb.py @@ -39,6 +39,13 @@ class OnPremiseGettingStartedWebTests(flask_unittest.ClientTestCase): config = GettingStartedWeb.build_config() app = GettingStartedWeb().build(config, logger).app + # The example's pipeline lives as long as its app, so it is closed once + # the tests are finished with the app. + @classmethod + def tearDownClass(cls): + GettingStartedWeb.pipeline.close() + super().tearDownClass() + def test_onpremise_getting_started_web(self, client): response = client.get('/') self.assertEqual(200, response.status_code) diff --git a/device-detection/fiftyone_devicedetection_examples/tox.ini b/device-detection/fiftyone_devicedetection_examples/tox.ini index 6b005879..0b6222c0 100644 --- a/device-detection/fiftyone_devicedetection_examples/tox.ini +++ b/device-detection/fiftyone_devicedetection_examples/tox.ini @@ -22,6 +22,12 @@ deps = -e../fiftyone_devicedetection_cloud/ -e../fiftyone_devicedetection_onpremise/ -e../fiftyone_devicedetection_shared/ + # The pipeline packages come from this checkout rather than a package + # index, so the tests run with the pipeline code they are released with. + -e../../pipeline/fiftyone_pipeline_core/ + -e../../pipeline/fiftyone_pipeline_engines/ + -e../../pipeline/fiftyone_pipeline_engines_fiftyone/ + -e../../pipeline/fiftyone_pipeline_cloudrequestengine/ commands = pwsh -ExecutionPolicy Bypass -File ../ci/build-onpremise-c-module.ps1 {[common]commands} diff --git a/device-detection/fiftyone_devicedetection_onpremise/readme.md b/device-detection/fiftyone_devicedetection_onpremise/readme.md index 284077ce..5a83973d 100644 --- a/device-detection/fiftyone_devicedetection_onpremise/readme.md +++ b/device-detection/fiftyone_devicedetection_onpremise/readme.md @@ -1,8 +1,8 @@ # 51Degrees Device Detection Engines - On-Premise -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_onpremise-readme.md&utm_term=51degrees-device-detection-engines-on-premise "THE Fastest and Most Accurate Device Detection") **v4 Device Detection Python** +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_onpremise&utm_term=51degrees-device-detection-engines-on-premise "THE Fastest and Most Accurate Device Detection") **v4 Device Detection Python** -[Developer Documentation](https://51degrees.com/device-detection-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_onpremise-readme.md&utm_term=51degrees-device-detection-engines-on-premise "Developer Documentation") | [Available Properties](https://51degrees.com/resources/property-dictionary?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_onpremise-readme.md&utm_term=51degrees-device-detection-engines-on-premise "View all available properties and values") +[Developer Documentation](https://51degrees.com/device-detection-python/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_onpremise&utm_term=51degrees-device-detection-engines-on-premise "Developer Documentation") | [Available Properties](https://51degrees.com/resources/property-dictionary?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_onpremise&utm_term=51degrees-device-detection-engines-on-premise "View all available properties and values") ## Introduction @@ -56,7 +56,7 @@ fd.process() print(fd.device.ismobile.value()) ``` -For more in-depth examples, check out the [examples](https://51degrees.com/device-detection-python/examples.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_onpremise-readme.md&utm_term=from-pypi) page in the documentation. +For more in-depth examples, check out the [examples](https://51degrees.com/device-detection-python/examples.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_onpremise&utm_term=from-pypi) page in the documentation. ### From GitHub @@ -77,3 +77,12 @@ If you've cloned the GitHub repository, you will be able to run the examples in To run the tests use: `python -m unittest discover -s tests -p test*.py -b` + +## Find out more + +- [Device Detection](https://51degrees.com/device-detection?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_onpremise&utm_term=find-out-more-device-detection) +- [Device Detection documentation](https://51degrees.com/documentation/_device_detection__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_onpremise&utm_term=find-out-more-documentation) +- [Getting started](https://51degrees.com/documentation/_device_detection__quickstart.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_onpremise&utm_term=find-out-more-getting-started) +- [Pricing](https://51degrees.com/pricing?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_onpremise&utm_term=find-out-more-pricing) +- https://github.com/51Degrees/51degrees-python +- https://github.com/51Degrees/51degrees-cxx diff --git a/device-detection/fiftyone_devicedetection_onpremise/setup.py b/device-detection/fiftyone_devicedetection_onpremise/setup.py index d4f60fbc..83b58226 100644 --- a/device-detection/fiftyone_devicedetection_onpremise/setup.py +++ b/device-detection/fiftyone_devicedetection_onpremise/setup.py @@ -232,6 +232,11 @@ def stage_native_source(): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", + project_urls={ + "Homepage": "https://51degrees.com/device-detection?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_onpremise&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_device_detection__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_onpremise&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description = """Device Detection and User-Agent parser for Python that detects the device, browser, operating system, and crawlers from the User-Agent and User-Agent Client Hints by reading a data file on your server. An alternative to UAParser.js, DeviceAtlas, and WURFL.""", keywords=[ "51degrees", @@ -256,5 +261,12 @@ def stage_native_source(): py_modules=["fiftyone_devicedetection_onpremise"], packages=["fiftyone_devicedetection_onpremise"], package_dir={"": "src"}, - install_requires=["fiftyone_devicedetection_shared", "fiftyone_pipeline_engines_fiftyone"], + install_requires=[ + "fiftyone_devicedetection_shared", + "fiftyone_pipeline_engines_fiftyone", + # The first versions with close(), which releases the native memory + # this package holds. + "fiftyone_pipeline_core>=4.5.42", + "fiftyone_pipeline_engines>=4.5.42", + ], ) diff --git a/device-detection/fiftyone_devicedetection_onpremise/src/fiftyone_devicedetection_onpremise/constants.py b/device-detection/fiftyone_devicedetection_onpremise/src/fiftyone_devicedetection_onpremise/constants.py index bb8b3b64..db622b40 100644 --- a/device-detection/fiftyone_devicedetection_onpremise/src/fiftyone_devicedetection_onpremise/constants.py +++ b/device-detection/fiftyone_devicedetection_onpremise/src/fiftyone_devicedetection_onpremise/constants.py @@ -41,3 +41,10 @@ 'on-premise device detection engine. Its results ' 'read from native memory, which is not safe to ' 'share between requests.') + +ENGINE_CLOSED = ('The on-premise device detection engine has been closed, ' + 'so it cannot process evidence. Build a new pipeline.') + +RESULTS_CLOSED = ('These device detection results have been closed, so ' + 'their properties cannot be read. Read the values needed ' + 'before closing the FlowData.') diff --git a/device-detection/fiftyone_devicedetection_onpremise/src/fiftyone_devicedetection_onpremise/devicedetection_onpremise.py b/device-detection/fiftyone_devicedetection_onpremise/src/fiftyone_devicedetection_onpremise/devicedetection_onpremise.py index d8de7ec7..632751ae 100644 --- a/device-detection/fiftyone_devicedetection_onpremise/src/fiftyone_devicedetection_onpremise/devicedetection_onpremise.py +++ b/device-detection/fiftyone_devicedetection_onpremise/src/fiftyone_devicedetection_onpremise/devicedetection_onpremise.py @@ -403,6 +403,20 @@ def on_registration(self, pipeline): super(DeviceDetectionOnPremise, self).on_registration(pipeline) self.init_engine() + def close(self): + + """! + + Stop the data file updates and release the native engine. Its data + is freed once no results read from it are still held, so close + every FlowData the engine processed as well. Closing the pipeline + closes the engine. + + """ + + super(DeviceDetectionOnPremise, self).close() + self.engine = None + def process_internal(self, flow_data): """! @@ -414,6 +428,9 @@ def process_internal(self, flow_data): """ + if getattr(self, "engine", None) is None: + raise Exception(ENGINE_CLOSED) + evidence = EvidenceDeviceDetectionSwig() for key, value in flow_data.evidence.get_all().items(): diff --git a/device-detection/fiftyone_devicedetection_onpremise/src/fiftyone_devicedetection_onpremise/swig_data.py b/device-detection/fiftyone_devicedetection_onpremise/src/fiftyone_devicedetection_onpremise/swig_data.py index b1b4e8dd..17987a59 100644 --- a/device-detection/fiftyone_devicedetection_onpremise/src/fiftyone_devicedetection_onpremise/swig_data.py +++ b/device-detection/fiftyone_devicedetection_onpremise/src/fiftyone_devicedetection_onpremise/swig_data.py @@ -25,6 +25,8 @@ from fiftyone_pipeline_engines.missingproperty_service import MissingPropertyService +from .constants import RESULTS_CLOSED + class OnPremiseMissingPropertyService(MissingPropertyService): def check(self, key, flow_element): @@ -59,6 +61,33 @@ def __init__(self, flow_element, swig_results): self.missing_property_service = OnPremiseMissingPropertyService() + def close(self): + + """! + Release the native results, which keep the engine's data in memory + while they are held, even after the engine is closed. Values already + read are unaffected, and reading a property afterwards raises an + exception. Closing the FlowData closes its results. + """ + + self.swig_results = None + + def get(self, key): + + """! + Get a property value, once the results are known to be open + @type key: string + @param key: The key of the property to retrieve + @returns mixed + """ + + # Checked here, because AspectData.get turns an exception from + # get_internal into a missing property message. + if self.swig_results is None: + raise Exception(RESULTS_CLOSED) + + return super(SwigData, self).get(key) + def get_internal(self, key): """! diff --git a/device-detection/fiftyone_devicedetection_onpremise/tests/conftest.py b/device-detection/fiftyone_devicedetection_onpremise/tests/conftest.py new file mode 100644 index 00000000..fc53a6d0 --- /dev/null +++ b/device-detection/fiftyone_devicedetection_onpremise/tests/conftest.py @@ -0,0 +1,119 @@ +# ********************************************************************* +# This Original Work is copyright of 51 Degrees Mobile Experts Limited. +# Copyright 2026 51 Degrees Mobile Experts Limited, Davidson House, +# Forbury Square, Reading, Berkshire, United Kingdom RG1 3EU. +# +# This Original Work is licensed under the European Union Public Licence +# (EUPL) v.1.2 and is subject to its terms as set out below. +# +# If a copy of the EUPL was not distributed with this file, You can obtain +# one at https://opensource.org/licenses/EUPL-1.2. +# +# The 'Compatible Licences' set out in the Appendix to the EUPL (as may be +# amended by the European Commission) shall be deemed incompatible for +# the purposes of the Work and the provisions of the compatibility +# clause in Article 5 of the EUPL shall not apply. +# +# If using the Work as, or as part of, a network application, by +# including the attribution notice(s) required under Article 5 of the EUPL +# in the end user terms of the application under an appropriate heading, +# such notice(s) shall fulfill the requirements of that article. +# ********************************************************************* + +# Fails a test that leaves the native results or engine of an on-premise +# engine alive, because they hold the engine's data in memory. The same +# file is in each test folder that runs on-premise engines. +# +# Reference counting frees a native object as soon as nothing holds it. A +# FlowData refers to itself through its evidence, and a pipeline through +# its elements, so when they are not closed only the cycle collector frees +# what they hold. The collector is off while each test runs, so whatever +# the test leaves open is still alive when it ends. Objects that existed +# before the test, such as a pipeline built in setUpClass, are checked once +# every test has run. + +import gc +import importlib + +import pytest + +# The SWIG modules of the on-premise engines, whichever are installed. +NATIVE_MODULES = [ + "fiftyone_devicedetection_onpremise.DeviceDetectionHashEngineModule", + "fiftyone_ipintelligence_onpremise.IpIntelligenceEngineModule", +] + + +def _native_types(): + types = [] + for name in NATIVE_MODULES: + try: + module = importlib.import_module(name) + except ImportError: + continue + types.append(module.ResultsBaseSwig) + types.append(module.EngineBaseSwig) + return tuple(types) + + +NATIVE_TYPES = _native_types() + +# A failed test is not reported again, because its traceback can hold the +# objects it would have closed. +_failed_tests = set() + +# Objects created when the tests were collected are checked at the end only +# when every test ran, because the teardown that closes them runs with the +# tests. +_some_tests_deselected = False + + +def _alive(): + # The real type is used, because isinstance also asks the object for + # its __class__, which some proxies cannot answer. + return [o for o in gc.get_objects() + if issubclass(type(o), NATIVE_TYPES)] + + +def _describe(names): + return (f"{len(names)} native object(s) still alive " + f"({', '.join(sorted(names))}). Close every FlowData and " + "pipeline the test creates, for example with a with " + "statement, so their native data is released straight away.") + + +def pytest_runtest_logreport(report): + if report.failed: + _failed_tests.add(report.nodeid) + + +def pytest_deselected(items): + global _some_tests_deselected + _some_tests_deselected = True + + +@pytest.fixture(autouse=True) +def native_objects_released(request): + gc.collect() + before = {id(o) for o in _alive()} + gc.disable() + try: + yield + left = [type(o).__name__ for o in _alive() if id(o) not in before] + finally: + gc.enable() + gc.collect() + if left and request.node.nodeid not in _failed_tests: + pytest.fail(_describe(left), pytrace=False) + + +def pytest_sessionfinish(session, exitstatus): + gc.collect() + left = [type(o).__name__ for o in _alive()] + if (left and exitstatus == pytest.ExitCode.OK and + not _some_tests_deselected): + reporter = session.config.pluginmanager.get_plugin("terminalreporter") + if reporter is not None: + reporter.write_line( + "After the last test, " + _describe(left), red=True) + session.exitstatus = pytest.ExitCode.TESTS_FAILED diff --git a/device-detection/fiftyone_devicedetection_onpremise/tests/test_automatic_data_file_updates.py b/device-detection/fiftyone_devicedetection_onpremise/tests/test_automatic_data_file_updates.py index ea5a031f..e1ac9944 100644 --- a/device-detection/fiftyone_devicedetection_onpremise/tests/test_automatic_data_file_updates.py +++ b/device-detection/fiftyone_devicedetection_onpremise/tests/test_automatic_data_file_updates.py @@ -68,8 +68,10 @@ def test_on_premise_data_update_url(self): data_update_verify_md5=False, data_update_url=f"http://127.0.0.1:{self.port}" ).build() + self.addCleanup(pipeline.close) - pipeline.create_flowdata().process() + with pipeline.create_flowdata() as flowdata: + flowdata.process() flow_element_data_file = pipeline.flow_elements[0].data_file self.assertIsInstance(flow_element_data_file, DeviceDetectionDataFile) @@ -85,8 +87,10 @@ def test_on_premise_data_old_property_data_file_update_base_url(self): data_update_verify_md5=False, data_file_update_base_url=f"http://127.0.0.1:{self.port}" ).build() + self.addCleanup(pipeline.close) - pipeline.create_flowdata().process() + with pipeline.create_flowdata() as flowdata: + flowdata.process() flow_element_data_file = pipeline.flow_elements[0].data_file self.assertIsInstance(flow_element_data_file, DeviceDetectionDataFile) @@ -106,7 +110,9 @@ def test_on_premise_data_update_url_config(self): ) pipeline = PipelineBuilder().build_from_configuration(config) - pipeline.create_flowdata().process() + self.addCleanup(pipeline.close) + with pipeline.create_flowdata() as flowdata: + flowdata.process() flow_element_data_file = pipeline.flow_elements[0].data_file self.assertIsInstance(flow_element_data_file, DeviceDetectionDataFile) @@ -126,7 +132,9 @@ def test_on_premise_data_config_old_property_data_file_update_base_url(self): ) pipeline = PipelineBuilder().build_from_configuration(config) - pipeline.create_flowdata().process() + self.addCleanup(pipeline.close) + with pipeline.create_flowdata() as flowdata: + flowdata.process() flow_element_data_file = pipeline.flow_elements[0].data_file self.assertIsInstance(flow_element_data_file, DeviceDetectionDataFile) @@ -142,8 +150,10 @@ def test_on_premise_data_update_verify_md5(self): data_update_verify_md5=True, data_update_url=f"http://127.0.0.1:{self.port}/md5" ).build() + self.addCleanup(pipeline.close) - pipeline.create_flowdata().process() + with pipeline.create_flowdata() as flowdata: + flowdata.process() flow_element_data_file = pipeline.flow_elements[0].data_file self.assertIsInstance(flow_element_data_file, DeviceDetectionDataFile) @@ -159,8 +169,10 @@ def test_on_premise_data_update_verify_md5_old_property(self): verify_md5=True, data_update_url=f"http://127.0.0.1:{self.port}/md5" ).build() + self.addCleanup(pipeline.close) - pipeline.create_flowdata().process() + with pipeline.create_flowdata() as flowdata: + flowdata.process() flow_element_data_file = pipeline.flow_elements[0].data_file self.assertIsInstance(flow_element_data_file, DeviceDetectionDataFile) @@ -180,7 +192,9 @@ def test_on_premise_data_update_verify_md5_config(self): ) pipeline = PipelineBuilder().build_from_configuration(config) - pipeline.create_flowdata().process() + self.addCleanup(pipeline.close) + with pipeline.create_flowdata() as flowdata: + flowdata.process() flow_element_data_file = pipeline.flow_elements[0].data_file self.assertIsInstance(flow_element_data_file, DeviceDetectionDataFile) @@ -200,7 +214,9 @@ def test_on_premise_data_update_verify_md5_config_old_property(self): ) pipeline = PipelineBuilder().build_from_configuration(config) - pipeline.create_flowdata().process() + self.addCleanup(pipeline.close) + with pipeline.create_flowdata() as flowdata: + flowdata.process() flow_element_data_file = pipeline.flow_elements[0].data_file self.assertIsInstance(flow_element_data_file, DeviceDetectionDataFile) @@ -217,8 +233,10 @@ def test_on_premise_data_update_use_url_formatter(self): data_update_url=f"http://127.0.0.1:{self.port}/", data_update_use_url_formatter=False ).build() + self.addCleanup(pipeline.close) - pipeline.create_flowdata().process() + with pipeline.create_flowdata() as flowdata: + flowdata.process() flow_element_data_file = pipeline.flow_elements[0].data_file self.assertIsInstance(flow_element_data_file, DeviceDetectionDataFile) @@ -238,7 +256,9 @@ def test_on_premise_data_update_use_url_formatter_config(self): ) pipeline = PipelineBuilder().build_from_configuration(config) - pipeline.create_flowdata().process() + self.addCleanup(pipeline.close) + with pipeline.create_flowdata() as flowdata: + flowdata.process() flow_element_data_file = pipeline.flow_elements[0].data_file self.assertIsInstance(flow_element_data_file, DeviceDetectionDataFile) @@ -253,8 +273,10 @@ def test_on_premise_data_update_status_rate_limit(self): verify_md5=True, data_update_url=f"http://127.0.0.1:{self.port}/status/429" ).build() + self.addCleanup(pipeline.close) - pipeline.create_flowdata().process() + with pipeline.create_flowdata() as flowdata: + flowdata.process() flow_element_data_file = pipeline.flow_elements[0].data_file self.assertIsInstance(flow_element_data_file, DeviceDetectionDataFile) @@ -325,7 +347,8 @@ def _init_hash_file_archive(): with gzip.open(archive_file_path, "wb", compresslevel=1) as f_out: shutil.copyfileobj(f_in, f_out) - return hashlib.md5(open(archive_file_path, "rb").read()).hexdigest() + with open(archive_file_path, "rb") as archive: + return hashlib.md5(archive.read()).hexdigest() @staticmethod def clear(): diff --git a/device-detection/fiftyone_devicedetection_onpremise/tests/test_devicedetection.py b/device-detection/fiftyone_devicedetection_onpremise/tests/test_devicedetection.py index 3cd885ce..11ed21f6 100644 --- a/device-detection/fiftyone_devicedetection_onpremise/tests/test_devicedetection.py +++ b/device-detection/fiftyone_devicedetection_onpremise/tests/test_devicedetection.py @@ -36,7 +36,7 @@ from fiftyone_pipeline_core.pipelinebuilder import PipelineBuilder from fiftyone_pipeline_engines.lru_cache import LRUEngineCache -data_file = "../../cxx/device-detection-data/51Degrees-LiteV4.1.hash" +data_file = "../../cxx/data/device-detection/51Degrees-LiteV4.1.hash" mobile_ua = "Mozilla/5.0 (iPhone; CPU iPhone OS 11_2 like Mac OS X) AppleWebKit/604.4.7 (KHTML, like Gecko) Mobile/15C114" @@ -46,7 +46,7 @@ # User agents file -with open('../../cxx/device-detection-data/20000 User Agents.csv', newline='') as file: +with open('../../cxx/data/device-detection/20000 User Agents.csv', newline='') as file: reader = csv.reader(file) user_agents = list(reader) no_user_agents = len(user_agents) @@ -61,6 +61,7 @@ def test_pipeline_builder_shareusage_init(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, licence_keys="").build() + self.addCleanup(pipeline.close) self.assertTrue(pipeline.flow_elements[0].datakey == "shareusage") @@ -71,6 +72,7 @@ def test_on_premise_engine_datafile(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="test", auto_update=True).build() + self.addCleanup(pipeline.close) self.assertTrue(isinstance( pipeline.flow_elements[0].data_file, DeviceDetectionDataFile)) @@ -83,6 +85,7 @@ def test_properties_onpremise(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) properties = pipeline.flow_elements[0].get_properties() @@ -95,6 +98,7 @@ def test_evidencekey_filter_onpremise(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) evidence_key_filter = pipeline.flow_elements[0].get_evidence_key_filter( ) @@ -110,8 +114,10 @@ def test_basic_get_onpremise(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.user-agent", mobile_ua) @@ -126,8 +132,10 @@ def test_process_with_no_evidence(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.process() @@ -140,8 +148,10 @@ def test_process_with_empty_user_agent(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.user-agent", "") @@ -156,8 +166,10 @@ def test_process_with_no_headers(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("irrelevant.evidence", "some-evidence") @@ -172,8 +184,10 @@ def test_process_with_no_useful_headers(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.irrelevant", "some-evidence") @@ -188,8 +202,10 @@ def test_case_insensitive_evidence_keys(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.User-Agent", mobile_ua) @@ -205,8 +221,10 @@ def test_missing_property_service_not_found_anywhere(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.User-Agent", mobile_ua) @@ -230,8 +248,10 @@ def test_missing_property_service_not_found_in_current(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.User-Agent", mobile_ua) @@ -262,8 +282,10 @@ def test_excluded_property(self): data_file_path=data_file, licence_keys="", restricted_properties=["ismobile"]).build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.User-Agent", mobile_ua) @@ -286,6 +308,7 @@ def test_evidencekey_filter_contains_user_agent(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) evidence_key_filter = pipeline.flow_elements[0].get_evidence_key_filter( ) @@ -300,6 +323,7 @@ def test_evidencekey_filter_contains_device_stock_ua(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) evidence_key_filter = pipeline.flow_elements[0].get_evidence_key_filter( ) @@ -314,6 +338,7 @@ def test_evidencekey_filter_contains_query_params(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) evidence_key_filter = pipeline.flow_elements[0].get_evidence_key_filter( ) @@ -330,6 +355,7 @@ def test_evidencekey_filter_case_insensitive_keys(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) evidence_key_filter = pipeline.flow_elements[0].get_evidence_key_filter( ) @@ -350,6 +376,7 @@ def test_evidencekey_filter_overrides(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) evidence_key_filter = pipeline.flow_elements[0].get_evidence_key_filter( ) @@ -366,8 +393,10 @@ def test_profile_overrides(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("query.header.user-agent", "Some user agent") @@ -384,8 +413,10 @@ def test_profile_overrides_no_headers(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("query.51D_ProfileIds", "12280|17779|17470|18092") @@ -400,8 +431,10 @@ def test_profile_overrides_deviceid(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("query.51D_ProfileIds", "12280-17779-17470-18092") @@ -416,8 +449,10 @@ def test_device_id(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.User-Agent", mobile_ua) @@ -443,8 +478,10 @@ def test_build_from_config(self): } pipeline = PipelineBuilder().build_from_configuration(config) + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.User-Agent", mobile_ua) @@ -459,8 +496,10 @@ def test_no_element_exists(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.process() @@ -481,8 +520,10 @@ def test_has_value_false(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.user-agent", "?£$%^&*") @@ -497,8 +538,10 @@ def test_matched_user_agents(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.user-agent", mobile_ua) @@ -519,8 +562,10 @@ def test_value_types(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.user-agent", mobile_ua) @@ -539,8 +584,10 @@ def test_available_properties(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.user-agent", mobile_ua) @@ -561,8 +608,10 @@ def test_engine_init_performance(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.user-agent", mobile_ua) @@ -586,8 +635,10 @@ def test_validate_data_true(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.user-agent", mobile_ua) @@ -613,8 +664,10 @@ def test_validate_data_false(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.evidence.add("header.user-agent", "") @@ -638,30 +691,89 @@ def test_engine_reload(self): pipeline = DeviceDetectionOnPremisePipelineBuilder( data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) fd = pipeline.create_flowdata() + self.addCleanup(fd.close) fd.process() fd.pipeline.get_element("device").engine.refreshData() + def test_flowdata_close_releases_results(self): + """! + Closing the FlowData releases its native results, and reading a + property afterwards raises an exception + """ + + pipeline = DeviceDetectionOnPremisePipelineBuilder( + data_file_path=data_file, usage_sharing=False, licence_keys="").build() + self.addCleanup(pipeline.close) + + with pipeline.create_flowdata() as fd: + fd.evidence.add("header.user-agent", mobile_ua) + fd.process() + ismobile = fd.device.ismobile.value() + + self.assertTrue(ismobile) + self.assertIsNone(fd.device.swig_results) + with self.assertRaises(Exception) as context: + fd.device.ismobile + self.assertIn("closed", str(context.exception)) + + def test_pipeline_close_releases_engine(self): + """! + Closing the pipeline releases the native engine, and processing + afterwards raises an exception + """ + + pipeline = DeviceDetectionOnPremisePipelineBuilder( + data_file_path=data_file, usage_sharing=False, licence_keys="").build() + engine = pipeline.get_element("device") + + pipeline.close() + + self.assertIsNone(engine.engine) + fd = pipeline.create_flowdata() + self.addCleanup(fd.close) + fd.evidence.add("header.user-agent", mobile_ua) + with self.assertRaises(Exception) as context: + fd.process() + self.assertIn("closed", str(context.exception)) + + def test_pipeline_close_stops_updates(self): + """! + Closing the pipeline cancels the engine's next automatic update, + whose timer would otherwise keep the engine alive + """ + + pipeline = DeviceDetectionOnPremisePipelineBuilder( + data_file_path=data_file, usage_sharing=False, licence_keys="test", + auto_update=True).build() + timer = pipeline.get_element("device").data_file.update_timer + + pipeline.close() + + timer.join(10) + self.assertFalse(timer.is_alive()) + def process_user_agent_list(self, pipeline): results = {"mobile": 0, "not_mobile": 0, "unknown": 0} for user_agent in user_agents: user_agent = user_agent[0] - flowdata = pipeline.create_flowdata() - flowdata.evidence.add("header.user-agent", user_agent) - flowdata.process() - - if flowdata.device.ismobile.has_value(): - if(flowdata.device.ismobile.value()): - results["mobile"] += 1 + with pipeline.create_flowdata() as flowdata: + flowdata.evidence.add("header.user-agent", user_agent) + flowdata.process() + + if flowdata.device.ismobile.has_value(): + if(flowdata.device.ismobile.value()): + results["mobile"] += 1 + else: + results["not_mobile"] += 1 else: - results["not_mobile"] += 1 - else: - results["unknown"] += 1 + results["unknown"] += 1 return results diff --git a/device-detection/fiftyone_devicedetection_onpremise/tests/test_properties.py b/device-detection/fiftyone_devicedetection_onpremise/tests/test_properties.py index def20c1c..df503004 100644 --- a/device-detection/fiftyone_devicedetection_onpremise/tests/test_properties.py +++ b/device-detection/fiftyone_devicedetection_onpremise/tests/test_properties.py @@ -28,7 +28,7 @@ from fiftyone_devicedetection_shared.utils import * from fiftyone_devicedetection_onpremise import constants -data_file = "../../cxx/device-detection-data/51Degrees-LiteV4.1.hash" +data_file = "../../cxx/data/device-detection/51Degrees-LiteV4.1.hash" mobile_ua = ("Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) " "AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile" "/11D167 Safari/9537.53") @@ -42,6 +42,10 @@ .add(deviceDetectionOnPremiseEngine) \ .build() +# Releases the native engine once every test in the module has run. +def tearDownModule(): + pipeline.close() + class PropertyTests(unittest.TestCase): def test_available_properties(self): @@ -50,6 +54,7 @@ def test_available_properties(self): """ flowData = pipeline.create_flowdata() + self.addCleanup(flowData.close) flowData.evidence.add("header.user-agent", mobile_ua) flowData.process() device = flowData.device @@ -75,6 +80,7 @@ def test_match_metrics_description(self): """ flowData = pipeline.create_flowdata() + self.addCleanup(flowData.close) flowData.evidence.add("header.user-agent", mobile_ua) flowData.process() @@ -110,6 +116,7 @@ def test_value_types(self): """ flowData = pipeline.create_flowdata() + self.addCleanup(flowData.close) flowData.evidence.add("header.user-agent", mobile_ua) flowData.process() device = flowData.device diff --git a/device-detection/fiftyone_devicedetection_onpremise/tests/test_swig.py b/device-detection/fiftyone_devicedetection_onpremise/tests/test_swig.py index 04b4ee50..67879404 100644 --- a/device-detection/fiftyone_devicedetection_onpremise/tests/test_swig.py +++ b/device-detection/fiftyone_devicedetection_onpremise/tests/test_swig.py @@ -28,7 +28,7 @@ from fiftyone_devicedetection_onpremise.swig_data import SwigData from fiftyone_devicedetection_shared.utils import * -data_file = "../../cxx/device-detection-data/51Degrees-LiteV4.1.hash" +data_file = "../../cxx/data/device-detection/51Degrees-LiteV4.1.hash" # Create a simple pipeline to access the engine with and process it with flow data deviceDetectionOnPremiseEngine = DeviceDetectionOnPremise( @@ -39,6 +39,10 @@ .add(deviceDetectionOnPremiseEngine) \ .build() +# Releases the native engine once every test in the module has run. +def tearDownModule(): + pipeline.close() + class SwigTests(unittest.TestCase): # Mock the native code parts so we are only testing the wrapper. diff --git a/device-detection/fiftyone_devicedetection_onpremise/tox.ini b/device-detection/fiftyone_devicedetection_onpremise/tox.ini index 4600cd37..5cdeeca3 100644 --- a/device-detection/fiftyone_devicedetection_onpremise/tox.ini +++ b/device-detection/fiftyone_devicedetection_onpremise/tox.ini @@ -22,6 +22,12 @@ deps = -e../fiftyone_devicedetection_cloud/ -e../fiftyone_devicedetection_shared/ -e../fiftyone_devicedetection_examples/ + # The pipeline packages come from this checkout rather than a package + # index, so the tests run with the pipeline code they are released with. + -e../../pipeline/fiftyone_pipeline_core/ + -e../../pipeline/fiftyone_pipeline_engines/ + -e../../pipeline/fiftyone_pipeline_engines_fiftyone/ + -e../../pipeline/fiftyone_pipeline_cloudrequestengine/ commands = python setup.py build_clib build_ext {[common]commands} diff --git a/device-detection/fiftyone_devicedetection_shared/readme.md b/device-detection/fiftyone_devicedetection_shared/readme.md index bbff13b6..1a5deb11 100644 --- a/device-detection/fiftyone_devicedetection_shared/readme.md +++ b/device-detection/fiftyone_devicedetection_shared/readme.md @@ -1,8 +1,8 @@ # 51Degrees Device Detection Engines - Shared -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_shared-readme.md&utm_term=51degrees-device-detection-engines-shared "THE Fastest and Most Accurate Device Detection") **v4 Device Detection Python** +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_shared&utm_term=51degrees-device-detection-engines-shared "THE Fastest and Most Accurate Device Detection") **v4 Device Detection Python** -[Developer Documentation](https://51degrees.com/device-detection-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_shared-readme.md&utm_term=51degrees-device-detection-engines-shared "Developer Documentation") | [Available Properties](https://51degrees.com/resources/property-dictionary?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_shared-readme.md&utm_term=51degrees-device-detection-engines-shared "View all available properties and values") +[Developer Documentation](https://51degrees.com/device-detection-python/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_shared&utm_term=51degrees-device-detection-engines-shared "Developer Documentation") | [Available Properties](https://51degrees.com/resources/property-dictionary?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_shared&utm_term=51degrees-device-detection-engines-shared "View all available properties and values") ## Introduction @@ -23,3 +23,11 @@ The Pipeline is a generic web request intelligence and data processing solution To run the tests use: `python -m unittest discover -s tests -p test*.py -b` + +## Find out more + +- [Device Detection](https://51degrees.com/device-detection?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_shared&utm_term=find-out-more-device-detection) +- [Device Detection documentation](https://51degrees.com/documentation/_device_detection__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_shared&utm_term=find-out-more-documentation) +- [Getting started](https://51degrees.com/documentation/_device_detection__quickstart.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_shared&utm_term=find-out-more-getting-started) +- [Pricing](https://51degrees.com/pricing?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_shared&utm_term=find-out-more-pricing) +- https://github.com/51Degrees/51degrees-python diff --git a/device-detection/fiftyone_devicedetection_shared/setup.py b/device-detection/fiftyone_devicedetection_shared/setup.py index e367b7d1..07b89cdf 100644 --- a/device-detection/fiftyone_devicedetection_shared/setup.py +++ b/device-detection/fiftyone_devicedetection_shared/setup.py @@ -41,7 +41,11 @@ def read(file_name): version=read("version.txt"), author_email="engineering@51degrees.com", author="51Degrees Engineering", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_shared-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/device-detection?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_shared&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_device_detection__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_shared&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("The 51Degrees Pipeline API is a generic web request intelligence and data processing solution with the ability to add a range of 51Degrees and/or custom plug ins (Engines). This project contains 51Degrees Device Detection engines that can be used with the Pipeline API."), long_description=read("readme.md"), long_description_content_type='text/markdown', diff --git a/device-detection/readme.md b/device-detection/readme.md index 379b1ed1..809d47e4 100644 --- a/device-detection/readme.md +++ b/device-detection/readme.md @@ -25,7 +25,7 @@ git submodule update --init --recursive ## Dependencies For runtime dependencies, see our [dependencies](https://51degrees.com/documentation/_info__dependencies.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=dependencies) page. -The [tested versions](https://51degrees.com/documentation/_info__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=dependencies) page shows +The [tested versions](https://51degrees.com/documentation/_product_summaries__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=dependencies) page shows the Python versions that we currently test against. The software may run fine against other versions, but additional caution should be applied. @@ -35,30 +35,33 @@ The API can either use our cloud service to get its data or it can use a local ( #### Cloud -You will require a [resource key](https://51degrees.com/documentation/_info__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=cloud) +You will require a [resource key](https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=cloud) to use the Cloud API. You can create resource keys using our [configurator](https://configure.51degrees.com/?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=cloud), see our -[documentation](https://51degrees.com/documentation/_concepts__configurator.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=cloud) on how to use this. +[documentation](https://51degrees.com/documentation/_services__configurator.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=cloud) on how to use this. #### On-Premise In order to perform device detection on-premise, you will need to use a 51Degrees data file. -This repository includes a free, 'lite' file in the 'device-detection-data' sub-module that has a -significantly reduced set of properties. To obtain a file with a more complete set of device -properties see the [51Degrees website](https://51degrees.com/pricing?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=on-premise). If you want to use the lite -file, you will need to install [GitLFS](https://git-lfs.github.com/): +A free, 'lite' file is available, with a significantly reduced set of properties. To obtain a +file with a more complete set of device properties see the [51Degrees website](https://51degrees.com/pricing?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=on-premise). + +The lite file is not stored in a repository. The [data](https://github.com/51Degrees/data) +repository, which is the `cxx/data` submodule, links to it from +`cxx/data/device-detection`. Download it from the root of this repository by running: ``` -sudo apt-get install git-lfs -git lfs install +sh cxx/data/get-data-files.sh 51Degrees-LiteV4.1.hash ``` -Then, navigate to 'src/fiftyone_devicedetection_onpremise/cxx/device-detection-data' and execute: +or, in PowerShell: ``` -git lfs pull +./cxx/data/get-data-files.ps1 51Degrees-LiteV4.1.hash ``` +The examples run this for themselves when the file is missing. + ## Folders * `fiftyone_devicedetection` - references both cloud and on-premise packages, contains generic Device-Detection Pipeline Builder. diff --git a/ip-intelligence/ci/README.md b/ip-intelligence/ci/README.md index a5089246..7f4b5fa1 100644 --- a/ip-intelligence/ci/README.md +++ b/ip-intelligence/ci/README.md @@ -6,7 +6,7 @@ The secrets each workflow passes are named in [.github/workflows](../.github/wor The following secrets are required: * `ACCESS_TOKEN` - GitHub [access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#about-personal-access-tokens) for cloning repos, creating PRs, etc. * Example: `github_pat_l0ng_r4nd0m_s7r1ng` -* `_51DEGREES_RESOURCE_KEY_BESPOKE` - [resource key](https://51degrees.com/documentation/_info__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=ci-readme.md&utm_term=api-specific-ci-cd-approach) for the cloud tests. It must include the IP Intelligence (`ip`) product, or every cloud test fails as if the service were broken. +* `_51DEGREES_RESOURCE_KEY_BESPOKE` - [resource key](https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=ci-readme.md&utm_term=api-specific-ci-cd-approach) for the cloud tests. It must include the IP Intelligence (`ip`) product, or every cloud test fails as if the service were broken. * Example: `R4nd0m-S7r1ng` * `PYPI_TOKEN` - [PyPI token](https://pypi.org/help/#apitoken) for publishing packages * Example: `pypi-sUp3r-l0nG_r4nd0m-s7r1Ng` diff --git a/ip-intelligence/ci/fetch-assets.ps1 b/ip-intelligence/ci/fetch-assets.ps1 index 00e2bd91..a26332bc 100644 --- a/ip-intelligence/ci/fetch-assets.ps1 +++ b/ip-intelligence/ci/fetch-assets.ps1 @@ -29,6 +29,14 @@ foreach ($asset in $assets) { } } +# The ASN file is not an asset, so the data repository's script downloads it +# beside its link in cxx/data/ip-intelligence, where the tests look for it. +$global:LASTEXITCODE = 0 +& "$PSScriptRoot/../../cxx/data/get-data-files.ps1" 51Degrees-IPIV4AsnIpiV41.ipi +if ($LASTEXITCODE -ne 0) { + throw "The ASN data file was not downloaded" +} + # If the Enterprise file was downloaded, prefer it for tests by exposing # it through the environment variable used by the test suites. if ($IpIntelligenceUrl) { diff --git a/ip-intelligence/fiftyone_ipintelligence/examples/gettingstarted.py b/ip-intelligence/fiftyone_ipintelligence/examples/gettingstarted.py index c7ea9615..a2511324 100644 --- a/ip-intelligence/fiftyone_ipintelligence/examples/gettingstarted.py +++ b/ip-intelligence/fiftyone_ipintelligence/examples/gettingstarted.py @@ -35,45 +35,48 @@ # Cloud #pipeline = IpIntelligencePipelineBuilder(resource_key = resource_key).build() -# We create a FlowData object from the pipeline -# this is used to add evidence to and then process +# The pipeline is closed at the end of the with block, which releases the +# native data it holds. +with pipeline: + # We create a FlowData object from the pipeline + # this is used to add evidence to and then process. It is closed at the + # end of its with block, which releases the results of the lookup. + with pipeline.create_flowdata() as flowdata1: -flowdata1 = pipeline.create_flowdata() + # Here we add an IPv4 address as evidence -# Here we add an IPv4 address as evidence + ipv4_address = "185.28.167.78" -ipv4_address = "185.28.167.78" + flowdata1.evidence.add("query.client-ip", ipv4_address) -flowdata1.evidence.add("query.client-ip", ipv4_address) + # Now we process the FlowData -# Now we process the FlowData + flowdata1.process() -flowdata1.process() + # To find the name the IP range is registered under we look at the + # registeredname property inside the IP Intelligence Engine -# To find the name the IP range is registered under we look at the -# registeredname property inside the IP Intelligence Engine + # first we check if this has a meaningful result -# first we check if this has a meaningful result + print("What is IP address " + ipv4_address + " registered as? ") + if flowdata1.ip.registeredname.has_value(): + print(flowdata1.ip.registeredname.value()) + else: + # Output why the value isn't meaningful + print(flowdata1.ip.registeredname.no_value_message()) -print("What is IP address " + ipv4_address + " registered as? ") -if flowdata1.ip.registeredname.has_value(): - print(flowdata1.ip.registeredname.value()) -else: - # Output why the value isn't meaningful - print(flowdata1.ip.registeredname.no_value_message()) + # Now we do the same with a new IP address, this time an IPv6 one -# Now we do the same with a new IP address, this time an IPv6 one + with pipeline.create_flowdata() as flowdata2: -flowdata2 = pipeline.create_flowdata() + ipv6_address = "2001:4860:4860::8888" -ipv6_address = "2001:4860:4860::8888" + flowdata2.evidence.add("query.client-ip", ipv6_address) -flowdata2.evidence.add("query.client-ip", ipv6_address) + flowdata2.process() -flowdata2.process() - -print("What is IP address " + ipv6_address + " registered as? ") -if flowdata2.ip.registeredname.has_value(): - print(flowdata2.ip.registeredname.value()) -else: - print(flowdata2.ip.registeredname.no_value_message()) + print("What is IP address " + ipv6_address + " registered as? ") + if flowdata2.ip.registeredname.has_value(): + print(flowdata2.ip.registeredname.value()) + else: + print(flowdata2.ip.registeredname.no_value_message()) diff --git a/ip-intelligence/fiftyone_ipintelligence/readme.md b/ip-intelligence/fiftyone_ipintelligence/readme.md index da4cdcb3..1b1c9eec 100644 --- a/ip-intelligence/fiftyone_ipintelligence/readme.md +++ b/ip-intelligence/fiftyone_ipintelligence/readme.md @@ -1,13 +1,13 @@ # 51Degrees IP Intelligence Engines -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence-readme.md&utm_term=51degrees-ip-intelligence-engines "Data rewards the curious") **Python IP Intelligence** +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence&utm_term=51degrees-ip-intelligence-engines "Data rewards the curious") **Python IP Intelligence** -[Developer Documentation](https://51degrees.com/ip-intelligence-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence-readme.md&utm_term=51degrees-ip-intelligence-engines "developer documentation") +[Developer Documentation](https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence&utm_term=51degrees-ip-intelligence-engines "developer documentation") ## Introduction This package aggregates the cloud and on-premise 51Degrees IP intelligence -engines for the [51Degrees Pipeline API](https://51degrees.com/documentation/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence-readme.md&utm_term=introduction) +engines for the [51Degrees Pipeline API](https://51degrees.com/documentation/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence&utm_term=introduction) and provides the `IpIntelligencePipelineBuilder` which builds a pipeline with either engine depending on the settings provided: @@ -34,3 +34,11 @@ flowdata.evidence.add("query.client-ip", "185.28.167.78") flowdata.process() print(flowdata.ip.registeredname.value()) ``` + +## Find out more + +- [IP Intelligence](https://51degrees.com/ip-intelligence?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence&utm_term=find-out-more-ip-intelligence) +- [IP Intelligence documentation](https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence&utm_term=find-out-more-documentation) +- [Getting started](https://51degrees.com/documentation/_ip_intelligence__quickstart.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence&utm_term=find-out-more-getting-started) +- [Pricing](https://51degrees.com/pricing?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence&utm_term=find-out-more-pricing) +- https://github.com/51Degrees/51degrees-python diff --git a/ip-intelligence/fiftyone_ipintelligence/setup.py b/ip-intelligence/fiftyone_ipintelligence/setup.py index ef684aa9..a079441e 100644 --- a/ip-intelligence/fiftyone_ipintelligence/setup.py +++ b/ip-intelligence/fiftyone_ipintelligence/setup.py @@ -40,7 +40,11 @@ def read(file_name): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/ip-intelligence?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("IP Intelligence and IP geolocation for Python that finds an IP address's likely location (country, region, town, and coordinates), connection type, and network, on-premise with a data file or in the cloud. An alternative to MaxMind and IPinfo."), keywords=[ "51degrees", diff --git a/ip-intelligence/fiftyone_ipintelligence_cloud/readme.md b/ip-intelligence/fiftyone_ipintelligence_cloud/readme.md index dba487eb..83d568f9 100644 --- a/ip-intelligence/fiftyone_ipintelligence_cloud/readme.md +++ b/ip-intelligence/fiftyone_ipintelligence_cloud/readme.md @@ -1,17 +1,17 @@ # 51Degrees IP Intelligence Engines - Cloud -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud-readme.md&utm_term=51degrees-ip-intelligence-engines-cloud "Data rewards the curious") **Python IP Intelligence** +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud&utm_term=51degrees-ip-intelligence-engines-cloud "Data rewards the curious") **Python IP Intelligence** -[Developer Documentation](https://51degrees.com/ip-intelligence-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud-readme.md&utm_term=51degrees-ip-intelligence-engines-cloud "developer documentation") +[Developer Documentation](https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud&utm_term=51degrees-ip-intelligence-engines-cloud "developer documentation") ## Introduction This package uses the 51Degrees cloud service to resolve IP addresses to location, network and registration information using the -[51Degrees Pipeline API](https://51degrees.com/documentation/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud-readme.md&utm_term=introduction). +[51Degrees Pipeline API](https://51degrees.com/documentation/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud&utm_term=introduction). To use the cloud service you will need a resource key. You can obtain one -from the [51Degrees Configurator](https://configure.51degrees.com/?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud-readme.md&utm_term=introduction). +from the [51Degrees Configurator](https://configure.51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud&utm_term=introduction). ## Usage @@ -34,3 +34,11 @@ print(flowdata.ip.registeredname.value()) - `tests` - unit tests. These mock the cloud service so no resource key is required; set the `resource_key` environment variable to also run the live cloud integration test. + +## Find out more + +- [IP Intelligence](https://51degrees.com/ip-intelligence?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud&utm_term=find-out-more-ip-intelligence) +- [IP Intelligence documentation](https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud&utm_term=find-out-more-documentation) +- [Getting started](https://51degrees.com/documentation/_ip_intelligence__quickstart.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud&utm_term=find-out-more-getting-started) +- [Pricing](https://51degrees.com/pricing?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud&utm_term=find-out-more-pricing) +- https://github.com/51Degrees/51degrees-python diff --git a/ip-intelligence/fiftyone_ipintelligence_cloud/setup.py b/ip-intelligence/fiftyone_ipintelligence_cloud/setup.py index 8eca633e..a5cd5e2f 100644 --- a/ip-intelligence/fiftyone_ipintelligence_cloud/setup.py +++ b/ip-intelligence/fiftyone_ipintelligence_cloud/setup.py @@ -40,7 +40,11 @@ def read(file_name): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/ip-intelligence?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_cloud&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("IP Intelligence and IP geolocation for Python that finds an IP address's likely location (country, region, town, and coordinates), connection type, and network using the 51Degrees cloud service. An alternative to MaxMind and IPinfo."), keywords=[ "51degrees", diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/readme.md b/ip-intelligence/fiftyone_ipintelligence_examples/readme.md index af0db8c7..c41c0630 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/readme.md +++ b/ip-intelligence/fiftyone_ipintelligence_examples/readme.md @@ -2,7 +2,7 @@ ![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-readme.md&utm_term=51degrees-ip-intelligence-engines-examples "Data rewards the curious") **Python IP Intelligence** -[Developer Documentation](https://51degrees.com/ip-intelligence-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-readme.md&utm_term=51degrees-ip-intelligence-engines-examples "developer documentation") +[Developer Documentation](https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-readme.md&utm_term=51degrees-ip-intelligence-engines-examples "developer documentation") ## Introduction @@ -34,11 +34,12 @@ python -m fiftyone_ipintelligence_examples.cloud.gettingstarted_web ## On-premise examples Require a `.ipi` data file in the format the engine reads (see -[Data files](../readme.md#data-files)). The -`ip-intelligence-data` submodule includes a freely available network-only -file (`51Degrees-IPIV4AsnIpiV41.ipi`) that the examples fall back to; it -only contains the ASN/network properties, so location and registration -properties are reported as "not present in the data file". +[Data files](../readme.md#data-files)). The data repository, at +`cxx/data`, links to a freely available network-only file +(`51Degrees-IPIV4AsnIpiV41.ipi`) that the examples fall back to, and an +example downloads it when it is missing. It only contains the ASN/network +properties, so location and registration properties are reported as "not +present in the data file". For the full set of properties (location, registration, etc.), a more complete data file is required - to obtain one for testing please @@ -50,7 +51,7 @@ the first command line argument. - `onpremise/metadata_console.py` - list the properties available from the data file. - `onpremise/offlineprocessing.py` - batch process the evidence file - from the ip-intelligence-data repository. + from the data repository. - `onpremise/performance_console.py` - benchmark the number of IP lookups performed per second. - `onpremise/compare_console.py` - compare the location returned for each diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/setup.py b/ip-intelligence/fiftyone_ipintelligence_examples/setup.py index 0ae1c723..cc127a01 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/setup.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/setup.py @@ -40,7 +40,11 @@ def read(file_name): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/ip-intelligence?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("Examples for the 51Degrees IP intelligence engines."), long_description=read("readme.md"), long_description_content_type='text/markdown', @@ -50,6 +54,8 @@ def read(file_name): install_requires=[ "fiftyone_ipintelligence", "fiftyone_ipintelligence_translation", + # The first version with close(), which the examples use. + "fiftyone_pipeline_core>=4.5.42", "flask", "ruamel.yaml" ], diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/cloud/getallproperties_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/cloud/getallproperties_console.py index 0c4eee08..75c8c9d9 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/cloud/getallproperties_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/cloud/getallproperties_console.py @@ -105,7 +105,7 @@ def main(argv): f"'{ExampleUtils.RESOURCE_KEY_ENV_VAR}'. The 51Degrees cloud " "service is accessed using a 'ResourceKey'. For more information " "see " - "https://51degrees.com/documentation/_info__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-cloud-getallproperties_console.py&utm_term=resource-key-required. " + "https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-cloud-getallproperties_console.py&utm_term=resource-key-required. " "A resource key with the properties required by this example " "can be created for free at https://configure.51degrees.com/?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-cloud-getallproperties_console.py&utm_term=resource-key-required.") diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/cloud/gettingstarted_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/cloud/gettingstarted_console.py index 882a8b03..7118db82 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/cloud/gettingstarted_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/cloud/gettingstarted_console.py @@ -81,7 +81,7 @@ def run(self, resource_key, logger, output, cloud_endpoint = None): # In this example, we use the IpIntelligencePipelineBuilder and # configure it in code. For more information about pipelines in # general see the documentation at - # https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-cloud-gettingstarted_console.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-cloud-gettingstarted_console.py&utm_term=run settings = { # Obtain a resource key from https://configure.51degrees.com/?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-cloud-gettingstarted_console.py&utm_term=run "resource_key": resource_key @@ -172,7 +172,7 @@ def main(argv): f"'{ExampleUtils.RESOURCE_KEY_ENV_VAR}'. The 51Degrees cloud " "service is accessed using a 'ResourceKey'. For more " "information see " - "https://51degrees.com/documentation/_info__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-cloud-gettingstarted_console.py&utm_term=resource-key-required. " + "https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-cloud-gettingstarted_console.py&utm_term=resource-key-required. " "A resource key with the properties required by this example " "can be created for free at https://configure.51degrees.com/?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-cloud-gettingstarted_console.py&utm_term=resource-key-required. " "Once complete, populate the environment variable mentioned at " diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/cloud/metadata_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/cloud/metadata_console.py index c4d26b6e..60db3582 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/cloud/metadata_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/cloud/metadata_console.py @@ -104,7 +104,7 @@ def main(argv): f"'{ExampleUtils.RESOURCE_KEY_ENV_VAR}'. The 51Degrees cloud " "service is accessed using a 'ResourceKey'. For more information " "see " - "https://51degrees.com/documentation/_info__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-cloud-metadata_console.py&utm_term=resource-key-required. " + "https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-cloud-metadata_console.py&utm_term=resource-key-required. " "A resource key with the properties required by this example " "can be created for free at https://configure.51degrees.com/?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-cloud-metadata_console.py&utm_term=resource-key-required.") diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/example_utils.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/example_utils.py index 56e3d783..81eb5829 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/example_utils.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/example_utils.py @@ -22,6 +22,8 @@ from datetime import datetime, timedelta import os +import shlex +import subprocess from pathlib import Path @@ -43,6 +45,15 @@ class ExampleUtils: # to use when running examples. DATA_FILE_ENV_VAR = "IPINTELLIGENCEDATAFILE" + # The data repository's IP intelligence folder, from the root of a + # checkout of this repository. Each file the data repository links to + # belongs in this folder, beside its .link file. + DATA_FOLDER = os.path.join("cxx", "data", "ip-intelligence") + + # Sent with each download, so the downloads these examples make can be + # counted. + DOWNLOAD_SOURCE = "examples-51degrees-python" + @staticmethod def get_human_readable(ip_data, property): try: @@ -112,8 +123,9 @@ def find_file(file_name): """! Find the specified file. First checks the data file environment variable, then searches up the directory tree from the current - working directory looking in 'assets' and 'ip-intelligence-data' - folders. + working directory looking in 'assets' and the data repository's + folder. A file the data repository links to is downloaded there + when it is missing. """ env = os.environ.get(ExampleUtils.DATA_FILE_ENV_VAR) @@ -123,15 +135,54 @@ def find_file(file_name): current = Path(os.getcwd()).resolve() roots = [current, *current.parents] for root in roots[:8]: - for sub_dir in [ - ".", - "assets", - "ip-intelligence-data", - os.path.join("cxx", "ip-intelligence-data")]: + for sub_dir in [".", "assets", ExampleUtils.DATA_FOLDER]: candidate = os.path.join(str(root), sub_dir, file_name) if os.path.isfile(candidate): return candidate + for root in roots[:8]: + candidate = os.path.join( + str(root), ExampleUtils.DATA_FOLDER, file_name) + if os.path.isfile(candidate + ".link"): + return ExampleUtils.__download(candidate) + + return None + + @staticmethod + def __download(path): + + """! + Download a file the data repository links to, by running the data + repository's own script for that file, and return its path, or None + when it could not be downloaded. The data repository is the folder + above the one that holds the file's .link. + """ + + name = os.path.basename(path) + data = os.path.dirname(os.path.dirname(path)) + if os.name == "nt": + command = [ + "powershell", "-NoProfile", "-ExecutionPolicy", "Bypass", + "-File", os.path.join(data, "get-data-files.ps1"), name, + "-Source", ExampleUtils.DOWNLOAD_SOURCE] + text = subprocess.list2cmdline(command) + else: + command = [ + "sh", os.path.join(data, "get-data-files.sh"), + "-s", ExampleUtils.DOWNLOAD_SOURCE, name] + text = shlex.join(command) + + # Flushed first, because the script writes to the same output. + print(f"{name} is missing, so it is being downloaded by running\n" + f"{text}", flush=True) + try: + subprocess.run(command, check=False) + except OSError: + pass + if os.path.isfile(path): + return path + print(f"{name} could not be downloaded. Run this command to try " + f"again.\n{text}") return None @staticmethod @@ -167,9 +218,10 @@ def check_data_file(pipeline, logger): f"This data file is more than " f"{ExampleUtils.DATA_FILE_AGE_WARNING} days old. A more " f"recent data file may be needed to correctly resolve " - f"the latest IP addresses. The latest data file is " - f"available from the ip-intelligence-data repository on " - f"GitHub https://github.com/51Degrees/ip-intelligence-data.") + f"the latest IP addresses. Get the latest free data files " + f"by running cxx/data/get-data-files.sh -u ip-intelligence, " + f"or cxx/data/get-data-files.ps1 ip-intelligence -Update " + f"on Windows.") if product == "Lite": logger.log("warning", diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/compare_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/compare_console.py index d0479c2e..e418f786 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/compare_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/compare_console.py @@ -30,8 +30,8 @@ # an IP-to-location service compare to real world information, especially # when evaluating different solutions. # -# The input CSV (geoip_comparison_evidence.csv in the ip-intelligence-data -# repository) provides these fields: +# The input CSV (geoip_comparison_evidence.csv in the data repository's +# ip-intelligence folder) provides these fields: # DateTimeUtc, Latitude, Longitude, Ip, AddressFamily, Continent, Country # # For each record the engine is used to obtain the latitude and longitude @@ -56,7 +56,7 @@ from fiftyone_pipeline_core.logger import Logger from fiftyone_ipintelligence_shared.example_constants import ENTERPRISE_DATAFILE_NAME -# Name of the truth CSV file in the ip-intelligence-data repository. +# Name of the truth CSV file in the data repository. GEOIP_COMPARISON_EVIDENCE_FILE_NAME = "geoip_comparison_evidence.csv" # Mean radius of the earth in kilometres, used for the distance calculation. @@ -96,65 +96,70 @@ def run(self, data_file, truth_file, logger, output, record_limit=RECORD_LIMIT): auto_update = False, licence_keys = "").add_logger(logger).build() - ExampleUtils.check_data_file(pipeline, logger) - - writer = csv.DictWriter(output, fieldnames=OUTPUT_FIELDS) - writer.writeheader() - - processed = 0 - found = 0 - with open(truth_file, "r", newline="", encoding="utf-8") as fin: - for truth in csv.DictReader(fin): - if processed >= record_limit: - break - processed += 1 - result = self.compare(pipeline, truth) - if result["Found"]: - found += 1 - writer.writerow(result) - - logger.log("info", - f"Compared {processed} records; {found} resolved to a location") + # The pipeline is closed at the end of the with block, which + # releases the native data it holds. + with pipeline: + ExampleUtils.check_data_file(pipeline, logger) + + writer = csv.DictWriter(output, fieldnames=OUTPUT_FIELDS) + writer.writeheader() + + processed = 0 + found = 0 + with open(truth_file, "r", newline="", encoding="utf-8") as fin: + for truth in csv.DictReader(fin): + if processed >= record_limit: + break + processed += 1 + result = self.compare(pipeline, truth) + if result["Found"]: + found += 1 + writer.writerow(result) + + logger.log("info", + f"Compared {processed} records; {found} resolved to a location") def compare(self, pipeline, truth): - data = pipeline.create_flowdata() - data.evidence.add_from_dict({"query.client-ip": truth["Ip"]}) - data.process() - ip_data = data.ip - - row = { - "DateTimeUtc": truth.get("DateTimeUtc", ""), - "Ip": truth.get("Ip", ""), - "AddressFamily": truth.get("AddressFamily", ""), - "TruthLatitude": truth.get("Latitude", ""), - "TruthLongitude": truth.get("Longitude", ""), - "TruthCountry": truth.get("Country", ""), - "ResultLatitude": "", - "ResultLongitude": "", - "ResultCountry": ExampleUtils.get_human_readable(ip_data, "country"), - "Confidence": ExampleUtils.get_human_readable(ip_data, "locationconfidence"), - "Found": False, - "DistanceKms": "", - } - - latitude = ExampleUtils.get_value(ip_data, "latitude") - longitude = ExampleUtils.get_value(ip_data, "longitude") - if latitude is not None and latitude.has_value() and \ - longitude is not None and longitude.has_value(): - row["ResultLatitude"] = latitude.value() - row["ResultLongitude"] = longitude.value() - row["Found"] = True - try: - row["DistanceKms"] = round(haversine_km( - float(truth["Latitude"]), float(truth["Longitude"]), - float(latitude.value()), float(longitude.value())), 3) - except (ValueError, TypeError): - # The truth record did not provide a usable latitude or - # longitude, so the distance cannot be calculated. - row["DistanceKms"] = "" - - return row + # The FlowData is closed when the method returns, which releases the + # results of the lookup straight away. + with pipeline.create_flowdata() as data: + data.evidence.add_from_dict({"query.client-ip": truth["Ip"]}) + data.process() + ip_data = data.ip + + row = { + "DateTimeUtc": truth.get("DateTimeUtc", ""), + "Ip": truth.get("Ip", ""), + "AddressFamily": truth.get("AddressFamily", ""), + "TruthLatitude": truth.get("Latitude", ""), + "TruthLongitude": truth.get("Longitude", ""), + "TruthCountry": truth.get("Country", ""), + "ResultLatitude": "", + "ResultLongitude": "", + "ResultCountry": ExampleUtils.get_human_readable(ip_data, "country"), + "Confidence": ExampleUtils.get_human_readable(ip_data, "locationconfidence"), + "Found": False, + "DistanceKms": "", + } + + latitude = ExampleUtils.get_value(ip_data, "latitude") + longitude = ExampleUtils.get_value(ip_data, "longitude") + if latitude is not None and latitude.has_value() and \ + longitude is not None and longitude.has_value(): + row["ResultLatitude"] = latitude.value() + row["ResultLongitude"] = longitude.value() + row["Found"] = True + try: + row["DistanceKms"] = round(haversine_km( + float(truth["Latitude"]), float(truth["Longitude"]), + float(latitude.value()), float(longitude.value())), 3) + except (ValueError, TypeError): + # The truth record did not provide a usable latitude or + # longitude, so the distance cannot be calculated. + row["DistanceKms"] = "" + + return row def main(argv): @@ -184,8 +189,8 @@ def main(argv): else: logger.log("error", "Failed to find a data file or truth file. Make sure the " - "ip-intelligence-data submodule has been updated by running " - "`git submodule update --recursive`. To obtain an enterprise " + "data submodule is there, by running " + "`git submodule update --init --recursive`. To obtain an enterprise " "data file for testing, please contact us: " "https://51degrees.com/contact-us?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-onpremise-compare_console.py&utm_term=data-file-required") diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/countries_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/countries_console.py index 35266657..eb239813 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/countries_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/countries_console.py @@ -71,42 +71,47 @@ def run(self, data_file, logger, output): add_country_translation(builder) pipeline = builder.add_logger(logger).build() - ExampleUtils.check_data_file(pipeline, logger) + # The pipeline is closed at the end of the with block, which + # releases the native data it holds. + with pipeline: + ExampleUtils.check_data_file(pipeline, logger) - for values in EVIDENCE_VALUES: - self.analyseEvidence(values, pipeline, output) + for values in EVIDENCE_VALUES: + self.analyseEvidence(values, pipeline, output) def analyseEvidence(self, evidence, pipeline, output): - data = pipeline.create_flowdata() - - message = [] - message.append("Input values:\n") - for key in evidence: - message.append(f"\t{key}: {evidence[key]}\n") - if TRANSLATION: - message.append(f"\tquery.translation: {TRANSLATION}\n") - output("".join(message)) - - # Add the IP address evidence, and the language to translate to. - data.evidence.add_from_dict(evidence) - if TRANSLATION: - data.evidence.add("query.translation", TRANSLATION) - - data.process() - - translated = data.get("countrynamestranslated") - - message = [] - message.append("Results:\n") - self.outputList( - translated, "CountryCodesGeographicalAll", - "Geographical countries (most probable first)", message) - self.outputList( - translated, "CountryCodesPopulationAll", - "Population countries (most probable first)", message) - self.outputDropdown(translated, message) - output("".join(message)) + # The FlowData is closed at the end of the with block, which releases + # the results of the lookup straight away. + with pipeline.create_flowdata() as data: + + message = [] + message.append("Input values:\n") + for key in evidence: + message.append(f"\t{key}: {evidence[key]}\n") + if TRANSLATION: + message.append(f"\tquery.translation: {TRANSLATION}\n") + output("".join(message)) + + # Add the IP address evidence, and the language to translate to. + data.evidence.add_from_dict(evidence) + if TRANSLATION: + data.evidence.add("query.translation", TRANSLATION) + + data.process() + + translated = data.get("countrynamestranslated") + + message = [] + message.append("Results:\n") + self.outputList( + translated, "CountryCodesGeographicalAll", + "Geographical countries (most probable first)", message) + self.outputList( + translated, "CountryCodesPopulationAll", + "Population countries (most probable first)", message) + self.outputDropdown(translated, message) + output("".join(message)) def outputList(self, translated, property_name, label, message): value = translated.get(property_name) @@ -150,8 +155,9 @@ def main(argv): else: logger.log("error", "Failed to find an IP intelligence data file. Make sure the " - "ip-intelligence-data submodule has been updated by running " - "`git submodule update --recursive`, or supply the path to a " + "data submodule is there, by running " + "`git submodule update --init --recursive`, and run this " + "example again to download the file, or supply the path to a " "data file as the first command line argument.") diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/datafileupdate_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/datafileupdate_console.py index 1236a3b1..37200afd 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/datafileupdate_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/datafileupdate_console.py @@ -155,7 +155,7 @@ def run(self, data_file, license_key, interactive, logger, output): update_service.on_complete(lambda status, file: update_event.set(status)) # Build the pipeline, configuring it for automatic updates. - IpIntelligencePipelineBuilder( + pipeline = IpIntelligencePipelineBuilder( # The file need not exist when using update on start-up, but its # directory must. Any file present is overwritten, so the engine # is configured to use a temporary copy. @@ -175,35 +175,39 @@ def run(self, data_file, license_key, interactive, logger, output): data_update_product_type = DATA_UPDATE_PRODUCT_TYPE, usage_sharing = False).add_logger(logger).build() - # Block until the start-up update check is complete. - update_event.wait() - output(f"Update on start-up complete - status - {update_event.status}") - - if update_event.status == UpdateStatus.AUTO_UPDATE_SUCCESS: - output("Modifying downloaded file to trigger reload - please wait " - "for that to complete") - update_event.clear() - - # Touching the file metadata triggers the filesystem watcher, - # demonstrating reload when a new file is dropped into place. - now = datetime.now().timestamp() - try: - os.utime(data_file, (now, now)) - except OSError: - raise Exception("Could not modify file time, abandoning example") - - if update_event.wait(120): - output(f"Update on file modification complete, status: " - f"{update_event.status}") + # The pipeline is closed at the end of the with block, which stops the + # automatic updates and the file system watcher, and releases the + # native data it holds. + with pipeline: + # Block until the start-up update check is complete. + update_event.wait() + output(f"Update on start-up complete - status - {update_event.status}") + + if update_event.status == UpdateStatus.AUTO_UPDATE_SUCCESS: + output("Modifying downloaded file to trigger reload - please wait " + "for that to complete") + update_event.clear() + + # Touching the file metadata triggers the filesystem watcher, + # demonstrating reload when a new file is dropped into place. + now = datetime.now().timestamp() + try: + os.utime(data_file, (now, now)) + except OSError: + raise Exception("Could not modify file time, abandoning example") + + if update_event.wait(120): + output(f"Update on file modification complete, status: " + f"{update_event.status}") + else: + output("Update on file modification timed out") else: - output("Update on file modification timed out") - else: - logger.log("error", "Auto update was not successful, abandoning example") - error_message = f"Auto update failed: {update_event.status}" - if update_event.status == UpdateStatus.AUTO_UPDATE_ERR_429_TOO_MANY_ATTEMPTS: - output(error_message) - else: - raise Exception(error_message) + logger.log("error", "Auto update was not successful, abandoning example") + error_message = f"Auto update failed: {update_event.status}" + if update_event.status == UpdateStatus.AUTO_UPDATE_ERR_429_TOO_MANY_ATTEMPTS: + output(error_message) + else: + raise Exception(error_message) output("Finished Example") diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_console.py index 0f211919..4988da7d 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_console.py @@ -32,10 +32,10 @@ # 2. How to pass input data (evidence) to the Pipeline # 3. How to retrieve the results # -# This example uses a local IP intelligence data file (.ipi). The -# ip-intelligence-data submodule provides a freely available data file; -# a more complete data file can be supplied as the first command line -# argument. +# This example uses a local IP intelligence data file (.ipi). The data +# repository links to a freely available data file, which the example +# downloads when it is missing. A more complete data file can be supplied +# as the first command line argument. # # Required PyPi Dependencies: # - fiftyone_ipintelligence @@ -75,14 +75,14 @@ def run(self, data_file, logger, output): # In this example, we use the IpIntelligencePipelineBuilder # and configure it in code. For more information about # pipelines in general see the documentation at - # https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-onpremise-gettingstarted_console.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-onpremise-gettingstarted_console.py&utm_term=run pipeline = IpIntelligencePipelineBuilder( data_file_path = data_file, # We use the low memory profile as its performance is # sufficient for this example. See the documentation for # more detail on this and other configuration options: - # https://51degrees.com/documentation/_features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-onpremise-gettingstarted_console.py&utm_term=run - # https://51degrees.com/documentation/_features__usage_sharing.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-onpremise-gettingstarted_console.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-onpremise-gettingstarted_console.py&utm_term=run + # https://51degrees.com/documentation/_pipeline_api__features__usage_sharing.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-onpremise-gettingstarted_console.py&utm_term=run performance_profile = "LowMemory", # inhibit sharing usage for this example, usually this # should be set "true" @@ -91,11 +91,16 @@ def run(self, data_file, logger, output): auto_update = False, licence_keys = "").add_logger(logger).build() - ExampleUtils.check_data_file(pipeline, logger) + # The pipeline is closed at the end of the with block, which + # releases the native data it holds. An application normally builds + # one pipeline when it starts, uses it for every lookup, and closes + # it when it stops. + with pipeline: + ExampleUtils.check_data_file(pipeline, logger) - # carry out some sample lookups - for values in EVIDENCE_VALUES: - self.analyseEvidence(values, pipeline, output) + # carry out some sample lookups + for values in EVIDENCE_VALUES: + self.analyseEvidence(values, pipeline, output) def analyseEvidence(self, evidence, pipeline, output): @@ -104,38 +109,41 @@ def analyseEvidence(self, evidence, pipeline, output): # through the pipeline. # Information required is called "evidence", in this case an # IP address supplied under the "query.client-ip" key. - data = pipeline.create_flowdata() + # Closing the FlowData, which the with statement does at the end of + # the block, releases the results of the lookup straight away, + # rather than when the garbage collector next runs. + with pipeline.create_flowdata() as data: - message = [] + message = [] - # List the evidence - message.append("Input values:\n") - for key in evidence: - message.append(f"\t{key}: {evidence[key]}\n") + # List the evidence + message.append("Input values:\n") + for key in evidence: + message.append(f"\t{key}: {evidence[key]}\n") - output("".join(message)) + output("".join(message)) - # Add the evidence values to the flow data - data.evidence.add_from_dict(evidence) + # Add the evidence values to the flow data + data.evidence.add_from_dict(evidence) - # Process the flow data. - data.process() + # Process the flow data. + data.process() - message = [] - message.append("Results:\n") + message = [] + message.append("Results:\n") - # Now that it's been processed, the flow data will have - # been populated with the result. In this case, we want - # information about the IP address, which we can get by - # asking for a result matching the key "ip" - ip_data = data.ip + # Now that it's been processed, the flow data will have + # been populated with the result. In this case, we want + # information about the IP address, which we can get by + # asking for a result matching the key "ip" + ip_data = data.ip - # Display the results of the lookup. See the property - # dictionary at https://51degrees.com/developers/property-dictionary?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-onpremise-gettingstarted_console.py&utm_term=analyseevidence - # for details of all available properties. - for name in OUTPUT_PROPERTIES: - self.outputValue(ip_data, name, message) - output("".join(message)) + # Display the results of the lookup. See the property + # dictionary at https://51degrees.com/developers/property-dictionary?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-onpremise-gettingstarted_console.py&utm_term=analyseevidence + # for details of all available properties. + for name in OUTPUT_PROPERTIES: + self.outputValue(ip_data, name, message) + output("".join(message)) def outputValue(self, ip_data, name, message): # Individual result values have a wrapper called @@ -170,8 +178,9 @@ def main(argv): else: logger.log("error", "Failed to find an IP intelligence data file. Make sure the " - "ip-intelligence-data submodule has been updated by running " - "`git submodule update --recursive`, or supply the path to a " + "data submodule is there, by running " + "`git submodule update --init --recursive`, and run this " + "example again to download the file, or supply the path to a " "data file as the first command line argument.") if __name__ == "__main__": diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_web/app.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_web/app.py index 49bc8b72..56ea9799 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_web/app.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_web/app.py @@ -88,44 +88,46 @@ def country_dropdown(translated): @app.route("/") def index(): - # Create the flowdata object for this request. - flowdata = pipeline.create_flowdata() - - # Use the IP address supplied as a query parameter, or fall back - # to the address of the requester. - client_ip = request.args.get("client-ip", request.remote_addr) - flowdata.evidence.add("query.client-ip", client_ip) - - # Supply the browser language so the country names are localized. The - # Accept-Language header is used by default, overridable with a - # "translation" query parameter. - accept_language = request.headers.get("Accept-Language") - if accept_language: - flowdata.evidence.add("header.accept-language", accept_language) - translation = request.args.get("translation") - if translation: - flowdata.evidence.add("query.translation", translation) - - # Process the flowdata - flowdata.process() - - ip_data = flowdata.ip - translated = flowdata.get("countrynamestranslated") - - return f""" -

51Degrees IP Intelligence - Getting Started Web (On-Premise)

-

IP address: {client_ip}

-
    -
  • Registered Name: {get_value(ip_data, "registeredname")}
    • -
  • Registered Owner: {get_value(ip_data, "registeredowner")}
    • -
  • Registered Country: {get_value(ip_data, "registeredcountry")}
    • -
-

Country

-

Most probable country first, then every other country alphabetically.

- {country_dropdown(translated)} -

Add ?client-ip=x.x.x.x to the url to lookup a different address, and - &translation=fr_FR to choose a language.

- """ + # Create the flowdata object for this request. It is closed when the + # route returns, once the page has been built from it, which releases + # the results of the lookup straight away. + with pipeline.create_flowdata() as flowdata: + + # Use the IP address supplied as a query parameter, or fall back + # to the address of the requester. + client_ip = request.args.get("client-ip", request.remote_addr) + flowdata.evidence.add("query.client-ip", client_ip) + + # Supply the browser language so the country names are localized. The + # Accept-Language header is used by default, overridable with a + # "translation" query parameter. + accept_language = request.headers.get("Accept-Language") + if accept_language: + flowdata.evidence.add("header.accept-language", accept_language) + translation = request.args.get("translation") + if translation: + flowdata.evidence.add("query.translation", translation) + + # Process the flowdata + flowdata.process() + + ip_data = flowdata.ip + translated = flowdata.get("countrynamestranslated") + + return f""" +

51Degrees IP Intelligence - Getting Started Web (On-Premise)

+

IP address: {client_ip}

+
    +
  • Registered Name: {get_value(ip_data, "registeredname")}
    • +
  • Registered Owner: {get_value(ip_data, "registeredowner")}
    • +
  • Registered Country: {get_value(ip_data, "registeredcountry")}
    • +
+

Country

+

Most probable country first, then every other country alphabetically.

+ {country_dropdown(translated)} +

Add ?client-ip=x.x.x.x to the url to lookup a different address, and + &translation=fr_FR to choose a language.

+ """ def main(data_file = None): @@ -133,11 +135,15 @@ def main(data_file = None): data_file = ExampleUtils.find_file(LITE_DATAFILE_NAME) if not data_file: raise Exception( - "Failed to find an IP intelligence data file. Download the " - "Lite file from https://51ddatafiles.blob.core.windows.net/" - "enterpriseipi/51Degrees-LiteIpiV41.ipi.gz") - build_pipeline(data_file) - app.run(debug=False) + "Failed to find an IP intelligence data file. Make sure the " + "data submodule is there, by running " + "`git submodule update --init --recursive`, and run this " + "example again to download the file, or supply the path to a " + "data file as the first command line argument.") + # The pipeline lives as long as the app, and is closed when the server + # stops, which releases the native data it holds. + with build_pipeline(data_file): + app.run(debug=False) if __name__ == "__main__": diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/metadata_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/metadata_console.py index e10df504..3819776c 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/metadata_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/metadata_console.py @@ -47,14 +47,17 @@ def run(self, data_file, logger, output): auto_update = False, licence_keys = "").add_logger(logger).build() - engine = pipeline.flow_elements_list["ip"] + # The pipeline is closed at the end of the with block, which + # releases the native data it holds. + with pipeline: + engine = pipeline.flow_elements_list["ip"] - output("IP Intelligence properties:") - for property_metadata in engine.properties.values(): - output(f" {property_metadata['name']} ({property_metadata['type']}) - " - f"category: {property_metadata['category']}") - if property_metadata.get("description"): - output(f" {property_metadata['description']}") + output("IP Intelligence properties:") + for property_metadata in engine.properties.values(): + output(f" {property_metadata['name']} ({property_metadata['type']}) - " + f"category: {property_metadata['category']}") + if property_metadata.get("description"): + output(f" {property_metadata['description']}") def main(argv): @@ -66,9 +69,8 @@ def main(argv): MetadataConsole().run(data_file, logger, print) else: logger.log("error", - "Failed to find an IP intelligence data file. Make sure the " - "ip-intelligence-data submodule has been updated by running " - "`git submodule update --recursive`. To obtain an enterprise " + "Failed to find an IP intelligence data file. Supply its path " + "as the first command line argument. To obtain an enterprise " "data file for testing, please contact us: " "https://51degrees.com/contact-us?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-onpremise-metadata_console.py&utm_term=data-file-required") diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/mixed_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/mixed_console.py index 11805592..2f4f5e18 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/mixed_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/mixed_console.py @@ -48,8 +48,8 @@ from fiftyone_ipintelligence_examples.example_utils import ExampleUtils from fiftyone_ipintelligence_shared.example_constants import ASN_DATAFILE_NAME -# Default name of the device detection data file (the lite file included in -# the device-detection-data repository). +# Default name of the device detection data file (the lite file the data +# repository links to). HASH_DATAFILE_NAME = "51Degrees-LiteV4.1.hash" # Combined evidence: a User-Agent for device detection and an IP address for @@ -140,38 +140,43 @@ def run(self, device_file, ip_file, logger, output): .add_logger(logger) \ .build() - for evidence in COMBINED_EVIDENCE: - self.analyse_evidence(evidence, pipeline, output) + # The pipeline is closed at the end of the with block, which closes + # both engines and releases the native data they hold. + with pipeline: + for evidence in COMBINED_EVIDENCE: + self.analyse_evidence(evidence, pipeline, output) def analyse_evidence(self, evidence, pipeline, output): - data = pipeline.create_flowdata() + # The FlowData is closed at the end of the with block, which releases + # the results of both engines straight away. + with pipeline.create_flowdata() as data: - message = [] - message.append("=" * 70 + "\n") - message.append("Input values:\n") - for key in evidence: - message.append(f"\t{key}: {evidence[key]}\n") - output("".join(message)) + message = [] + message.append("=" * 70 + "\n") + message.append("Input values:\n") + for key in evidence: + message.append(f"\t{key}: {evidence[key]}\n") + output("".join(message)) - # Add the evidence values to the flow data and process it. Both - # engines run against the same flow data. - data.evidence.add_from_dict(evidence) - data.process() + # Add the evidence values to the flow data and process it. Both + # engines run against the same flow data. + data.evidence.add_from_dict(evidence) + data.process() - message = [] + message = [] - # Device detection results, keyed under "device". - message.append("Device Detection Results:\n") - for name in DEVICE_PROPERTIES: - self.outputValue(data.device, name, message) + # Device detection results, keyed under "device". + message.append("Device Detection Results:\n") + for name in DEVICE_PROPERTIES: + self.outputValue(data.device, name, message) - # IP intelligence results, keyed under "ip". - message.append("\nIP Intelligence Results:\n") - for name in IP_PROPERTIES: - self.outputValue(data.ip, name, message) + # IP intelligence results, keyed under "ip". + message.append("\nIP Intelligence Results:\n") + for name in IP_PROPERTIES: + self.outputValue(data.ip, name, message) - output("".join(message)) + output("".join(message)) def outputValue(self, aspect_data, name, message): # `get_value` returns None when the property is not present in the @@ -199,14 +204,19 @@ def main(argv): else: if not device_file: logger.log("error", - "Failed to find a device detection data file. Make sure the " - "device-detection-data submodule has been updated, or supply " - "the path as the first command line argument.") + "Failed to find a device detection data file. Supply its " + "path as the first command line argument. Get the free Lite " + "file by running " + "cxx/data/get-data-files.sh 51Degrees-LiteV4.1.hash, or " + "cxx/data/get-data-files.ps1 51Degrees-LiteV4.1.hash on " + "Windows, which saves it in cxx/data/device-detection.") if not ip_file: logger.log("error", "Failed to find an IP intelligence data file. Make sure the " - "ip-intelligence-data submodule has been updated, or supply " - "the path as the second command line argument.") + "data submodule is there, by running " + "`git submodule update --init --recursive`, and run this " + "example again to download the file, or supply the path as " + "the second command line argument.") if __name__ == "__main__": main(sys.argv[1:]) diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/offlineprocessing.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/offlineprocessing.py index a32c11cb..4785249e 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/offlineprocessing.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/offlineprocessing.py @@ -23,8 +23,8 @@ ## @example onpremise/offlineprocessing.py # Provides an example of processing a YAML file containing evidence for # IP intelligence. There are 20,000 IP addresses in the supplied file of -# evidence records ('evidence.yml' in the ip-intelligence-data -# repository). +# evidence records ('evidence.yml' in the data repository's +# ip-intelligence folder). # # We create a pipeline using the on-premise engine, deserialize the # YAML evidence records, pass each entry to the pipeline and write the @@ -65,41 +65,47 @@ def run(self, data_file, evidence_yaml, logger, output): auto_update = False, licence_keys = "").add_logger(logger).build() - ExampleUtils.check_data_file(pipeline, logger) + # The pipeline is closed at the end of the with block, which + # releases the native data it holds. + with pipeline: + ExampleUtils.check_data_file(pipeline, logger) - records = 0 - with open(evidence_yaml, "r") as fin: - for evidence in yaml.load_all(fin): - if records >= RECORD_LIMIT: - break + records = 0 + with open(evidence_yaml, "r") as fin: + for evidence in yaml.load_all(fin): + if records >= RECORD_LIMIT: + break - # write the yaml document separator - output("---") - self.analyse_evidence(evidence, pipeline, output, yaml) - records += 1 + # write the yaml document separator + output("---") + self.analyse_evidence(evidence, pipeline, output, yaml) + records += 1 - # write the yaml document end marker - output("...") - logger.log("info", f"Processed {records} evidence records") + # write the yaml document end marker + output("...") + logger.log("info", f"Processed {records} evidence records") def analyse_evidence(self, evidence, pipeline, output, yaml): - data = pipeline.create_flowdata() + # The FlowData is closed at the end of the with block, once the + # values needed have been read, which releases the results of the + # lookup straight away. + with pipeline.create_flowdata() as data: - # Add the evidence values to the flow data. The YAML loader - # returns a dict subclass so convert to a plain dict first. - data.evidence.add_from_dict(dict(evidence)) + # Add the evidence values to the flow data. The YAML loader + # returns a dict subclass so convert to a plain dict first. + data.evidence.add_from_dict(dict(evidence)) - # Process the flow data. - data.process() + # Process the flow data. + data.process() - ip_data = data.ip + ip_data = data.ip - result = dict(evidence) + result = dict(evidence) - # Add the values that we want to store against the record. - result["ip.registeredname"] = ExampleUtils.get_human_readable(ip_data, "registeredname") - result["ip.registeredcountry"] = ExampleUtils.get_human_readable(ip_data, "registeredcountry") + # Add the values that we want to store against the record. + result["ip.registeredname"] = ExampleUtils.get_human_readable(ip_data, "registeredname") + result["ip.registeredcountry"] = ExampleUtils.get_human_readable(ip_data, "registeredcountry") buffer = io.BytesIO() yaml.dump(result, buffer) @@ -120,8 +126,8 @@ def main(argv): else: logger.log("error", "Failed to find a data file or evidence file. Make sure the " - "ip-intelligence-data submodule has been updated by running " - "`git submodule update --recursive`.") + "data submodule is there, by running " + "`git submodule update --init --recursive`.") if __name__ == "__main__": main(sys.argv[1:]) diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/performance_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/performance_console.py index 7ca71715..1d910d81 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/performance_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/performance_console.py @@ -25,7 +25,7 @@ # # This example measures the performance of the on-premise IP intelligence # engine. It processes the IP addresses from the evidence file supplied in -# the ip-intelligence-data repository and reports the number of IP lookups +# the data repository and reports the number of IP lookups # performed per second. # # Requesting fewer properties reduces processing time. If you do not specify @@ -84,11 +84,13 @@ def _build_pipeline(data_file, performance_profile, logger=None): def _process_record(pipeline, record): - data = pipeline.create_flowdata() - data.evidence.add_from_dict(record) - data.process() - value = ExampleUtils.get_value(data.ip, PERFORMANCE_PROPERTY) - return value is not None and value.has_value() + # The FlowData is closed when the function returns, which releases the + # results of the lookup straight away. + with pipeline.create_flowdata() as data: + data.evidence.add_from_dict(record) + data.process() + value = ExampleUtils.get_value(data.ip, PERFORMANCE_PROPERTY) + return value is not None and value.has_value() def _time_lookups(pipeline, records): @@ -115,8 +117,10 @@ def _worker(data_file, performance_profile, records, queue): count and elapsed seconds back to the parent. """ - pipeline = _build_pipeline(data_file, performance_profile) - elapsed = _time_lookups(pipeline, records) + # The pipeline is closed at the end of the with block, which releases + # the native data it holds. + with _build_pipeline(data_file, performance_profile) as pipeline: + elapsed = _time_lookups(pipeline, records) queue.put((len(records), elapsed)) @@ -156,9 +160,10 @@ def run(self, data_file, evidence, logger, output, workers=None, # wall time of the run. elapsed = max(elapsed for _records, elapsed in results) else: - pipeline = _build_pipeline(data_file, performance_profile, logger) - ExampleUtils.check_data_file(pipeline, logger) - elapsed = _time_lookups(pipeline, evidence) + with _build_pipeline( + data_file, performance_profile, logger) as pipeline: + ExampleUtils.check_data_file(pipeline, logger) + elapsed = _time_lookups(pipeline, evidence) per_second = count / elapsed if elapsed > 0 else 0 ms_per_lookup = (elapsed / count) * 1000 if count > 0 else 0 @@ -225,8 +230,8 @@ def main(argv): else: logger.log("error", "Failed to find a data file or evidence file. Make sure the " - "ip-intelligence-data submodule has been updated by running " - "`git submodule update --recursive`. To obtain an enterprise " + "data submodule is there, by running " + "`git submodule update --init --recursive`. To obtain an enterprise " "data file for testing, please contact us: " "https://51degrees.com/contact-us?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-onpremise-performance_console.py&utm_term=data-file-required") diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/tests/conftest.py b/ip-intelligence/fiftyone_ipintelligence_examples/tests/conftest.py new file mode 100644 index 00000000..fc53a6d0 --- /dev/null +++ b/ip-intelligence/fiftyone_ipintelligence_examples/tests/conftest.py @@ -0,0 +1,119 @@ +# ********************************************************************* +# This Original Work is copyright of 51 Degrees Mobile Experts Limited. +# Copyright 2026 51 Degrees Mobile Experts Limited, Davidson House, +# Forbury Square, Reading, Berkshire, United Kingdom RG1 3EU. +# +# This Original Work is licensed under the European Union Public Licence +# (EUPL) v.1.2 and is subject to its terms as set out below. +# +# If a copy of the EUPL was not distributed with this file, You can obtain +# one at https://opensource.org/licenses/EUPL-1.2. +# +# The 'Compatible Licences' set out in the Appendix to the EUPL (as may be +# amended by the European Commission) shall be deemed incompatible for +# the purposes of the Work and the provisions of the compatibility +# clause in Article 5 of the EUPL shall not apply. +# +# If using the Work as, or as part of, a network application, by +# including the attribution notice(s) required under Article 5 of the EUPL +# in the end user terms of the application under an appropriate heading, +# such notice(s) shall fulfill the requirements of that article. +# ********************************************************************* + +# Fails a test that leaves the native results or engine of an on-premise +# engine alive, because they hold the engine's data in memory. The same +# file is in each test folder that runs on-premise engines. +# +# Reference counting frees a native object as soon as nothing holds it. A +# FlowData refers to itself through its evidence, and a pipeline through +# its elements, so when they are not closed only the cycle collector frees +# what they hold. The collector is off while each test runs, so whatever +# the test leaves open is still alive when it ends. Objects that existed +# before the test, such as a pipeline built in setUpClass, are checked once +# every test has run. + +import gc +import importlib + +import pytest + +# The SWIG modules of the on-premise engines, whichever are installed. +NATIVE_MODULES = [ + "fiftyone_devicedetection_onpremise.DeviceDetectionHashEngineModule", + "fiftyone_ipintelligence_onpremise.IpIntelligenceEngineModule", +] + + +def _native_types(): + types = [] + for name in NATIVE_MODULES: + try: + module = importlib.import_module(name) + except ImportError: + continue + types.append(module.ResultsBaseSwig) + types.append(module.EngineBaseSwig) + return tuple(types) + + +NATIVE_TYPES = _native_types() + +# A failed test is not reported again, because its traceback can hold the +# objects it would have closed. +_failed_tests = set() + +# Objects created when the tests were collected are checked at the end only +# when every test ran, because the teardown that closes them runs with the +# tests. +_some_tests_deselected = False + + +def _alive(): + # The real type is used, because isinstance also asks the object for + # its __class__, which some proxies cannot answer. + return [o for o in gc.get_objects() + if issubclass(type(o), NATIVE_TYPES)] + + +def _describe(names): + return (f"{len(names)} native object(s) still alive " + f"({', '.join(sorted(names))}). Close every FlowData and " + "pipeline the test creates, for example with a with " + "statement, so their native data is released straight away.") + + +def pytest_runtest_logreport(report): + if report.failed: + _failed_tests.add(report.nodeid) + + +def pytest_deselected(items): + global _some_tests_deselected + _some_tests_deselected = True + + +@pytest.fixture(autouse=True) +def native_objects_released(request): + gc.collect() + before = {id(o) for o in _alive()} + gc.disable() + try: + yield + left = [type(o).__name__ for o in _alive() if id(o) not in before] + finally: + gc.enable() + gc.collect() + if left and request.node.nodeid not in _failed_tests: + pytest.fail(_describe(left), pytrace=False) + + +def pytest_sessionfinish(session, exitstatus): + gc.collect() + left = [type(o).__name__ for o in _alive()] + if (left and exitstatus == pytest.ExitCode.OK and + not _some_tests_deselected): + reporter = session.config.pluginmanager.get_plugin("terminalreporter") + if reporter is not None: + reporter.write_line( + "After the last test, " + _describe(left), red=True) + session.exitstatus = pytest.ExitCode.TESTS_FAILED diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/tests/test_examples.py b/ip-intelligence/fiftyone_ipintelligence_examples/tests/test_examples.py index b0e359a3..0ada1f94 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/tests/test_examples.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/tests/test_examples.py @@ -29,10 +29,10 @@ from fiftyone_ipintelligence_shared.example_constants import ASN_DATAFILE_NAME from fiftyone_ipintelligence_shared.example_constants import EVIDENCE_FILE_NAME -# The tests run against the freely available data file committed to the -# ip-intelligence-data submodule, so they only rely on properties that the -# free data file provides (location properties such as Latitude, Longitude -# and Areas are not present in it). +# The tests run against the freely available ASN data file the data +# repository links to, so they only rely on properties that the free data +# file provides (location properties such as Latitude, Longitude and Areas +# are not present in it). data_file = ExampleUtils.find_file(ASN_DATAFILE_NAME) evidence_file = ExampleUtils.find_file(EVIDENCE_FILE_NAME) truth_file = ExampleUtils.find_file("geoip_comparison_evidence.csv") @@ -61,8 +61,8 @@ class ExampleTests(unittest.TestCase): @unittest.skipUnless( bool(data_file), "No IP intelligence data file available") def test_onpremise_datafile_loads(self): - # Verify the freely available data file committed to the - # ip-intelligence-data submodule loads and processes correctly. + # Verify the freely available ASN data file loads and processes + # correctly. from fiftyone_ipintelligence.ipintelligence_pipelinebuilder import IpIntelligencePipelineBuilder pipeline = IpIntelligencePipelineBuilder( data_file_path=data_file, @@ -70,7 +70,9 @@ def test_onpremise_datafile_loads(self): usage_sharing=False, auto_update=False, licence_keys="").build() + self.addCleanup(pipeline.close) data = pipeline.create_flowdata() + self.addCleanup(data.close) data.evidence.add_from_dict({"query.client-ip": "8.8.8.8"}) data.process() # The supplied IP address is echoed back in the results. diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/tox.ini b/ip-intelligence/fiftyone_ipintelligence_examples/tox.ini index 0008718b..3bc4fbd3 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/tox.ini +++ b/ip-intelligence/fiftyone_ipintelligence_examples/tox.ini @@ -24,6 +24,13 @@ deps = # these packages is on a package index yet, so it comes from the # working copy as the others do. -e../fiftyone_ipintelligence_translation/ + # The pipeline packages come from this checkout rather than a package + # index, so the tests run with the pipeline code they are released with. + -e../../pipeline/fiftyone_pipeline_core/ + -e../../pipeline/fiftyone_pipeline_engines/ + -e../../pipeline/fiftyone_pipeline_engines_fiftyone/ + -e../../pipeline/fiftyone_pipeline_cloudrequestengine/ + -e../../pipeline/fiftyone_pipeline_translation/ # --cov is needed even though coverage of the examples is not the point: # common-ci's run-integration-tests.ps1 moves the .coverage file every # package writes, and throws when there is none, after the tests pass. diff --git a/ip-intelligence/fiftyone_ipintelligence_onpremise/readme.md b/ip-intelligence/fiftyone_ipintelligence_onpremise/readme.md index b7eb9ac3..569058a1 100644 --- a/ip-intelligence/fiftyone_ipintelligence_onpremise/readme.md +++ b/ip-intelligence/fiftyone_ipintelligence_onpremise/readme.md @@ -1,14 +1,14 @@ # 51Degrees IP Intelligence Engines - OnPremise -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_onpremise-readme.md&utm_term=51degrees-ip-intelligence-engines-onpremise "Data rewards the curious") **Python IP Intelligence** +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_onpremise&utm_term=51degrees-ip-intelligence-engines-onpremise "Data rewards the curious") **Python IP Intelligence** -[Developer Documentation](https://51degrees.com/ip-intelligence-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_onpremise-readme.md&utm_term=51degrees-ip-intelligence-engines-onpremise "developer documentation") +[Developer Documentation](https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_onpremise&utm_term=51degrees-ip-intelligence-engines-onpremise "developer documentation") ## Introduction This package resolves IP addresses to location, network and registration information using a local `.ipi` data file and the -[51Degrees Pipeline API](https://51degrees.com/documentation/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_onpremise-readme.md&utm_term=introduction). +[51Degrees Pipeline API](https://51degrees.com/documentation/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_onpremise&utm_term=introduction). The native [51degrees-cxx](https://github.com/51Degrees/51degrees-cxx/blob/main/docs/ipi/README.md) library is included as a git submodule and wrapped with SWIG. @@ -28,12 +28,17 @@ python setup.py build_clib build_ext ## Data files -The on-premise engine requires a `.ipi` data file. A Lite data file can be -downloaded from -[51Degrees](https://51ddatafiles.blob.core.windows.net/enterpriseipi/51Degrees-LiteIpiV41.ipi.gz), -see also the [ip-intelligence-data](https://github.com/51Degrees/ip-intelligence-data) -repository which is included as a nested submodule and contains evidence -files for testing. +The on-premise engine requires a `.ipi` data file. The free Lite data file, +`51Degrees-LiteV41.ipi`, is linked from the +[data](https://github.com/51Degrees/data) repository, which is included as +the nested submodule `cxx/data` and contains evidence files for testing. From +the root of a checkout of +[51degrees-python](https://github.com/51Degrees/51degrees-python), download it +by running `sh cxx/data/get-data-files.sh 51Degrees-LiteV41.ipi`, or +`./cxx/data/get-data-files.ps1 51Degrees-LiteV41.ipi` in PowerShell. Without a +checkout, download it from +https://distributor.51degrees.com/api/v2/free/ipi-lite-v41, which sends it +gzip compressed. ## Usage @@ -41,7 +46,7 @@ files for testing. from fiftyone_ipintelligence_onpremise.ipintelligence_onpremise_pipelinebuilder import IpIntelligenceOnPremisePipelineBuilder pipeline = IpIntelligenceOnPremisePipelineBuilder( - data_file_path="51Degrees-LiteIpiV41.ipi", + data_file_path="51Degrees-LiteV41.ipi", licence_keys="", usage_sharing=False).build() @@ -50,3 +55,12 @@ flowdata.evidence.add("query.client-ip", "185.28.167.78") flowdata.process() print(flowdata.ip.registeredname.value()) ``` + +## Find out more + +- [IP Intelligence](https://51degrees.com/ip-intelligence?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_onpremise&utm_term=find-out-more-ip-intelligence) +- [IP Intelligence documentation](https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_onpremise&utm_term=find-out-more-documentation) +- [Getting started](https://51degrees.com/documentation/_ip_intelligence__quickstart.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_onpremise&utm_term=find-out-more-getting-started) +- [Pricing](https://51degrees.com/pricing?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_onpremise&utm_term=find-out-more-pricing) +- https://github.com/51Degrees/51degrees-python +- https://github.com/51Degrees/51degrees-cxx diff --git a/ip-intelligence/fiftyone_ipintelligence_onpremise/setup.py b/ip-intelligence/fiftyone_ipintelligence_onpremise/setup.py index 6c2d0349..2d850cf7 100644 --- a/ip-intelligence/fiftyone_ipintelligence_onpremise/setup.py +++ b/ip-intelligence/fiftyone_ipintelligence_onpremise/setup.py @@ -250,6 +250,11 @@ def stage_native_source(): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", + project_urls={ + "Homepage": "https://51degrees.com/ip-intelligence?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_onpremise&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_onpremise&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description = """IP Intelligence and IP geolocation for Python that finds an IP address's likely location (country, region, town, and coordinates), connection type, and network by reading a data file on your server. An alternative to MaxMind and IPinfo.""", keywords=[ "51degrees", @@ -272,5 +277,12 @@ def stage_native_source(): py_modules=["fiftyone_ipintelligence_onpremise"], packages=["fiftyone_ipintelligence_onpremise"], package_dir={"": "src"}, - install_requires=["fiftyone_ipintelligence_shared", "fiftyone_pipeline_engines_fiftyone"], + install_requires=[ + "fiftyone_ipintelligence_shared", + "fiftyone_pipeline_engines_fiftyone", + # The first versions with close(), which releases the native memory + # this package holds. + "fiftyone_pipeline_core>=4.5.42", + "fiftyone_pipeline_engines>=4.5.42", + ], ) diff --git a/ip-intelligence/fiftyone_ipintelligence_onpremise/src/fiftyone_ipintelligence_onpremise/constants.py b/ip-intelligence/fiftyone_ipintelligence_onpremise/src/fiftyone_ipintelligence_onpremise/constants.py index 3e129185..deea6331 100644 --- a/ip-intelligence/fiftyone_ipintelligence_onpremise/src/fiftyone_ipintelligence_onpremise/constants.py +++ b/ip-intelligence/fiftyone_ipintelligence_onpremise/src/fiftyone_ipintelligence_onpremise/constants.py @@ -29,3 +29,10 @@ 'on-premise IP intelligence engine. Its results ' 'read from native memory, which is not safe to ' 'share between requests.') + +ENGINE_CLOSED = ('The on-premise IP intelligence engine has been closed, ' + 'so it cannot process evidence. Build a new pipeline.') + +RESULTS_CLOSED = ('These IP intelligence results have been closed, so ' + 'their properties cannot be read. Read the values needed ' + 'before closing the FlowData.') diff --git a/ip-intelligence/fiftyone_ipintelligence_onpremise/src/fiftyone_ipintelligence_onpremise/ipintelligence_onpremise.py b/ip-intelligence/fiftyone_ipintelligence_onpremise/src/fiftyone_ipintelligence_onpremise/ipintelligence_onpremise.py index c724d64d..37b45904 100644 --- a/ip-intelligence/fiftyone_ipintelligence_onpremise/src/fiftyone_ipintelligence_onpremise/ipintelligence_onpremise.py +++ b/ip-intelligence/fiftyone_ipintelligence_onpremise/src/fiftyone_ipintelligence_onpremise/ipintelligence_onpremise.py @@ -328,6 +328,20 @@ def on_registration(self, pipeline): super(IpIntelligenceOnPremise, self).on_registration(pipeline) self.init_engine() + def close(self): + + """! + + Stop the data file updates and release the native engine. Its data + is freed once no results read from it are still held, so close + every FlowData the engine processed as well. Closing the pipeline + closes the engine. + + """ + + super(IpIntelligenceOnPremise, self).close() + self.engine = None + def get_client_ips(self, flow_data): """! @@ -379,6 +393,9 @@ def process_internal(self, flow_data): """ + if getattr(self, "engine", None) is None: + raise Exception(ENGINE_CLOSED) + evidence = EvidenceIpiSwig() for key, value in flow_data.evidence.get_all().items(): diff --git a/ip-intelligence/fiftyone_ipintelligence_onpremise/src/fiftyone_ipintelligence_onpremise/swig_data.py b/ip-intelligence/fiftyone_ipintelligence_onpremise/src/fiftyone_ipintelligence_onpremise/swig_data.py index ea285c84..40413cf3 100644 --- a/ip-intelligence/fiftyone_ipintelligence_onpremise/src/fiftyone_ipintelligence_onpremise/swig_data.py +++ b/ip-intelligence/fiftyone_ipintelligence_onpremise/src/fiftyone_ipintelligence_onpremise/swig_data.py @@ -25,6 +25,8 @@ from fiftyone_pipeline_engines.missingproperty_service import MissingPropertyService +from .constants import RESULTS_CLOSED + # Number of decimal places used when converting WKB geometry # properties (e.g. Areas) to their WKT string form. WKT_DECIMAL_PLACES = 16 @@ -71,6 +73,33 @@ def __init__(self, flow_element, swig_results, echo_ip=None, echo_ipv6=None): self.missing_property_service = OnPremiseMissingPropertyService() + def close(self): + + """! + Release the native results, which keep the engine's data in memory + while they are held, even after the engine is closed. Values already + read are unaffected, and reading a property afterwards raises an + exception. Closing the FlowData closes its results. + """ + + self.swig_results = None + + def get(self, key): + + """! + Get a property value, once the results are known to be open + @type key: string + @param key: The key of the property to retrieve + @returns mixed + """ + + # Checked here, because AspectData.get turns an exception from + # get_internal into a missing property message. + if self.swig_results is None: + raise Exception(RESULTS_CLOSED) + + return super(SwigData, self).get(key) + @staticmethod def utf8_string_to_str(utf8_string): diff --git a/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/conftest.py b/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/conftest.py new file mode 100644 index 00000000..fc53a6d0 --- /dev/null +++ b/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/conftest.py @@ -0,0 +1,119 @@ +# ********************************************************************* +# This Original Work is copyright of 51 Degrees Mobile Experts Limited. +# Copyright 2026 51 Degrees Mobile Experts Limited, Davidson House, +# Forbury Square, Reading, Berkshire, United Kingdom RG1 3EU. +# +# This Original Work is licensed under the European Union Public Licence +# (EUPL) v.1.2 and is subject to its terms as set out below. +# +# If a copy of the EUPL was not distributed with this file, You can obtain +# one at https://opensource.org/licenses/EUPL-1.2. +# +# The 'Compatible Licences' set out in the Appendix to the EUPL (as may be +# amended by the European Commission) shall be deemed incompatible for +# the purposes of the Work and the provisions of the compatibility +# clause in Article 5 of the EUPL shall not apply. +# +# If using the Work as, or as part of, a network application, by +# including the attribution notice(s) required under Article 5 of the EUPL +# in the end user terms of the application under an appropriate heading, +# such notice(s) shall fulfill the requirements of that article. +# ********************************************************************* + +# Fails a test that leaves the native results or engine of an on-premise +# engine alive, because they hold the engine's data in memory. The same +# file is in each test folder that runs on-premise engines. +# +# Reference counting frees a native object as soon as nothing holds it. A +# FlowData refers to itself through its evidence, and a pipeline through +# its elements, so when they are not closed only the cycle collector frees +# what they hold. The collector is off while each test runs, so whatever +# the test leaves open is still alive when it ends. Objects that existed +# before the test, such as a pipeline built in setUpClass, are checked once +# every test has run. + +import gc +import importlib + +import pytest + +# The SWIG modules of the on-premise engines, whichever are installed. +NATIVE_MODULES = [ + "fiftyone_devicedetection_onpremise.DeviceDetectionHashEngineModule", + "fiftyone_ipintelligence_onpremise.IpIntelligenceEngineModule", +] + + +def _native_types(): + types = [] + for name in NATIVE_MODULES: + try: + module = importlib.import_module(name) + except ImportError: + continue + types.append(module.ResultsBaseSwig) + types.append(module.EngineBaseSwig) + return tuple(types) + + +NATIVE_TYPES = _native_types() + +# A failed test is not reported again, because its traceback can hold the +# objects it would have closed. +_failed_tests = set() + +# Objects created when the tests were collected are checked at the end only +# when every test ran, because the teardown that closes them runs with the +# tests. +_some_tests_deselected = False + + +def _alive(): + # The real type is used, because isinstance also asks the object for + # its __class__, which some proxies cannot answer. + return [o for o in gc.get_objects() + if issubclass(type(o), NATIVE_TYPES)] + + +def _describe(names): + return (f"{len(names)} native object(s) still alive " + f"({', '.join(sorted(names))}). Close every FlowData and " + "pipeline the test creates, for example with a with " + "statement, so their native data is released straight away.") + + +def pytest_runtest_logreport(report): + if report.failed: + _failed_tests.add(report.nodeid) + + +def pytest_deselected(items): + global _some_tests_deselected + _some_tests_deselected = True + + +@pytest.fixture(autouse=True) +def native_objects_released(request): + gc.collect() + before = {id(o) for o in _alive()} + gc.disable() + try: + yield + left = [type(o).__name__ for o in _alive() if id(o) not in before] + finally: + gc.enable() + gc.collect() + if left and request.node.nodeid not in _failed_tests: + pytest.fail(_describe(left), pytrace=False) + + +def pytest_sessionfinish(session, exitstatus): + gc.collect() + left = [type(o).__name__ for o in _alive()] + if (left and exitstatus == pytest.ExitCode.OK and + not _some_tests_deselected): + reporter = session.config.pluginmanager.get_plugin("terminalreporter") + if reporter is not None: + reporter.write_line( + "After the last test, " + _describe(left), red=True) + session.exitstatus = pytest.ExitCode.TESTS_FAILED diff --git a/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/constants.py b/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/constants.py index 6f9123f3..2c5180f8 100644 --- a/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/constants.py +++ b/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/constants.py @@ -42,11 +42,12 @@ def get_data_file_path(): os.path.dirname(__file__), "..", "..", "assets", "51Degrees-LiteIpiV41.ipi") - # The ASN data file committed to the repository's ip-intelligence-data - # submodule. Unlike the Lite and Enterprise files it needs no download - # or secret, so it is always present wherever the engine can be built. + # The free ASN data file, which ci/fetch-assets.ps1 downloads with the + # data repository's script, beside its link in cxx/data/ip-intelligence. + # Unlike the Enterprise file it needs no secret. @staticmethod - def get_committed_data_file_path(): + def get_asn_data_file_path(): return os.path.join( os.path.dirname(__file__), - "..", "..", "ip-intelligence-data", ASN_DATAFILE_NAME) + "..", "..", "..", "cxx", "data", "ip-intelligence", + ASN_DATAFILE_NAME) diff --git a/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/test_ipintelligence.py b/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/test_ipintelligence.py index 13f5d042..7b14231c 100644 --- a/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/test_ipintelligence.py +++ b/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/test_ipintelligence.py @@ -49,8 +49,13 @@ def setUpClass(cls): .add(cls.engine) \ .build() + @classmethod + def tearDownClass(cls): + cls.pipeline.close() + def process(self, key, value): flowdata = self.pipeline.create_flowdata() + self.addCleanup(flowdata.close) flowdata.evidence.add(key, value) flowdata.process() return flowdata @@ -133,6 +138,24 @@ def test_process_ipv6(self): ipv6 = flowdata.ip.get("ipv6") self.assertTrue(ipv6.has_value()) + def test_flowdata_close_releases_results(self): + """! + Closing the FlowData releases its native results, and reading a + property afterwards raises an exception. + """ + + with self.pipeline.create_flowdata() as flowdata: + flowdata.evidence.add( + "query.client-ip", TestConstants.IPV4_ADDRESS) + flowdata.process() + registered_name = flowdata.ip.get("registeredname") + + self.assertTrue(registered_name.has_value()) + self.assertIsNone(flowdata.ip.swig_results) + with self.assertRaises(Exception) as context: + flowdata.ip.get("registeredname") + self.assertIn("closed", str(context.exception)) + def test_missing_property(self): """! Accessing a property that does not exist raises an exception @@ -171,48 +194,49 @@ def test_missing_licence_key(self): self.assertIn("licence key is required", str(context.exception)) -class CommittedDataFileTests(unittest.TestCase): +class AsnDataFileTests(unittest.TestCase): """! - The native engine accepts exactly one data file format version, so the - cxx (51degrees-cxx) pin and the ip-intelligence-data pin must agree. - They once did not: the engine moved to format 4.6 while the data pin - still held the 4.5 ASN file. These tests open the ASN file committed to - the data submodule, which needs no download or secret, so pins that - disagree fail here on any machine. - - They do not prove the engine reads the data files 51Degrees publishes. - That is IpIntelligenceOnPremiseTests above, against the Lite or - Enterprise file CI downloads; it skips when neither is present. + The native engine accepts exactly one data file format version. These + tests open the free ASN data file. The Distributor always serves the + latest one published, and it needs no secret, so an engine that cannot + read the format 51Degrees publishes fails here on any machine. + + IpIntelligenceOnPremiseTests above runs against the Lite or Enterprise + file CI downloads, and skips when neither is present. test_onpremise_datafile_loads in the examples package also opens the ASN file, but it skips silently when the file is not found and it runs after the examples build, so it is not relied on here. """ - def test_committed_data_file_present(self): + def test_asn_data_file_present(self): """! - The committed data file exists. If it does not, the - ip-intelligence-data submodule was not checked out, and the test + The ASN data file has been downloaded. If it has not, the test below would prove nothing. """ self.assertTrue( - os.path.isfile(TestConstants.get_committed_data_file_path()), - "Run 'git submodule update --init --recursive'.") + os.path.isfile(TestConstants.get_asn_data_file_path()), + "Download it from the repository root by running " + "'sh cxx/data/get-data-files.sh 51Degrees-IPIV4AsnIpiV41.ipi', " + "or './cxx/data/get-data-files.ps1 51Degrees-IPIV4AsnIpiV41.ipi' " + "in PowerShell.") - def test_committed_data_file_opens(self): + def test_asn_data_file_opens(self): """! - The engine opens the committed data file, so it reads that file's - format version, and returns its properties for an IPv4 address. + The engine opens the ASN data file, so it reads that file's format + version, and returns its properties for an IPv4 address. """ engine = IpIntelligenceOnPremise( - data_file_path=TestConstants.get_committed_data_file_path(), + data_file_path=TestConstants.get_asn_data_file_path(), licence_keys="") pipeline = PipelineBuilder().add(engine).build() + self.addCleanup(pipeline.close) self.assertIn("asnname", engine.properties) flowdata = pipeline.create_flowdata() + self.addCleanup(flowdata.close) flowdata.evidence.add("query.client-ip", TestConstants.IPV4_ADDRESS) flowdata.process() @@ -263,3 +287,31 @@ def test_pipeline_builder_refuses_cache(self): cache=LRUEngineCache()).build() self.assertIn("results cache", str(context.exception)) + + +@unittest.skipUnless( + os.path.isfile(data_file), + "No IP intelligence data file found at '%s'. Run ci/fetch-assets.ps1 " + "or set the IPINTELLIGENCEDATAFILE environment variable." % data_file) +class IpIntelligenceCloseTests(unittest.TestCase): + + def test_pipeline_close_releases_engine(self): + """! + Closing the pipeline releases the native engine, and processing + afterwards raises an exception. + """ + + engine = IpIntelligenceOnPremise( + data_file_path=data_file, + licence_keys="") + pipeline = PipelineBuilder().add(engine).build() + + pipeline.close() + + self.assertIsNone(engine.engine) + flowdata = pipeline.create_flowdata() + self.addCleanup(flowdata.close) + flowdata.evidence.add("query.client-ip", TestConstants.IPV4_ADDRESS) + with self.assertRaises(Exception) as context: + flowdata.process() + self.assertIn("closed", str(context.exception)) diff --git a/ip-intelligence/fiftyone_ipintelligence_onpremise/tox.ini b/ip-intelligence/fiftyone_ipintelligence_onpremise/tox.ini index 3dc3b4c9..211e4571 100644 --- a/ip-intelligence/fiftyone_ipintelligence_onpremise/tox.ini +++ b/ip-intelligence/fiftyone_ipintelligence_onpremise/tox.ini @@ -24,6 +24,12 @@ deps = # of it in beside the wheel tox builds and made examples and on-premise # depend on each other. -e../fiftyone_ipintelligence_shared/ + # The pipeline packages come from this checkout rather than a package + # index, so the tests run with the pipeline code they are released with. + -e../../pipeline/fiftyone_pipeline_core/ + -e../../pipeline/fiftyone_pipeline_engines/ + -e../../pipeline/fiftyone_pipeline_engines_fiftyone/ + -e../../pipeline/fiftyone_pipeline_cloudrequestengine/ commands = python setup.py build_clib build_ext {[common]commands} diff --git a/ip-intelligence/fiftyone_ipintelligence_shared/readme.md b/ip-intelligence/fiftyone_ipintelligence_shared/readme.md index 7980c868..ced1fe62 100644 --- a/ip-intelligence/fiftyone_ipintelligence_shared/readme.md +++ b/ip-intelligence/fiftyone_ipintelligence_shared/readme.md @@ -1,14 +1,14 @@ # 51Degrees IP Intelligence Engines - Shared -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_shared-readme.md&utm_term=51degrees-ip-intelligence-engines-shared "Data rewards the curious") **Python IP Intelligence** +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_shared&utm_term=51degrees-ip-intelligence-engines-shared "Data rewards the curious") **Python IP Intelligence** -[Developer Documentation](https://51degrees.com/ip-intelligence-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_shared-readme.md&utm_term=51degrees-ip-intelligence-engines-shared "developer documentation") +[Developer Documentation](https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_shared&utm_term=51degrees-ip-intelligence-engines-shared "developer documentation") ## Introduction This package contains shared code used by both the cloud and on-premise variants of the 51Degrees IP intelligence engines for the -[51Degrees Pipeline API](https://51degrees.com/documentation/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_shared-readme.md&utm_term=introduction). +[51Degrees Pipeline API](https://51degrees.com/documentation/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_shared&utm_term=introduction). It is not intended to be used directly. See [fiftyone_ipintelligence](https://pypi.org/project/fiftyone-ipintelligence/) @@ -26,3 +26,11 @@ instead. * The `fiftyone_pipeline_core`, `fiftyone_pipeline_engines`, `fiftyone_pipeline_cloudrequestengine` and `fiftyone_pipeline_engines_fiftyone` packages + +## Find out more + +- [IP Intelligence](https://51degrees.com/ip-intelligence?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_shared&utm_term=find-out-more-ip-intelligence) +- [IP Intelligence documentation](https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_shared&utm_term=find-out-more-documentation) +- [Getting started](https://51degrees.com/documentation/_ip_intelligence__quickstart.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_shared&utm_term=find-out-more-getting-started) +- [Pricing](https://51degrees.com/pricing?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_shared&utm_term=find-out-more-pricing) +- https://github.com/51Degrees/51degrees-python diff --git a/ip-intelligence/fiftyone_ipintelligence_shared/setup.py b/ip-intelligence/fiftyone_ipintelligence_shared/setup.py index c4480d4e..7f89e1d0 100644 --- a/ip-intelligence/fiftyone_ipintelligence_shared/setup.py +++ b/ip-intelligence/fiftyone_ipintelligence_shared/setup.py @@ -40,7 +40,11 @@ def read(file_name): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_shared-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/ip-intelligence?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_shared&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_shared&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("Shared code used by the cloud and on-premise variants of the 51Degrees IP Intelligence engines for the 51Degrees Pipeline API."), long_description=read("readme.md"), long_description_content_type='text/markdown', diff --git a/ip-intelligence/fiftyone_ipintelligence_shared/src/fiftyone_ipintelligence_shared/example_constants.py b/ip-intelligence/fiftyone_ipintelligence_shared/src/fiftyone_ipintelligence_shared/example_constants.py index e11dd421..35945298 100644 --- a/ip-intelligence/fiftyone_ipintelligence_shared/src/fiftyone_ipintelligence_shared/example_constants.py +++ b/ip-intelligence/fiftyone_ipintelligence_shared/src/fiftyone_ipintelligence_shared/example_constants.py @@ -38,12 +38,13 @@ # Note: the .NET examples' Constants.cs still uses the legacy name # "51Degrees-EnterpriseIpiV41.ipi", but the published enterprise/ASN data # files use the "IPIV4" infix (cf. 51Degrees-IPIV4AsnIpiV41.ipi in the -# ip-intelligence-data repo), so the current name is used here. +# data repository), so the current name is used here. ENTERPRISE_DATAFILE_NAME = '51Degrees-IPIV4EnterpriseIpiV41.ipi' -LITE_DATAFILE_NAME = '51Degrees-LiteIpiV41.ipi' -# The ASN data file is the freely available data file committed -# to the ip-intelligence-data submodule. It only contains the network -# (ASN) properties, but is useful for testing the on-premise engine against -# the current data format without an enterprise data file. +# The free Lite data file, named as the data repository links to it. +LITE_DATAFILE_NAME = '51Degrees-LiteV41.ipi' +# The ASN data file is a freely available data file the data repository +# links to. It only contains the network (ASN) properties, but is useful for +# testing the on-premise engine against the current data format without an +# enterprise data file. ASN_DATAFILE_NAME = '51Degrees-IPIV4AsnIpiV41.ipi' EVIDENCE_FILE_NAME = 'evidence.yml' diff --git a/ip-intelligence/fiftyone_ipintelligence_translation/.gitignore b/ip-intelligence/fiftyone_ipintelligence_translation/.gitignore index 36ed5ddc..c39739ff 100644 --- a/ip-intelligence/fiftyone_ipintelligence_translation/.gitignore +++ b/ip-intelligence/fiftyone_ipintelligence_translation/.gitignore @@ -1,3 +1,3 @@ -# Translation YAML staged from the ip-intelligence-data submodule at build -# time (see setup.py stage_translations). Not committed; generated on build. +# Translation YAML staged from cxx/data/ip-intelligence at build time (see +# setup.py stage_translations). Not committed; generated on build. src/fiftyone_ipintelligence_translation/translations/ diff --git a/ip-intelligence/fiftyone_ipintelligence_translation/readme.md b/ip-intelligence/fiftyone_ipintelligence_translation/readme.md index ad37b2fc..7f12b228 100644 --- a/ip-intelligence/fiftyone_ipintelligence_translation/readme.md +++ b/ip-intelligence/fiftyone_ipintelligence_translation/readme.md @@ -1,8 +1,8 @@ # IP Intelligence Country Translation -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_translation-readme.md&utm_term=ip-intelligence-country-translation "Data rewards the curious") **Python IP Intelligence Country Translation** +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_translation&utm_term=ip-intelligence-country-translation "Data rewards the curious") **Python IP Intelligence Country Translation** -[Developer Documentation](https://51degrees.com/documentation/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_translation-readme.md&utm_term=ip-intelligence-country-translation "Developer Documentation") +[Developer Documentation](https://51degrees.com/documentation/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_translation&utm_term=ip-intelligence-country-translation "Developer Documentation") ## Introduction @@ -33,3 +33,11 @@ Built on top of: * [fiftyone_pipeline_core](https://pypi.org/project/fiftyone-pipeline-core/) * [fiftyone_pipeline_translation](https://pypi.org/project/fiftyone-pipeline-translation/) + +## Find out more + +- [IP Intelligence](https://51degrees.com/ip-intelligence?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_translation&utm_term=find-out-more-ip-intelligence) +- [IP Intelligence documentation](https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_translation&utm_term=find-out-more-documentation) +- [Getting started](https://51degrees.com/documentation/_ip_intelligence__quickstart.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_translation&utm_term=find-out-more-getting-started) +- [Pricing](https://51degrees.com/pricing?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_translation&utm_term=find-out-more-pricing) +- https://github.com/51Degrees/51degrees-python diff --git a/ip-intelligence/fiftyone_ipintelligence_translation/setup.py b/ip-intelligence/fiftyone_ipintelligence_translation/setup.py index a29e4097..91ca1dd4 100644 --- a/ip-intelligence/fiftyone_ipintelligence_translation/setup.py +++ b/ip-intelligence/fiftyone_ipintelligence_translation/setup.py @@ -40,14 +40,17 @@ def read(file_name): def stage_translations(): - """Copy the translation YAML files shipped in the ip-intelligence-data - submodule into the package so they are included in the built distribution. - This mirrors how the .NET project embeds the same files as resources. If the - submodule is not present the package falls back to reading it at run time, - so any failure here is ignored.""" + """Copy the translation YAML files from the data repository, the + cxx/data submodule, into the package so they are included in the built + distribution. This mirrors how the .NET project embeds the same files as + resources. A build from a source distribution has no cxx folder, but its + package already holds the files, so a missing folder is skipped. Any + failure here is ignored, and a package run from a source checkout reads + the folder at run time instead.""" here = os.path.dirname(os.path.abspath(__file__)) - source = os.path.join(here, "..", "ip-intelligence-data", "Translations") + source = os.path.join( + here, "..", "..", "cxx", "data", "ip-intelligence", "Translations") dest = os.path.join( here, "src", "fiftyone_ipintelligence_translation", "translations") if not os.path.isdir(source): @@ -77,7 +80,11 @@ def copy_yaml(folder): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_translation-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/ip-intelligence?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_translation&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_translation&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("Country translation engines for 51Degrees on-premise IP Intelligence. Turns the weighted ISO country codes returned by the IP Intelligence engine into localized country names and complete ordered country lists."), long_description=read("readme.md"), long_description_content_type='text/markdown', diff --git a/ip-intelligence/fiftyone_ipintelligence_translation/src/fiftyone_ipintelligence_translation/resources.py b/ip-intelligence/fiftyone_ipintelligence_translation/src/fiftyone_ipintelligence_translation/resources.py index eb9aff55..4c0ffc4c 100644 --- a/ip-intelligence/fiftyone_ipintelligence_translation/src/fiftyone_ipintelligence_translation/resources.py +++ b/ip-intelligence/fiftyone_ipintelligence_translation/src/fiftyone_ipintelligence_translation/resources.py @@ -30,14 +30,15 @@ def _find_submodule_translations(): - """Walk up from this package looking for the ip-intelligence-data - submodule's Translations directory, used when the package is run from a - source checkout rather than an installed distribution.""" + """Walk up from this package looking for the data repository's + Translations directory, cxx/data/ip-intelligence/Translations, used when + the package is run from a source checkout rather than an installed + distribution.""" current = _PACKAGE_DIR for _ in range(8): candidate = os.path.join( - current, "ip-intelligence-data", "Translations") + current, "cxx", "data", "ip-intelligence", "Translations") if os.path.isdir(candidate): return candidate parent = os.path.dirname(current) @@ -50,9 +51,9 @@ def _find_submodule_translations(): def _translation_files(): """Return a dict of translation file name -> absolute path. The packaged - translations directory is preferred; if it is empty the ip-intelligence-data - submodule is used as a fallback (countrycodes at the root, countries under - OSM).""" + translations directory is preferred; if it is empty the data repository's + Translations directory is used as a fallback (countrycodes at the root, + countries under OSM).""" files = {} diff --git a/ip-intelligence/fiftyone_ipintelligence_translation/tests/test_translation.py b/ip-intelligence/fiftyone_ipintelligence_translation/tests/test_translation.py index 4decac97..199ca3b6 100644 --- a/ip-intelligence/fiftyone_ipintelligence_translation/tests/test_translation.py +++ b/ip-intelligence/fiftyone_ipintelligence_translation/tests/test_translation.py @@ -20,6 +20,7 @@ # such notice(s) shall fulfill the requirements of that article. # ********************************************************************* +import os import unittest import yaml @@ -34,6 +35,7 @@ from fiftyone_ipintelligence_translation.countries_translation_engine_builder import ( CountriesTranslationEngineBuilder) from fiftyone_ipintelligence_translation.comparer import make_comparer +from fiftyone_ipintelligence_translation import resources from fiftyone_ipintelligence_translation.resources import ( get_all_countries, get_country_resources) @@ -448,5 +450,18 @@ def test_population_translated_names(self): self.assertNotEqual(geo_codes[0], pop_codes[0]) +class PackagedTranslationsTests(unittest.TestCase): + + def test_translations_are_packaged(self): + # The tests run against the built package. A build that cannot find + # the data repository's Translations folder still succeeds, without + # the YAML files, and the other tests can still pass by reading the + # folder from the checkout, so this checks the package itself. + folder = resources._PACKAGED_TRANSLATIONS + names = os.listdir(folder) if os.path.isdir(folder) else [] + self.assertIn("countrycodes.en_GB.yml", names) + self.assertIn("countries.de_DE.yml", names) + + if __name__ == "__main__": unittest.main() diff --git a/ip-intelligence/ip-intelligence-data b/ip-intelligence/ip-intelligence-data deleted file mode 160000 index b0f0b6af..00000000 --- a/ip-intelligence/ip-intelligence-data +++ /dev/null @@ -1 +0,0 @@ -Subproject commit b0f0b6afe4fe741bc93a7bd6a7a2ad1058dd6e71 diff --git a/ip-intelligence/readme.md b/ip-intelligence/readme.md index 68e79e2e..7a1be44e 100644 --- a/ip-intelligence/readme.md +++ b/ip-intelligence/readme.md @@ -2,7 +2,7 @@ ![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=51degrees-ip-intelligence-engines "Data rewards the curious") **Python IP Intelligence** -[Developer Documentation](https://51degrees.com/ip-intelligence-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=51degrees-ip-intelligence-engines "developer documentation") +[Developer Documentation](https://51degrees.com/documentation/_ip_intelligence__index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=51degrees-ip-intelligence-engines "developer documentation") ## Introduction @@ -21,17 +21,16 @@ design of this API. For runtime dependencies, see our [dependencies](https://51degrees.com/documentation/_info__dependencies.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=dependencies) -page. The [tested versions](https://51degrees.com/documentation/_info__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=dependencies) +page. The [tested versions](https://51degrees.com/documentation/_product_summaries__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=dependencies) page shows the Python versions that we currently test against. ### Git submodules The on-premise engine wraps the native [51degrees-cxx](https://github.com/51Degrees/51degrees-cxx/blob/main/docs/ipi/README.md) -library, and the -[ip-intelligence-data](https://github.com/51Degrees/ip-intelligence-data) -repository (data and evidence files used for testing) is also included, both -as git submodules. When cloning, use: +library, which is included as the `cxx` git submodule. It has the +[data](https://github.com/51Degrees/data) repository (data and evidence files +used for testing) as its own submodule, at `cxx/data`. When cloning, use: ``` git clone --recurse-submodules https://github.com/51Degrees/51degrees-python.git @@ -45,13 +44,25 @@ git submodule update --init --recursive ### Data files -The on-premise engine requires a `.ipi` data file. The -[ip-intelligence-data](https://github.com/51Degrees/ip-intelligence-data) -submodule provides a freely available network-only data file +The on-premise engine requires a `.ipi` data file. The data repository's +`cxx/data/ip-intelligence` folder holds the evidence files used for testing, +and links to a freely available network-only data file (`51Degrees-IPIV4AsnIpiV41.ipi`), which the examples tests and -`CommittedDataFileTests` use, along with the evidence files used for -testing. The on-premise unit tests default to -`assets/51Degrees-LiteIpiV41.ipi`. +`AsnDataFileTests` use. The linked files are not stored in a repository. +Download them from the root of this repository by running: + +``` +sh cxx/data/get-data-files.sh ip-intelligence +``` + +or, in PowerShell: + +``` +./cxx/data/get-data-files.ps1 ip-intelligence +``` + +An example whose file is missing runs the script for that file itself. The +on-premise unit tests default to `assets/51Degrees-LiteIpiV41.ipi`. For the full set of properties (location, registration, etc.) an enterprise data file is required. To obtain one for testing, please @@ -59,12 +70,12 @@ data file is required. To obtain one for testing, please line argument to the examples, or place it within the project space. Note: the engine reads exactly one data file format version, set by -`FIFTYONE_DEGREES_IPI_TARGET_VERSION_MAJOR` and `_MINOR` in the -`ip-intelligence-cxx` submodule's `src/ipi.c`, and refuses any other as -"an unsupported version". Move the `ip-intelligence-cxx` and -`ip-intelligence-data` pins together. `CommittedDataFileTests` in +`FIFTYONE_DEGREES_IPI_TARGET_VERSION_MAJOR` and `_MINOR` in the `cxx` +submodule's `src/ipi/ipi.c`, and refuses any other as "an unsupported +version". The ASN data file is downloaded from the Distributor, which always +serves the latest one published, so `AsnDataFileTests` in [test_ipintelligence.py](fiftyone_ipintelligence_onpremise/tests/test_ipintelligence.py) -fails when the two disagree. +fails when the engine cannot read the format 51Degrees publishes. ## Folders @@ -77,8 +88,6 @@ fails when the two disagree. wraps ip-intelligence-cxx via SWIG. - `fiftyone_ipintelligence_shared` - code shared between the engines. - `fiftyone_ipintelligence_examples` - usage examples for both engines. -- `ip-intelligence-data` - submodule containing the test data file and - evidence files used by the examples and tests. ## Installation diff --git a/location/ci/README.md b/location/ci/README.md index 8e1cb35c..399dcd50 100644 --- a/location/ci/README.md +++ b/location/ci/README.md @@ -4,7 +4,7 @@ This API complies with the `common-ci` approach. The following secrets are required: * `ACCESS_TOKEN` - GitHub [access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#about-personal-access-tokens) for cloning repos, creating PRs, etc. * Example: `github_pat_l0ng_r4nd0m_s7r1ng` -* `SUPER_RESOURCE_KEY` - [resource key](https://51degrees.com/documentation/_info__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=ci-readme.md&utm_term=api-specific-ci-cd-approach) for integration tests +* `SUPER_RESOURCE_KEY` - [resource key](https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=ci-readme.md&utm_term=api-specific-ci-cd-approach) for integration tests * Example: `R4nd0m-S7r1ng` * `PYPI_TOKEN` - [PyPI token](https://pypi.org/help/#apitoken) for publishing packages * Example: `pypi-sUp3r-l0nG_r4nd0m-s7r1Ng` diff --git a/location/examples/cloud/gettingstarted.py b/location/examples/cloud/gettingstarted.py index f5a4041c..a0052c3f 100644 --- a/location/examples/cloud/gettingstarted.py +++ b/location/examples/cloud/gettingstarted.py @@ -77,4 +77,4 @@ if flow_data.location.country.has_value(): print(flow_data.location.country.value()) else: - print(flow_data.location.no_value_message()) + print(flow_data.location.country.no_value_message()) diff --git a/location/readme.md b/location/readme.md index 5077f856..e3c7ee9d 100644 --- a/location/readme.md +++ b/location/readme.md @@ -1,9 +1,9 @@ # 51Degrees Geo-Location Engines -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=51degrees-geo-location-engines) +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_location&utm_term=51degrees-geo-location-engines) **v4 Location Python** -[Developer Documentation](https://51degrees.com/location-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=51degrees-geo-location-engines "Developer documentation") +[Developer Documentation](https://51degrees.com/location-python/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_location&utm_term=51degrees-geo-location-engines "Developer documentation") ## Introduction @@ -13,8 +13,8 @@ The Pipeline is a generic web request intelligence and data processing solution ## Dependencies -For runtime dependencies, see our [dependencies](https://51degrees.com/documentation/_info__dependencies.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=dependencies) page. -The [tested versions](https://51degrees.com/documentation/_info__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=dependencies) page shows the Python versions that we currently test against. The software may run fine against other versions, but additional caution should be applied. +For runtime dependencies, see our [dependencies](https://51degrees.com/documentation/_info__dependencies.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_location&utm_term=dependencies) page. +The [tested versions](https://51degrees.com/documentation/_product_summaries__tested_versions.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_location&utm_term=dependencies) page shows the Python versions that we currently test against. The software may run fine against other versions, but additional caution should be applied. ## Installation and Examples @@ -24,7 +24,7 @@ The [tested versions](https://51degrees.com/documentation/_info__tested_versions You can confirm this is working with the following micro-example. -* Create a resource key for free with the 51Degrees [configurator](https://configure.51degrees.com/6CTsmbPx?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=from-pypi). This defines the properties you want to access. +* Create a resource key for free with the 51Degrees [configurator](https://configure.51degrees.com/6CTsmbPx?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_location&utm_term=from-pypi). This defines the properties you want to access. * On the 'implement' page of the configurator, copy the resource key and replace YOUR_RESOURCE_KEY in the example below. Save this as examplelocation.py * Run the example with `python examplelocation.py` * Feel free to try different locations and property values. @@ -61,3 +61,10 @@ Execute `export FLASK_APP=web` where `web` is the example file, and start your a Execute `$env:FLASK_APP = "web"` where `web` is the example file, and start your application via `flask run`. +## Find out more + +- [Geolocation](https://51degrees.com/geolocation?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_location&utm_term=find-out-more-geolocation) +- [Reverse geocoding documentation](https://51degrees.com/documentation/_reverse_geocoding__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_location&utm_term=find-out-more-documentation) +- [Getting started](https://51degrees.com/documentation/_reverse_geocoding__quickstart.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_location&utm_term=find-out-more-getting-started) +- [Pricing](https://51degrees.com/pricing?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_location&utm_term=find-out-more-pricing) +- https://github.com/51Degrees/51degrees-python diff --git a/location/setup.py b/location/setup.py index 9c8df34b..b7e00ecd 100644 --- a/location/setup.py +++ b/location/setup.py @@ -41,7 +41,11 @@ def read(file_name): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/geolocation?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_location&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_reverse_geocoding__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_location&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("51Degrees Geolocation performs reverse geocoding (using longitude and latitude information) to populate postal addresses."), long_description=read("readme.md"), long_description_content_type='text/markdown', diff --git a/pipeline/ci/README.md b/pipeline/ci/README.md index 8e1cb35c..399dcd50 100644 --- a/pipeline/ci/README.md +++ b/pipeline/ci/README.md @@ -4,7 +4,7 @@ This API complies with the `common-ci` approach. The following secrets are required: * `ACCESS_TOKEN` - GitHub [access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#about-personal-access-tokens) for cloning repos, creating PRs, etc. * Example: `github_pat_l0ng_r4nd0m_s7r1ng` -* `SUPER_RESOURCE_KEY` - [resource key](https://51degrees.com/documentation/_info__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=ci-readme.md&utm_term=api-specific-ci-cd-approach) for integration tests +* `SUPER_RESOURCE_KEY` - [resource key](https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=ci-readme.md&utm_term=api-specific-ci-cd-approach) for integration tests * Example: `R4nd0m-S7r1ng` * `PYPI_TOKEN` - [PyPI token](https://pypi.org/help/#apitoken) for publishing packages * Example: `pypi-sUp3r-l0nG_r4nd0m-s7r1Ng` diff --git a/pipeline/fiftyone_pipeline_cloudrequestengine/readme.md b/pipeline/fiftyone_pipeline_cloudrequestengine/readme.md index 5d82e16d..5979c0e8 100644 --- a/pipeline/fiftyone_pipeline_cloudrequestengine/readme.md +++ b/pipeline/fiftyone_pipeline_cloudrequestengine/readme.md @@ -1,8 +1,8 @@ # Cloud Request Engine -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine-readme.md&utm_term=cloud-request-engine "Data rewards the curious") **Python Pipeline Cloud Request Engine** +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine&utm_term=cloud-request-engine "Data rewards the curious") **Python Pipeline Cloud Request Engine** -[Developer Documentation](https://51degrees.com/pipeline-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine-readme.md&utm_term=cloud-request-engine "Developer Documentation") +[Developer Documentation](https://51degrees.com/pipeline-python/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine&utm_term=cloud-request-engine "Developer Documentation") ## Introduction @@ -16,7 +16,7 @@ The Pipeline is a generic web request intelligence and data processing solution This package uses the `engines` class created by the `fiftyone-pipeline-engines`. It makes available: -* A `Cloud Request Engine` which calls the 51Degrees cloud service to fetch properties and metadata about them based on a provided resource key. Get a resource key at https://configure.51degrees.com/?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine-readme.md&utm_term=this-package-fiftyone_pipeline_cloudrequestengine +* A `Cloud Request Engine` which calls the 51Degrees cloud service to fetch properties and metadata about them based on a provided resource key. Get a resource key at https://configure.51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine&utm_term=this-package-fiftyone_pipeline_cloudrequestengine * A `Cloud Engine` template which reads data from the Cloud Request Engine. ## Credentials @@ -79,3 +79,11 @@ It is used by the cloud versions of the following 51Degrees engines: - [**fiftyone_devicedetection**](https://pypi.org/project/fiftyone-devicedetection/) - Get details about the devices accessing your web page - [**fiftyone_location**](https://pypi.org/project/fiftyone-location/) - Get postal address details from the location of devices accessing your web page + +## Find out more + +- [Build your own](https://51degrees.com/build-your-own?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine&utm_term=find-out-more-build-your-own) +- [Pipeline API documentation](https://51degrees.com/documentation/_pipeline_api__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine&utm_term=find-out-more-documentation) +- [Pipeline API examples](https://51degrees.com/documentation/_pipeline_api__examples__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine&utm_term=find-out-more-examples) +- [Cloud service documentation](https://51degrees.com/documentation/_services__cloud__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine&utm_term=find-out-more-cloud-service) +- https://github.com/51Degrees/51degrees-python diff --git a/pipeline/fiftyone_pipeline_cloudrequestengine/setup.py b/pipeline/fiftyone_pipeline_cloudrequestengine/setup.py index 434c0a25..223d23d8 100644 --- a/pipeline/fiftyone_pipeline_cloudrequestengine/setup.py +++ b/pipeline/fiftyone_pipeline_cloudrequestengine/setup.py @@ -42,7 +42,11 @@ def read(file_name): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/build-your-own?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_pipeline_api__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("This package contains a shared engine that is used to make requests to the 51Degrees cloud service on behalf of other data services such as device detection or location."), long_description=read("readme.md"), long_description_content_type='text/markdown', diff --git a/pipeline/fiftyone_pipeline_cloudrequestengine/src/fiftyone_pipeline_cloudrequestengine/clouddata.py b/pipeline/fiftyone_pipeline_cloudrequestengine/src/fiftyone_pipeline_cloudrequestengine/clouddata.py index d28a9c86..4f4f8c88 100644 --- a/pipeline/fiftyone_pipeline_cloudrequestengine/src/fiftyone_pipeline_cloudrequestengine/clouddata.py +++ b/pipeline/fiftyone_pipeline_cloudrequestengine/src/fiftyone_pipeline_cloudrequestengine/clouddata.py @@ -42,7 +42,7 @@ def check(self, key, flow_element): + flow_element.datakey + " are " + ", ".join(list(flow_element.get_properties().keys())) + ". For more details on resource keys, see our explainer: " - "https://51degrees.com/documentation/_info__resource_keys.html?utm_source=code&utm_medium=comment&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine-src-fiftyone_pipeline_cloudrequestengine-clouddata.py&utm_term=property-not-found") + "https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=code&utm_medium=comment&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine-src-fiftyone_pipeline_cloudrequestengine-clouddata.py&utm_term=property-not-found") class CloudData(AspectDataDictionary): diff --git a/pipeline/fiftyone_pipeline_cloudrequestengine/src/fiftyone_pipeline_cloudrequestengine/cloudengine.py b/pipeline/fiftyone_pipeline_cloudrequestengine/src/fiftyone_pipeline_cloudrequestengine/cloudengine.py index 2d872d16..9f285f75 100644 --- a/pipeline/fiftyone_pipeline_cloudrequestengine/src/fiftyone_pipeline_cloudrequestengine/cloudengine.py +++ b/pipeline/fiftyone_pipeline_cloudrequestengine/src/fiftyone_pipeline_cloudrequestengine/cloudengine.py @@ -73,7 +73,7 @@ def on_registration(self, pipeline): # key in use, a resource key or a licence key, so a section missing # from them means that key does not carry the product at all. if not self.datakey in pipeline.flow_elements_list["cloud"].flow_element_properties: - raise Exception("The key in use does not include access to any properties under the engine with key " + self.datakey + " that was added to the pipeline. For more details on resource keys, see our explainer: https://51degrees.com/documentation/_info__resource_keys.html?utm_source=code&utm_medium=comment&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine-src-fiftyone_pipeline_cloudrequestengine-cloudengine.py&utm_term=engine-properties-not-accessible " + "Available engine data keys are: " + str([e for e in pipeline.flow_elements_list["cloud"].flow_element_properties])) + raise Exception("The key in use does not include access to any properties under the engine with key " + self.datakey + " that was added to the pipeline. For more details on resource keys, see our explainer: https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=code&utm_medium=comment&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine-src-fiftyone_pipeline_cloudrequestengine-cloudengine.py&utm_term=engine-properties-not-accessible " + "Available engine data keys are: " + str([e for e in pipeline.flow_elements_list["cloud"].flow_element_properties])) self.properties = pipeline.flow_elements_list["cloud"].flow_element_properties[self.datakey] @@ -95,7 +95,7 @@ def pipeline_element_not_found(self, element, flowdata): """ - raise Exception("The key in use does not include access to any properties under " + element + ". For more details on resource keys, see our explainer: https://51degrees.com/documentation/_info__resource_keys.html?utm_source=code&utm_medium=comment&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine-src-fiftyone_pipeline_cloudrequestengine-cloudengine.py&utm_term=element-properties-not-accessible " + "Available element data keys are: " + str([e for e in flowdata.pipeline.flow_elements_display_list])) + raise Exception("The key in use does not include access to any properties under " + element + ". For more details on resource keys, see our explainer: https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=code&utm_medium=comment&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_cloudrequestengine-src-fiftyone_pipeline_cloudrequestengine-cloudengine.py&utm_term=element-properties-not-accessible " + "Available element data keys are: " + str([e for e in flowdata.pipeline.flow_elements_display_list])) def process_internal(self, flowdata): diff --git a/pipeline/fiftyone_pipeline_core/readme.md b/pipeline/fiftyone_pipeline_core/readme.md index 078967db..ad3470f1 100644 --- a/pipeline/fiftyone_pipeline_core/readme.md +++ b/pipeline/fiftyone_pipeline_core/readme.md @@ -1,8 +1,8 @@ # Pipeline Core -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_core-readme.md&utm_term=pipeline-core "Data rewards the curious") **Python Pipeline Core** +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_core&utm_term=pipeline-core "Data rewards the curious") **Python Pipeline Core** -[Developer Documentation](https://51degrees.com/pipeline-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_core-readme.md&utm_term=pipeline-core "Developer Documentation") +[Developer Documentation](https://51degrees.com/pipeline-python/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_core&utm_term=pipeline-core "Developer Documentation") ## Introduction @@ -24,4 +24,11 @@ Engines created by 51Degrees: ## Requirements * Python 3.8+ -* The `flask` python library to run the web examples +* The `flask` python library to run the web examples + +## Find out more + +- [Build your own](https://51degrees.com/build-your-own?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_core&utm_term=find-out-more-build-your-own) +- [Pipeline API documentation](https://51degrees.com/documentation/_pipeline_api__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_core&utm_term=find-out-more-documentation) +- [Pipeline API examples](https://51degrees.com/documentation/_pipeline_api__examples__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_core&utm_term=find-out-more-examples) +- https://github.com/51Degrees/51degrees-python diff --git a/pipeline/fiftyone_pipeline_core/setup.py b/pipeline/fiftyone_pipeline_core/setup.py index a79339f4..1da2b014 100644 --- a/pipeline/fiftyone_pipeline_core/setup.py +++ b/pipeline/fiftyone_pipeline_core/setup.py @@ -41,7 +41,11 @@ def read(file_name): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_core-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/build-your-own?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_core&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_pipeline_api__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_core&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("This package defines the essential components of the 51Degrees Pipeline API such as flow elements, flow data and evidence. It also packages together JavaScript served by a pipeline and allows for client side requests for additional data populated by evidence from the client side."), long_description=read("readme.md"), long_description_content_type='text/markdown', diff --git a/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/callback_url.py b/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/callback_url.py new file mode 100644 index 00000000..afafa240 --- /dev/null +++ b/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/callback_url.py @@ -0,0 +1,274 @@ +# ********************************************************************* +# This Original Work is copyright of 51 Degrees Mobile Experts Limited. +# Copyright 2026 51 Degrees Mobile Experts Limited, Davidson House, +# Forbury Square, Reading, Berkshire, United Kingdom RG1 3EU. +# +# This Original Work is licensed under the European Union Public Licence +# (EUPL) v.1.2 and is subject to its terms as set out below. +# +# If a copy of the EUPL was not distributed with this file, You can obtain +# one at https://opensource.org/licenses/EUPL-1.2. +# +# The 'Compatible Licences' set out in the Appendix to the EUPL (as may be +# amended by the European Commission) shall be deemed incompatible for +# the purposes of the Work and the provisions of the compatibility +# clause in Article 5 of the EUPL shall not apply. +# +# If using the Work as, or as part of, a network application, by +# including the attribution notice(s) required under Article 5 of the EUPL +# in the end user terms of the application under an appropriate heading, +# such notice(s) shall fulfill the requirements of that article. +# ********************************************************************* + +"""! +The callback URL the client script posts to for updated JSON. The protocol +and host usually come from the request, and the script holds the URL in a +single-quoted string, so the URL is checked before it is written into the +script. A URL that is not valid is refused. In the user information, in +anything after the host and port, and in the endpoint, the characters a URL +cannot hold or that could end the string or the script are percent-encoded. + +See https://github.com/51Degrees/specifications/blob/main/pipeline-specification/pipeline-elements/javascript-builder.md#callback-url +""" + +import re +from collections import namedtuple + +# Whether the URL can be written into the script, and the URL. That is the +# URL to write into the script when it is valid, and otherwise the URL as it +# was built, for the log. +CallbackUrl = namedtuple("CallbackUrl", ["valid", "url"]) + +# The largest port number. +MAX_PORT = 65535 + +# The most digits a port can have once its leading zeros are dropped. +MAX_PORT_DIGITS = 5 + +# The printable ASCII characters that are percent-encoded, besides the +# control characters, the space and everything outside ASCII. +ENCODED_CHARACTERS = frozenset(b"\"'<>\\^`{|}") + +HEX_DIGITS = frozenset(b"0123456789ABCDEFabcdef") + +# Text made only of the Unicode White_Space characters, which counts as a +# part that is not set. The first range is tab, line feed, vertical tab, +# form feed and carriage return. +BLANK_PATTERN = re.compile( + r"[\t-\r \u0085\u00a0\u1680\u2000-\u200a" + r"\u2028\u2029\u202f\u205f\u3000]*") + +# A scheme as RFC 3986 section 3.1 defines one, being a letter followed by +# letters, digits, '+', '-' and '.'. +SCHEME_PATTERN = re.compile(r"[A-Za-z][A-Za-z0-9+.-]*") + +# Where the authority ends, which is at the first '/', '?' or '#' of the +# host. +AUTHORITY_END_PATTERN = re.compile(r"[/?#]") + +# A label of a host name, being ASCII letters, digits, '-' and '_'. +LABEL_PATTERN = re.compile(r"[A-Za-z0-9_-]+") + +# An IPv6 zone identifier, being unreserved characters as RFC 3986 section +# 2.3 defines them. +ZONE_PATTERN = re.compile(r"[A-Za-z0-9._~-]+") + +# A port as it follows the host, being ':' and any number of digits. +PORT_PATTERN = re.compile(r":[0-9]*") + +# One group of an IPv6 address. +HEX_GROUP_PATTERN = re.compile(r"[0-9A-Fa-f]{1,4}") + +# An IPv4 address as RFC 3986 section 3.2.2 defines one, being four numbers +# from 0 to 255 with no leading zeros. +_DEC_OCTET = r"(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])" +IPV4_PATTERN = re.compile(_DEC_OCTET + r"(\." + _DEC_OCTET + r"){3}") + +# A lone surrogate, which has no UTF-8 encoding, so it is written as U+FFFD, +# as a browser writes it. +SURROGATE_PATTERN = re.compile(r"[\ud800-\udfff]") + + +def is_blank(text): + + """! + Whether a part of the URL is not set. + + @type text: object + @param text: The part + @rtype: bool + @return: True if the part is not a string, or is empty or only + whitespace + """ + + return (not isinstance(text, str) + or BLANK_PATTERN.fullmatch(text) is not None) + + +def callback_url(protocol, host, endpoint): + + """! + Build the callback URL from its parts and check it. One slash separates + the host and the endpoint, being added when neither has one and removed + when both do. The protocol and the host are written in lower case and + the rest as given, apart from the percent-encoding. + + @type protocol: str + @param protocol: The protocol, such as https + @type host: str + @param host: The host, which can start with user information and end + with a port, a path, a query or a fragment + @type endpoint: str + @param endpoint: The endpoint + @rtype: CallbackUrl + @return: The callback URL, or None when the protocol, host or endpoint + is not set + """ + + if is_blank(protocol) or is_blank(host) or is_blank(endpoint): + return None + host_has_slash = host.endswith("/") + endpoint_has_slash = endpoint.startswith("/") + if not host_has_slash and not endpoint_has_slash: + endpoint = "/" + endpoint + elif host_has_slash and endpoint_has_slash: + endpoint = endpoint[1:] + built = protocol + "://" + host + endpoint + if SCHEME_PATTERN.fullmatch(protocol) is None: + return CallbackUrl(False, built) + match = AUTHORITY_END_PATTERN.search(host) + end = len(host) if match is None else match.start() + authority = _checked_authority(host[:end]) + if authority is None: + return CallbackUrl(False, built) + return CallbackUrl( + True, + protocol.lower() + "://" + authority + + _encoded(host[end:]) + _encoded(endpoint)) + + +def _checked_authority(authority): + + """! + The user information, host and port as they are written into the URL, + or None when they are not valid. The user information ends at the first + '@', so a second '@' is part of the host and refused. + """ + + user, at, host_port = authority.partition("@") + if not at: + return _checked_host_port(authority) + host_port = _checked_host_port(host_port) + if host_port is None: + return None + return _encoded(user) + "@" + host_port + + +def _checked_host_port(host_port): + + """! + The host and optional port in lower case, or None when they are not + valid. The host is an IPv6 address in brackets with an optional zone + identifier, or a name of labels separated by dots with a dot allowed at + the end. The port is ':' followed by digits up to 65535, or by nothing. + """ + + if host_port.startswith("["): + close = host_port.find("]") + if close == -1: + return None + address, separator, zone = host_port[1:close].partition("%25") + if separator and ZONE_PATTERN.fullmatch(zone) is None: + return None + if not _is_ipv6(address): + return None + host = host_port[:close + 1] + port = host_port[close + 1:] + else: + host, colon, port = host_port.partition(":") + port = colon + port + name = host[:-1] if host.endswith(".") else host + for label in name.split("."): + if LABEL_PATTERN.fullmatch(label) is None: + return None + if port and not _is_port(port): + return None + return host.lower() + port + + +def _is_port(text): + + """! + Whether the text is ':' followed by digits whose value is no more than + 65535, or by nothing. + """ + + if PORT_PATTERN.fullmatch(text) is None: + return False + digits = text[1:].lstrip("0") + return len(digits) <= MAX_PORT_DIGITS and ( + not digits or int(digits) <= MAX_PORT) + + +def _is_ipv6(text): + + """! + Whether the text is an IPv6 address as RFC 3986 section 3.2.2 defines + one. That is eight groups of one to four hexadecimal digits separated by + ':', where one '::' can stand for one or more groups of zeros and the + last two groups can be written as an IPv4 address. + """ + + gap = text.find("::") + if gap == -1: + return _groups(text, True) == 8 + if text.find("::", gap + 1) != -1: + return False + head = _groups(text[:gap], False) + tail = _groups(text[gap + 2:], True) + return head >= 0 and tail >= 0 and head + tail <= 7 + + +def _groups(run, ipv4_at_end): + + """! + The number of 16 bit groups in a run of groups separated by ':', or -1 + when the run is not valid. An empty run has none, and an IPv4 address at + the end of the run, where one is allowed, counts as two. + """ + + if not run: + return 0 + parts = run.split(":") + count = 0 + for i, part in enumerate(parts): + if HEX_GROUP_PATTERN.fullmatch(part): + count += 1 + elif (ipv4_at_end and i == len(parts) - 1 + and IPV4_PATTERN.fullmatch(part)): + count += 2 + else: + return -1 + return count + + +def _encoded(text): + + """! + The text with each byte of its UTF-8 encoding percent-encoded where the + byte is a control character, a space, outside ASCII or one of + ENCODED_CHARACTERS, and with a '%' encoded where two hexadecimal digits + do not follow it. + """ + + data = SURROGATE_PATTERN.sub("\ufffd", text).encode("utf-8") + out = [] + for i, b in enumerate(data): + if b == 0x25: + keep = (i + 2 < len(data) + and data[i + 1] in HEX_DIGITS + and data[i + 2] in HEX_DIGITS) + else: + keep = 0x20 < b < 0x7F and b not in ENCODED_CHARACTERS + out.append(chr(b) if keep else "%{:02X}".format(b)) + return "".join(out) diff --git a/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/flowdata.py b/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/flowdata.py index daa6b387..c6f83c7c 100644 --- a/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/flowdata.py +++ b/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/flowdata.py @@ -94,6 +94,34 @@ def process(self): return self + def close(self): + + """! + Release the native resources held by the element data in this + FlowData, such as the results of an on-premise engine, which keep + the engine's data in memory until they are released. Use a with + statement so that this happens as soon as the FlowData is finished + with, rather than when the garbage collector next runs. + + Only element data with a close method of its own is closed, so + element data that needs no cleanup, such as an entry in a results + cache, is left alone. Values already read are unaffected. + + """ + + for element_data in list(self.data.values()): + # Looked up on the class, because ElementData treats an + # attribute it does not have as the name of a property. + close = getattr(type(element_data), "close", None) + if callable(close): + close(element_data) + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc_value, exc_traceback): + self.close() + def get_from_element(self, flow_element): """! diff --git a/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/flowelement.py b/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/flowelement.py index 87ee2800..73e6922d 100644 --- a/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/flowelement.py +++ b/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/flowelement.py @@ -77,6 +77,17 @@ def on_registration(self, pipeline): pass + def close(self): + + """! + Release the resources the element holds. The pipeline calls this + when it is closed. This element holds none, and elements that hold + native data or threads override it. + + """ + + pass + def get_evidence_key_filter(self): """! diff --git a/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/javascriptbuilder.py b/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/javascriptbuilder.py index 1f9e206b..31d063a7 100644 --- a/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/javascriptbuilder.py +++ b/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/javascriptbuilder.py @@ -38,6 +38,7 @@ from .evidence_keyfilter import EvidenceKeyFilter from .elementdata_dictionary import ElementDataDictionary from .constants import Constants +from .callback_url import callback_url # Evidence keys the rendered script is not configured with. Both are @@ -295,6 +296,10 @@ def __init__(self, settings = {} ): self.exclude_from_messages = True + # Set once a callback URL that is not valid has been logged, because + # a misconfigured element would otherwise log on every request. + self._invalid_url_logged = False + # Load template file contents into memory template = Path(__file__).absolute().parent / "js_templates" / "JavaScriptResource.mustache" @@ -413,27 +418,24 @@ def process_internal(self, flowdata): variables["_parameters"] = json.dumps( variables["_parameters"], separators=(",", ":")) - if variables["_host"] and variables["_protocol"] and variables["_endpoint"]: - - # The callback URL is the protocol, the host and the endpoint, - # with exactly one slash between the host and the endpoint, as - # in the .NET builder. No query values are added to it. They - # reach the callback in the request body through the parameters - # above, and the script appends the session id and the sequence - # itself. Adding every evidence value here, as this builder used - # to, sent those two twice and, because the evidence reaching - # this element is not narrowed to query values, also wrote the - # end user's User-Agent, IP address and host headers into the - # served script's URL. - host = variables["_host"] - endpoint = variables["_endpoint"] - if not endpoint.startswith("/") and not host.endswith("/"): - endpoint = "/" + endpoint - elif endpoint.startswith("/") and host.endswith("/"): - endpoint = endpoint[1:] - - variables["_url"] = variables["_protocol"] + "://" + host + endpoint - + # The callback URL is the protocol, the host and the endpoint, as in + # the .NET builder, and callback_url checks it. No query values are + # added to it, because they reach the callback in the request body + # through the parameters above, and the script appends the session + # id and the sequence itself. + url = callback_url( + variables["_protocol"], variables["_host"], variables["_endpoint"]) + if url is not None and not url.valid and not self._invalid_url_logged: + self._invalid_url_logged = True + self._log( + "error", + "The generated URL " + repr(url.url) + " is not valid, so " + "the JavaScript has no callback URL. Check the JavaScript " + "builder's protocol, host and endpoint, and the protocol and " + "host evidence. This is reported once.") + + if url is not None and url.valid: + variables["_url"] = url.url variables["_updateEnabled"] = True else: variables["_updateEnabled"] = False diff --git a/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/js_templates b/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/js_templates index d3f96049..c773ce22 160000 --- a/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/js_templates +++ b/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/js_templates @@ -1 +1 @@ -Subproject commit d3f96049bde389951c79638047b8ea80dfdd5cdf +Subproject commit c773ce221ccec6fcb67fbdd5ad2175a66fab11d6 diff --git a/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/pipeline.py b/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/pipeline.py index cac6aea1..4ee68108 100644 --- a/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/pipeline.py +++ b/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/pipeline.py @@ -33,7 +33,7 @@ class Pipeline: """ - def __init__(self, flow_elements, logger=Logger(), suppress_process_exceptions = False, data_file_update_service = None): + def __init__(self, flow_elements, logger=Logger(), suppress_process_exceptions = False, data_file_update_service = None, auto_close_elements = True): """! Pipeline constructor. @@ -49,6 +49,12 @@ def __init__(self, flow_elements, logger=Logger(), suppress_process_exceptions = exceptions added to FlowData.errors otherwise will throw the exception occurred during the processing of first element. + @type auto_close_elements: boolean + @param auto_close_elements: If true, the default, closing the + pipeline closes its flow elements. Set it to false when the elements + are shared with another pipeline, and close each element once every + pipeline using it is finished with. + @rtype: Pipeline @return: Returns a Pipeline @@ -63,6 +69,8 @@ def __init__(self, flow_elements, logger=Logger(), suppress_process_exceptions = self.flow_elements_display_list = [] self.suppress_process_exceptions = suppress_process_exceptions + + self.auto_close_elements = auto_close_elements if data_file_update_service != None: self.data_file_update_service = data_file_update_service @@ -92,6 +100,44 @@ def create_flowdata(self): return FlowData(self) + def close(self): + + """! + Close the flow elements, which releases the native data they hold + and stops the updates of their data files, unless the pipeline was + built with auto_close_elements set to false. Use a with statement + so that this happens when the pipeline is finished with. + + FlowData created by the pipeline are not closed by it. Close each + one as well, because the results it holds keep the engine's data in + memory until they are released. + + """ + + if not self.auto_close_elements: + return + + # Every element is closed even if one fails, and the first failure + # is raised once they all have been. + failure = None + for flow_element in self.flow_elements: + close = getattr(flow_element, "close", None) + if close is None: + continue + try: + close() + except Exception as e: + if failure is None: + failure = e + if failure is not None: + raise failure + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc_value, exc_traceback): + self.close() + def log(self, level, message): """! diff --git a/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/pipelinebuilder.py b/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/pipelinebuilder.py index 72f89752..c3fa52e0 100644 --- a/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/pipelinebuilder.py +++ b/pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/pipelinebuilder.py @@ -52,6 +52,9 @@ def __init__(self, settings= {}): while processing is re-raised from flow_data.process(); if True, errors are stored on flow_data.errors and process() returns normally. Recommended True for web apps so a processing failure degrades gracefully instead of failing the request. + `auto_close_elements (Bool)` - Whether closing the pipeline closes its flow elements + (default true). Set to False when the elements are shared with another pipeline, + and close each element once every pipeline using it is finished with. @return: Returns a Pipeline Builder """ @@ -82,6 +85,11 @@ def __init__(self, settings= {}): else: self.suppress_process_exceptions = False + if "auto_close_elements" in settings: + self.auto_close_elements = settings["auto_close_elements"] + else: + self.auto_close_elements = True + def get_javascript_elements(self): """ Adds the JavaScriptBuilder, JSONBundler and SequenceElement to the pipeline if @@ -148,7 +156,7 @@ def build(self): self.flow_elements.extend(self.get_javascript_elements()) self.flow_elements.extend(self.get_setheader_elements()) - return Pipeline(self.flow_elements, logger=self.logger, suppress_process_exceptions=self.suppress_process_exceptions, data_file_update_service=self.data_file_update_service) + return Pipeline(self.flow_elements, logger=self.logger, suppress_process_exceptions=self.suppress_process_exceptions, data_file_update_service=self.data_file_update_service, auto_close_elements=self.auto_close_elements) def add_logger(self, logger): """ @@ -176,27 +184,46 @@ def build_from_configuration(self, config): @rtype: Pipeline @return: Returns built pipeline + If an element fails to build, the elements already built are closed + and the error is raised. + """ flow_elements = [] - for element in config["PipelineOptions"]["Elements"]: - flow_element = importlib.import_module(element["elementPath"]).__getattribute__(element["elementName"]) - if("elementParameters" in element): - arguments = self._map_properties_names( - { - "verify_md5": "data_update_verify_md5", - "data_file_update_base_url": "data_update_url" - }, - element["elementParameters"], - ) - - flow_element = flow_element(**arguments) - else: - flow_element = flow_element() - flow_elements.append(flow_element) - - return Pipeline(flow_elements, suppress_process_exceptions=self._get_suppress_process_exceptions(config)) + try: + for element in config["PipelineOptions"]["Elements"]: + flow_element = importlib.import_module(element["elementPath"]).__getattribute__(element["elementName"]) + if("elementParameters" in element): + arguments = self._map_properties_names( + { + "verify_md5": "data_update_verify_md5", + "data_file_update_base_url": "data_update_url" + }, + element["elementParameters"], + ) + + flow_element = flow_element(**arguments) + else: + flow_element = flow_element() + flow_elements.append(flow_element) + + return Pipeline(flow_elements, suppress_process_exceptions=self._get_suppress_process_exceptions(config), auto_close_elements=self.auto_close_elements) + except BaseException: + # A failure to close is logged rather than raised, so the caller + # sees the error that stopped the build. + for flow_element in flow_elements: + close = getattr(flow_element, "close", None) + if close is None: + continue + try: + close() + except Exception as e: + self.logger.log( + "error", + "Could not close {} after the pipeline failed to " + "build: {}".format(type(flow_element).__name__, e)) + raise def _get_suppress_process_exceptions(self, config): """ diff --git a/pipeline/fiftyone_pipeline_core/tests/test_callback_url.py b/pipeline/fiftyone_pipeline_core/tests/test_callback_url.py new file mode 100644 index 00000000..9048401d --- /dev/null +++ b/pipeline/fiftyone_pipeline_core/tests/test_callback_url.py @@ -0,0 +1,200 @@ +# ********************************************************************* +# This Original Work is copyright of 51 Degrees Mobile Experts Limited. +# Copyright 2026 51 Degrees Mobile Experts Limited, Davidson House, +# Forbury Square, Reading, Berkshire, United Kingdom RG1 3EU. +# +# This Original Work is licensed under the European Union Public Licence +# (EUPL) v.1.2 and is subject to its terms as set out below. +# +# If a copy of the EUPL was not distributed with this file, You can obtain +# one at https://opensource.org/licenses/EUPL-1.2. +# +# The 'Compatible Licences' set out in the Appendix to the EUPL (as may be +# amended by the European Commission) shall be deemed incompatible for +# the purposes of the Work and the provisions of the compatibility +# clause in Article 5 of the EUPL shall not apply. +# +# If using the Work as, or as part of, a network application, by +# including the attribution notice(s) required under Article 5 of the EUPL +# in the end user terms of the application under an appropriate heading, +# such notice(s) shall fulfill the requirements of that article. +# ********************************************************************* + +""" +Checks the callback URL against the cases every implementation of the +JavaScript builder shares, in the same order, so that each one refuses and +writes the same URLs. +""" + +import unittest + +from parameterized import parameterized + +from fiftyone_pipeline_core.callback_url import CallbackUrl, callback_url + +ENDPOINT = "/51dpipeline/json" + + +def _url(protocol, host): + return callback_url(protocol, host, ENDPOINT) + + +def _valid(text): + return CallbackUrl(True, text) + + +class CallbackUrlTests(unittest.TestCase): + + def test_adds_a_single_slash(self): + self.assertEqual( + _valid("https://example.com/51dpipeline/json"), + callback_url("https", "example.com", "51dpipeline/json")) + + def test_keeps_a_single_slash_on_the_endpoint(self): + self.assertEqual( + _valid("https://example.com/51dpipeline/json"), + callback_url("https", "example.com", "/51dpipeline/json")) + + def test_collapses_a_double_slash(self): + self.assertEqual( + _valid("http://example.com/json"), + callback_url("http", "example.com/", "/json")) + + def test_missing_when_a_part_is_empty(self): + self.assertIsNone(callback_url("https", "", "/json")) + self.assertIsNone(callback_url("", "example.com", "/json")) + self.assertIsNone(callback_url("https", "example.com", "")) + self.assertIsNone(callback_url("https", " ", "/json")) + + @parameterized.expand([ + ("EXAMPLE.com", "https://example.com/51dpipeline/json"), + ("example.com:8080", "https://example.com:8080/51dpipeline/json"), + ("example.com:", "https://example.com:/51dpipeline/json"), + ("example.com:65535", "https://example.com:65535/51dpipeline/json"), + ("a_b.com", "https://a_b.com/51dpipeline/json"), + ("-ab.com", "https://-ab.com/51dpipeline/json"), + ("127.0.0.1:80", "https://127.0.0.1:80/51dpipeline/json"), + ("[::1]", "https://[::1]/51dpipeline/json"), + ("[::1]:8080", "https://[::1]:8080/51dpipeline/json"), + ("[fe80::1%25eth0]", "https://[fe80::1%25eth0]/51dpipeline/json"), + ("user:pw@example.com", + "https://user:pw@example.com/51dpipeline/json"), + ("u x@example.com", "https://u%20x@example.com/51dpipeline/json"), + ("u\tx@example.com", "https://u%09x@example.com/51dpipeline/json"), + ("example.com.", "https://example.com./51dpipeline/json"), + ("example.com/path", "https://example.com/path/51dpipeline/json"), + ("example.com?x", "https://example.com?x/51dpipeline/json"), + ("example.com/a b", "https://example.com/a%20b/51dpipeline/json"), + ("example.com/a%zzb", + "https://example.com/a%25zzb/51dpipeline/json"), + ("example.com/a%41b", "https://example.com/a%41b/51dpipeline/json"), + ("example.com/\u00fc", + "https://example.com/%C3%BC/51dpipeline/json"), + ("example.com/a\nb", "https://example.com/a%0Ab/51dpipeline/json"), + ("example.com/a\u2028b", + "https://example.com/a%E2%80%A8b/51dpipeline/json"), + ]) + def test_accepts_the_host(self, host, expected): + self.assertEqual(_valid(expected), _url("https", host)) + + @parameterized.expand([ + ("example.com:65536",), + ("example.com:99999",), + ("example.com:abc",), + ("exa mple.com",), + ("example.com');alert(1);//",), + ("a\"b.com",), + ("a'b.com",), + ("a",), + ]) + def test_nothing_given_can_end_the_string(self, host): + """A quote, a backslash or a line break never reaches the script, + wherever in the URL it was given.""" + + result = _url("https", host) + self.assertTrue(result.valid, repr(host)) + for c in "'\"\\<>\n\r\u2028\u2029": + self.assertNotIn(c, result.url, repr(host)) + + def test_the_endpoint_is_encoded(self): + self.assertEqual( + _valid("https://example.com/json%27x"), + callback_url("https", "example.com", "/json'x")) + + @parameterized.expand([ + ("b\u00fccher.de",), + ("a\u00a0b.com",), + ("a\u200bb.com",), + ("a\u2028b.com",), + ]) + def test_refuses_the_host_outside_ascii(self, host): + """A host must be ASCII, so an internationalized name is given as + A-labels, and anything else outside ASCII is refused.""" + + self.assertFalse(_url("https", host).valid, repr(host)) + + def test_writes_a_lone_surrogate_as_the_replacement_character(self): + """Not one of the shared cases, because only some languages' strings + can hold a lone surrogate. It has no UTF-8 encoding, so it is written + as U+FFFD, as a browser writes it.""" + + self.assertEqual( + _valid("https://example.com/%EF%BF%BD/51dpipeline/json"), + _url("https", "example.com/\ud800")) + + +if __name__ == "__main__": + unittest.main() diff --git a/pipeline/fiftyone_pipeline_core/tests/test_close.py b/pipeline/fiftyone_pipeline_core/tests/test_close.py new file mode 100644 index 00000000..e99583d5 --- /dev/null +++ b/pipeline/fiftyone_pipeline_core/tests/test_close.py @@ -0,0 +1,182 @@ +# ********************************************************************* +# This Original Work is copyright of 51 Degrees Mobile Experts Limited. +# Copyright 2026 51 Degrees Mobile Experts Limited, Davidson House, +# Forbury Square, Reading, Berkshire, United Kingdom RG1 3EU. +# +# This Original Work is licensed under the European Union Public Licence +# (EUPL) v.1.2 and is subject to its terms as set out below. +# +# If a copy of the EUPL was not distributed with this file, You can obtain +# one at https://opensource.org/licenses/EUPL-1.2. +# +# The 'Compatible Licences' set out in the Appendix to the EUPL (as may be +# amended by the European Commission) shall be deemed incompatible for +# the purposes of the Work and the provisions of the compatibility +# clause in Article 5 of the EUPL shall not apply. +# +# If using the Work as, or as part of, a network application, by +# including the attribution notice(s) required under Article 5 of the EUPL +# in the end user terms of the application under an appropriate heading, +# such notice(s) shall fulfill the requirements of that article. +# ********************************************************************* + +import unittest + +from fiftyone_pipeline_core.elementdata import ElementData +from fiftyone_pipeline_core.elementdata_dictionary import ElementDataDictionary +from fiftyone_pipeline_core.flowelement import FlowElement +from fiftyone_pipeline_core.pipelinebuilder import PipelineBuilder + + +class ClosingData(ElementData): + """Element data with a close method of its own, as the results of an + on-premise engine have.""" + + def __init__(self, flow_element): + super(ClosingData, self).__init__(flow_element) + self.closed = False + + def close(self): + self.closed = True + + +class ClosingElement(FlowElement): + """Adds ClosingData to each FlowData, and counts its own closes.""" + + def __init__(self): + super(ClosingElement, self).__init__() + self.datakey = "closing" + self.closes = 0 + + def process_internal(self, flowdata): + flowdata.set_element_data(ClosingData(self)) + + def close(self): + self.closes += 1 + + +class PlainElement(FlowElement): + """Adds element data that needs no cleanup.""" + + def __init__(self): + super(PlainElement, self).__init__() + self.datakey = "plain" + + def process_internal(self, flowdata): + flowdata.set_element_data(ElementDataDictionary(self, {"close": 1})) + + +class FailingElement(FlowElement): + """Fails to close.""" + + def __init__(self): + super(FailingElement, self).__init__() + self.datakey = "failing" + + def close(self): + raise Exception("close failed") + + +class RecordedElement(FlowElement): + """Records each instance that is closed, for a test whose pipeline + failed to build and so cannot be reached.""" + + closed = [] + + def __init__(self): + super(RecordedElement, self).__init__() + self.datakey = "recorded" + + def close(self): + RecordedElement.closed.append(self) + + +class BrokenElement(FlowElement): + """Fails to build.""" + + def __init__(self): + raise ValueError("element failed to build") + + +def build(*elements, **settings): + settings["add_javascript_builder"] = False + settings["use_setheader_properties"] = False + builder = PipelineBuilder(settings) + for element in elements: + builder.add(element) + return builder.build() + + +class CloseTests(unittest.TestCase): + + # Element data with a close method is closed with the FlowData, and + # element data without one is left alone, even with a property called + # close. + def test_flowdata_closes_element_data(self): + with build(ClosingElement(), PlainElement()) as pipeline: + with pipeline.create_flowdata() as flowdata: + flowdata.process() + closing = flowdata.get("closing") + self.assertFalse(closing.closed) + self.assertTrue(closing.closed) + self.assertEqual(1, flowdata.get("plain").get("close")) + + # The with statement closes the FlowData when an exception is raised. + def test_flowdata_closed_on_exception(self): + with build(ClosingElement()) as pipeline: + with self.assertRaises(ValueError): + with pipeline.create_flowdata() as flowdata: + flowdata.process() + closing = flowdata.get("closing") + raise ValueError() + self.assertTrue(closing.closed) + + # A FlowData that was never processed holds nothing to close. + def test_unprocessed_flowdata_closes(self): + with build(ClosingElement()) as pipeline: + pipeline.create_flowdata().close() + + def test_pipeline_closes_elements(self): + element = ClosingElement() + with build(element): + self.assertEqual(0, element.closes) + self.assertEqual(1, element.closes) + + # Elements shared with another pipeline are left for the caller to + # close. + def test_pipeline_leaves_elements_open(self): + element = ClosingElement() + with build(element, auto_close_elements=False): + pass + self.assertEqual(0, element.closes) + + # Every element is closed even when one fails, and the failure is + # raised afterwards. + def test_pipeline_closes_every_element(self): + element = ClosingElement() + pipeline = build(FailingElement(), element) + with self.assertRaises(Exception) as context: + pipeline.close() + self.assertEqual("close failed", str(context.exception)) + self.assertEqual(1, element.closes) + + # When an element in a configuration fails to build, the elements built + # before it are closed, and the error that stopped the build is raised. + def test_failed_configuration_closes_built_elements(self): + RecordedElement.closed = [] + config = { + "PipelineOptions": { + "Elements": [ + {"elementName": "RecordedElement", "elementPath": __name__}, + {"elementName": "BrokenElement", "elementPath": __name__} + ] + } + } + with self.assertRaises(ValueError) as context: + PipelineBuilder().build_from_configuration(config) + self.assertEqual("element failed to build", str(context.exception)) + self.assertEqual(1, len(RecordedElement.closed)) + + +if __name__ == "__main__": + unittest.main() diff --git a/pipeline/fiftyone_pipeline_core/tests/test_javascriptbuilder_variables.py b/pipeline/fiftyone_pipeline_core/tests/test_javascriptbuilder_variables.py index d2c13a41..4c9302f4 100644 --- a/pipeline/fiftyone_pipeline_core/tests/test_javascriptbuilder_variables.py +++ b/pipeline/fiftyone_pipeline_core/tests/test_javascriptbuilder_variables.py @@ -27,6 +27,40 @@ from fiftyone_pipeline_core import javascriptbuilder from fiftyone_pipeline_core.pipelinebuilder import PipelineBuilder +from .classes.memorylogger import MemoryLogger + + +class _CapturedAll: + """Records the variables of every render and renders nothing.""" + + def __init__(self): + self.variables = [] + + def render(self, template, variables): + self.variables.append(dict(variables)) + return "" + + +def _render_with_hosts(hosts): + """Renders once for each host given, all through the same element, with + the host taken from the request's evidence. Returns the variables of + each render and the errors the pipeline logged.""" + + logger = MemoryLogger("error") + pipeline = PipelineBuilder({"javascript_builder_settings": { + "minify": False, "endpoint": "/json"}}).add_logger(logger).build() + captured = _CapturedAll() + with mock.patch.object( + javascriptbuilder.chevron, "render", captured.render): + for host in hosts: + flowdata = pipeline.create_flowdata() + flowdata.evidence.add("header.host", host) + flowdata.process() + errors = [ + entry["message"] for entry in logger.memory_log + if entry["level"] == "error"] + return captured.variables, errors + class _Captured: """Records the variables of the last render and renders nothing.""" @@ -147,6 +181,56 @@ def test_url_keeps_a_query_string_given_in_the_endpoint(self): self.assertEqual( "https://example.com/json?tenant=1", variables["_url"]) + def test_refused_url_is_left_out_and_logged_once(self): + """A host the check refuses gives a script with no callback, and the + refusal is logged as an error the first time only, whatever the + requests in between.""" + + rendered, errors = _render_with_hosts( + ["exa mple.com", "exa mple.com", "example.com", "exa mple.com"]) + + for i, variables in enumerate(rendered): + if i == 2: + self.assertEqual( + "https://example.com/json", variables["_url"]) + self.assertTrue(variables["_updateEnabled"]) + else: + self.assertNotIn("_url", variables) + self.assertFalse(variables["_updateEnabled"]) + self.assertEqual(1, len(errors), errors) + self.assertIn("'https://exa mple.com/json'", errors[0]) + + def test_refused_url_is_logged_on_one_line(self): + """The URL is quoted and escaped in the log, so a line break in it + cannot start a new log line.""" + + _, errors = _render_with_hosts(["a\nb.com\r\u2028"]) + + self.assertEqual(1, len(errors), errors) + for c in "\n\r\u2028": + self.assertNotIn(c, errors[0]) + self.assertIn("a\\nb.com\\r\\u2028", errors[0]) + + def test_blank_host_gives_no_callback_and_is_not_logged(self): + """A host made only of whitespace is not set, so the script has no + callback and nothing is logged.""" + + rendered, errors = _render_with_hosts([" "]) + + self.assertNotIn("_url", rendered[0]) + self.assertFalse(rendered[0]["_updateEnabled"]) + self.assertEqual([], errors) + + def test_host_from_the_request_is_written_as_checked(self): + """A host from the request reaches the script as the check writes + it, in lower case and with anything that could end the string the + script holds it in percent-encoded.""" + + rendered, _ = _render_with_hosts(["EXAMPLE.com/');alert(1);//"]) + + self.assertEqual( + "https://example.com/%27);alert(1);//json", rendered[0]["_url"]) + if __name__ == "__main__": unittest.main() diff --git a/pipeline/fiftyone_pipeline_did/readme.md b/pipeline/fiftyone_pipeline_did/readme.md index 837353b0..fe51f236 100644 --- a/pipeline/fiftyone_pipeline_did/readme.md +++ b/pipeline/fiftyone_pipeline_did/readme.md @@ -713,3 +713,10 @@ is refreshed by common-ci's `update-example-assets` step. not a format limit. - **No creation of new 51Dids.** This is a parser; new 51Dids are issued by the 51Degrees cloud / on-premise hashing engines. + +## Find out more + +- [51Did](https://51degrees.com/developers/51did?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_did&utm_term=find-out-more-51did) +- [51Did documentation](https://51degrees.com/documentation/_identifiers_51_did.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_did&utm_term=find-out-more-documentation) +- [51Did inspector](https://51degrees.com/developers/51did-inspector?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_did&utm_term=find-out-more-51did-inspector) +- https://github.com/51Degrees/51degrees-python diff --git a/pipeline/fiftyone_pipeline_did/setup.py b/pipeline/fiftyone_pipeline_did/setup.py index 0c0fb67b..7d5e581b 100644 --- a/pipeline/fiftyone_pipeline_did/setup.py +++ b/pipeline/fiftyone_pipeline_did/setup.py @@ -41,7 +41,11 @@ def read(file_name): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_did-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/developers/51did?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_did&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_identifiers_51_did.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_did&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("Strongly typed reader and cloud client for the 51Did (51Degrees Identifier) value returned by the 51Degrees Cloud service. Parses the OWID envelope in either base64 alphabet and exposes the Flags, License Id and match key plus the identifier type, and verifies a 51Did's signature offline or through the cloud and redeems a sealed creator context result on the server. Compare match keys, never envelopes."), long_description=read("readme.md"), long_description_content_type='text/markdown', diff --git a/pipeline/fiftyone_pipeline_engines/readme.md b/pipeline/fiftyone_pipeline_engines/readme.md index 95eebd5d..b9993940 100644 --- a/pipeline/fiftyone_pipeline_engines/readme.md +++ b/pipeline/fiftyone_pipeline_engines/readme.md @@ -1,8 +1,8 @@ # Pipeline Engines -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines-readme.md&utm_term=pipeline-engines "Data rewards the curious") **Python Pipeline Engines** +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines&utm_term=pipeline-engines "Data rewards the curious") **Python Pipeline Engines** -[Developer Documentation](https://51degrees.com/pipeline-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines-readme.md&utm_term=pipeline-engines "Developer Documentation") +[Developer Documentation](https://51degrees.com/pipeline-python/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines&utm_term=pipeline-engines "Developer Documentation") ## Introduction @@ -30,3 +30,9 @@ Engines created by 51Degrees: * Python 3.8+ * The gitversion lib using `python -m pip install gitversion` +## Find out more + +- [Build your own](https://51degrees.com/build-your-own?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines&utm_term=find-out-more-build-your-own) +- [Pipeline API documentation](https://51degrees.com/documentation/_pipeline_api__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines&utm_term=find-out-more-documentation) +- [Pipeline API examples](https://51degrees.com/documentation/_pipeline_api__examples__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines&utm_term=find-out-more-examples) +- https://github.com/51Degrees/51degrees-python diff --git a/pipeline/fiftyone_pipeline_engines/setup.py b/pipeline/fiftyone_pipeline_engines/setup.py index b3b09ee2..d98d758e 100644 --- a/pipeline/fiftyone_pipeline_engines/setup.py +++ b/pipeline/fiftyone_pipeline_engines/setup.py @@ -41,7 +41,11 @@ def read(file_name): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/build-your-own?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_pipeline_api__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("This package extends the flow element class created by the fiftyone-pipeline-core package into a specialized type of flow element called an engine."), long_description=read("readme.md"), long_description_content_type='text/markdown', diff --git a/pipeline/fiftyone_pipeline_engines/src/fiftyone_pipeline_engines/datafile_update_service.py b/pipeline/fiftyone_pipeline_engines/src/fiftyone_pipeline_engines/datafile_update_service.py index 59168577..c4c6b047 100644 --- a/pipeline/fiftyone_pipeline_engines/src/fiftyone_pipeline_engines/datafile_update_service.py +++ b/pipeline/fiftyone_pipeline_engines/src/fiftyone_pipeline_engines/datafile_update_service.py @@ -90,6 +90,10 @@ def update_data_file(self, data_file): @param data_file: Datafile to update """ + # A data file whose engine has been closed is not updated. + if not self.__is_registered(data_file): + return False + # exit if already updating if data_file.updating: @@ -230,6 +234,9 @@ def check_next_update(self, data_file, run_after_error = False): @param data_file: Datafile to check updates for """ + if not self.__is_registered(data_file): + return + if data_file.update_on_start and not data_file.updated_on_start and not run_after_error: self.update_data_file(data_file) @@ -241,6 +248,7 @@ def check_next_update(self, data_file, run_after_error = False): t = threading.Timer(interval, self.async_update_data_file, args=[data_file]) t.daemon = True t.name = "AutoUpdate - Scheduled" + data_file.update_timer = t t.start() def register_pipeline(self, pipeline): @@ -265,6 +273,26 @@ def register_data_file(self, data_file): data_file.last_changed_time = os.stat(data_file.path).st_mtime self.track_local_data_file_changes(data_file) + def unregister_data_file(self, data_file): + """! + Stop the checks and updates of a datafile, for example when its + engine is closed. Checks already scheduled are cancelled, and a + check already running schedules no other, so nothing is left + holding the data file or its engine. + @type data_file: DataFile + @param data_file: Datafile to stop checking + """ + + data_file.registered = False + for name in ("update_timer", "watcher_timer"): + timer = getattr(data_file, name, None) + if timer is not None: + timer.cancel() + + def __is_registered(self, data_file): + # A data file checked without being registered first is checked as + # before. Only one that has been unregistered is left alone. + return getattr(data_file, "registered", True) def track_local_data_file_changes(self, data_file): """! @@ -272,10 +300,12 @@ def track_local_data_file_changes(self, data_file): the file system watcher. A refresh that fails is logged, and reported with AUTO_UPDATE_REFRESH_FAILED as on the update route. The next check is scheduled whatever happens, so that a later file - is still picked up. + is still picked up, until the data file is unregistered. @type data_file: DataFile @param data_file: Datafile to check updates for """ + if not self.__is_registered(data_file): + return try: changed_time = os.stat(data_file.path).st_mtime except OSError: @@ -296,13 +326,15 @@ def track_local_data_file_changes(self, data_file): "' failed. " + str(e)) self.__emit(UpdateStatus.AUTO_UPDATE_REFRESH_FAILED, data_file) finally: - t = threading.Timer( - self.FILE_SYSTEM_WATCHER_INTERVAL_SECONDS, - self.track_local_data_file_changes, - args=[data_file]) - t.daemon = True - t.name = "AutoUpdate - FileWatcher" - t.start() + if self.__is_registered(data_file): + t = threading.Timer( + self.FILE_SYSTEM_WATCHER_INTERVAL_SECONDS, + self.track_local_data_file_changes, + args=[data_file]) + t.daemon = True + t.name = "AutoUpdate - FileWatcher" + data_file.watcher_timer = t + t.start() def on_complete(self, on_complete_method): self.on_complete_methods.append(on_complete_method) diff --git a/pipeline/fiftyone_pipeline_engines/src/fiftyone_pipeline_engines/engine.py b/pipeline/fiftyone_pipeline_engines/src/fiftyone_pipeline_engines/engine.py index f22f654c..5ef3556a 100644 --- a/pipeline/fiftyone_pipeline_engines/src/fiftyone_pipeline_engines/engine.py +++ b/pipeline/fiftyone_pipeline_engines/src/fiftyone_pipeline_engines/engine.py @@ -150,6 +150,22 @@ def on_registration(self, pipeline): pipeline.data_file_update_service = DataFileUpdateService(pipeline) pipeline.data_file_update_service.register_data_file(self.data_file) + def close(self): + + """! + + Stop the checks and updates of the engine's data file, whose timers + would otherwise keep the engine alive. Engines that hold native data + release it as well. + + """ + + if self.data_file: + for pipeline in self.pipelines: + service = getattr(pipeline, "data_file_update_service", None) + if service is not None: + service.unregister_data_file(self.data_file) + def register_data_file(self, data_file): """! diff --git a/pipeline/fiftyone_pipeline_engines/tests/test_datafile_update_service.py b/pipeline/fiftyone_pipeline_engines/tests/test_datafile_update_service.py index af290377..5fd9c872 100644 --- a/pipeline/fiftyone_pipeline_engines/tests/test_datafile_update_service.py +++ b/pipeline/fiftyone_pipeline_engines/tests/test_datafile_update_service.py @@ -73,10 +73,16 @@ def __init__(self, pending, function, args): self.pending = pending self.function = function self.args = args + self.cancelled = False def start(self): self.pending.append(self) + def cancel(self): + self.cancelled = True + if self in self.pending: + self.pending.remove(self) + class FileSystemWatcherTests(unittest.TestCase): @@ -149,6 +155,39 @@ def testFileSystemWatcherKeepsWatchingWhileFileIsMissing(self): self.assertNotIn( UpdateStatus.AUTO_UPDATE_REFRESH_FAILED, self.statuses) + # Unregistering, as closing the engine does, cancels the next check, and + # a check already running neither refreshes nor schedules another, so + # nothing is left holding the data file. + def testUnregisterStopsFileSystemWatcher(self): + data_file = StandInDataFile(self.path) + self.service.register_data_file(data_file) + check = self.pending[0] + + self.service.unregister_data_file(data_file) + self.assertTrue(check.cancelled) + self.assertEqual(0, len(self.pending)) + + self.write("version 2") + check.function(*check.args) + self.assertEqual("version 1", data_file.data) + self.assertEqual(0, len(self.pending)) + + # Unregistering cancels the next automatic update, and an update that + # starts afterwards does nothing. + def testUnregisterStopsAutomaticUpdates(self): + data_file = StandInDataFile(self.path) + data_file.file_system_watcher = False + data_file.auto_update = True + data_file.polling_interval = 30 + data_file.update_time_maximum_randomisation = 0 + self.service.register_data_file(data_file) + check = self.pending[0] + + self.service.unregister_data_file(data_file) + self.assertTrue(check.cancelled) + self.assertEqual(0, len(self.pending)) + self.assertFalse(self.service.update_data_file(data_file)) + if __name__ == "__main__": unittest.main() diff --git a/pipeline/fiftyone_pipeline_engines_fiftyone/readme.md b/pipeline/fiftyone_pipeline_engines_fiftyone/readme.md index 49d04854..8dba8ae2 100644 --- a/pipeline/fiftyone_pipeline_engines_fiftyone/readme.md +++ b/pipeline/fiftyone_pipeline_engines_fiftyone/readme.md @@ -1,8 +1,8 @@ # Pipeline Engines Fiftyone -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines_fiftyone-readme.md&utm_term=pipeline-engines-fiftyone "Data rewards the curious") **Python Pipeline Engines** +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines_fiftyone&utm_term=pipeline-engines-fiftyone "Data rewards the curious") **Python Pipeline Engines** -[Developer Documentation](https://51degrees.com/pipeline-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines_fiftyone-readme.md&utm_term=pipeline-engines-fiftyone "Developer Documentation") +[Developer Documentation](https://51degrees.com/pipeline-python/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines_fiftyone&utm_term=pipeline-engines-fiftyone "Developer Documentation") ## Introduction @@ -15,3 +15,10 @@ It includes a ShareUsage engine that sends usage data to 51Degrees in zipped bat ## Requirements * Python 3.8+ + +## Find out more + +- [Build your own](https://51degrees.com/build-your-own?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines_fiftyone&utm_term=find-out-more-build-your-own) +- [Pipeline API documentation](https://51degrees.com/documentation/_pipeline_api__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines_fiftyone&utm_term=find-out-more-documentation) +- [Pipeline API examples](https://51degrees.com/documentation/_pipeline_api__examples__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines_fiftyone&utm_term=find-out-more-examples) +- https://github.com/51Degrees/51degrees-python diff --git a/pipeline/fiftyone_pipeline_engines_fiftyone/setup.py b/pipeline/fiftyone_pipeline_engines_fiftyone/setup.py index a4c89ef1..b9445460 100644 --- a/pipeline/fiftyone_pipeline_engines_fiftyone/setup.py +++ b/pipeline/fiftyone_pipeline_engines_fiftyone/setup.py @@ -40,7 +40,11 @@ def read(file_name): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines_fiftyone-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/build-your-own?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines_fiftyone&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_pipeline_api__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_engines_fiftyone&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("The 51Degrees Pipeline API is a generic web request intelligence and data processing solution with the ability to add a range of 51Degrees and/or custom plug ins (Engines). It includes a ShareUsage engine that sends usage data to 51Degrees in zipped batches."), long_description=read("readme.md"), long_description_content_type='text/markdown', diff --git a/pipeline/fiftyone_pipeline_engines_fiftyone/src/fiftyone_pipeline_engines_fiftyone/share_usage.py b/pipeline/fiftyone_pipeline_engines_fiftyone/src/fiftyone_pipeline_engines_fiftyone/share_usage.py index 55ef341f..3173318e 100644 --- a/pipeline/fiftyone_pipeline_engines_fiftyone/src/fiftyone_pipeline_engines_fiftyone/share_usage.py +++ b/pipeline/fiftyone_pipeline_engines_fiftyone/src/fiftyone_pipeline_engines_fiftyone/share_usage.py @@ -122,6 +122,17 @@ def __init__( def __del__(self): self.thread_pool.shutdown(wait=True) + def close(self): + + """! + Wait for any usage data already being sent, then stop the threads + that send it. The pipeline calls this when it is closed. + + """ + + super(ShareUsage, self).close() + self.thread_pool.shutdown(wait=True) + def get_constant_xml(self): if self.constant_xml is None: coreVersion = version("fiftyone_pipeline_core") diff --git a/pipeline/fiftyone_pipeline_translation/readme.md b/pipeline/fiftyone_pipeline_translation/readme.md index e38c02dd..4c0450eb 100644 --- a/pipeline/fiftyone_pipeline_translation/readme.md +++ b/pipeline/fiftyone_pipeline_translation/readme.md @@ -1,8 +1,8 @@ # Pipeline Translation -![51Degrees](https://51degrees.com/img/logo.png?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_translation-readme.md&utm_term=pipeline-translation "Data rewards the curious") **Python Pipeline Translation** +![51Degrees](https://51degrees.com/img/logo.png?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_translation&utm_term=pipeline-translation "Data rewards the curious") **Python Pipeline Translation** -[Developer Documentation](https://51degrees.com/pipeline-python/index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_translation-readme.md&utm_term=pipeline-translation "Developer Documentation") +[Developer Documentation](https://51degrees.com/pipeline-python/index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_translation&utm_term=pipeline-translation "Developer Documentation") ## Introduction @@ -24,3 +24,10 @@ header). This package is built on top of: * [fiftyone_pipeline_core](https://pypi.org/project/fiftyone-pipeline-core/) + +## Find out more + +- [Build your own](https://51degrees.com/build-your-own?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_translation&utm_term=find-out-more-build-your-own) +- [Pipeline API documentation](https://51degrees.com/documentation/_pipeline_api__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_translation&utm_term=find-out-more-documentation) +- [Pipeline API examples](https://51degrees.com/documentation/_pipeline_api__examples__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_translation&utm_term=find-out-more-examples) +- https://github.com/51Degrees/51degrees-python diff --git a/pipeline/fiftyone_pipeline_translation/setup.py b/pipeline/fiftyone_pipeline_translation/setup.py index fa2a9f4d..6460ddb6 100644 --- a/pipeline/fiftyone_pipeline_translation/setup.py +++ b/pipeline/fiftyone_pipeline_translation/setup.py @@ -43,7 +43,11 @@ def read(file_name): version=read("version.txt"), author="51Degrees Engineering", author_email="engineering@51degrees.com", - url="https://51degrees.com/?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_translation-setup.py&utm_term=url", + project_urls={ + "Homepage": "https://51degrees.com/build-your-own?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_translation&utm_term=homepage", + "Documentation": "https://51degrees.com/documentation/_pipeline_api__index.html?utm_source=pypi&utm_medium=package&utm_campaign=51degrees-python&utm_content=fiftyone_pipeline_translation&utm_term=documentation", + "Source": "https://github.com/51Degrees/51degrees-python", + }, description=("This package adds a generic translation flow element to the fiftyone-pipeline, which translates values from a source element into another language using YAML translation files."), long_description=read("readme.md"), long_description_content_type='text/markdown', diff --git a/pipeline/readme.md b/pipeline/readme.md index a5c12c34..ea7419ec 100755 --- a/pipeline/readme.md +++ b/pipeline/readme.md @@ -24,7 +24,7 @@ The examples are described under Examples below. The `owid-python` folder is a g ## Dependencies For runtime dependencies, see our [dependencies](https://51degrees.com/documentation/_info__dependencies.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=dependencies) page. -The [tested versions](https://51degrees.com/documentation/_info__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=dependencies) page shows the Python versions that we currently test against. The software may run fine against other versions, but additional caution should be applied. +The [tested versions](https://51degrees.com/documentation/_product_summaries__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=dependencies) page shows the Python versions that we currently test against. The software may run fine against other versions, but additional caution should be applied. ## Installation @@ -56,7 +56,7 @@ The tests that call the 51Degrees cloud service read a resource key from the `re ## Examples There are several examples available that demonstrate how to make use of the Pipeline API in isolation. These are described in the table below. -If you want examples that demonstrate how to use 51Degrees products such as device detection, then these are available in the corresponding [folder](https://github.com/51Degrees/51degrees-python/tree/main/device-detection) and on our [website](https://51degrees.com/documentation/_examples__device_detection__index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=examples). +If you want examples that demonstrate how to use 51Degrees products such as device detection, then these are available in the corresponding [folder](https://github.com/51Degrees/51degrees-python/tree/main/device-detection) and on our [website](https://51degrees.com/documentation/_device_detection__examples__index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=examples). | Example | Description | |-----------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------|