diff --git a/AGENTS.md b/AGENTS.md index d163ccbfe..971eb5d05 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,7 +1,226 @@ # Working in this repository -Notes on what is not obvious from the code. Keep them short and current. -Building, testing and the layout are described in the [README](README.md). +If you have been asked to get an example running, follow +[Run the examples](#run-the-examples). It gives every step in order, from a +fresh clone to the output. The sections after it are notes on what is not +obvious from the code, for changing it. Keep them short and current. The +[README](README.md) describes the build, the tests and the layout. + +## Run the examples + +Run each command from the repository root unless the step says otherwise. +Device Detection's on-premise `GettingStartedOnPrem` is the quickest example +to show, because its data file is about 35 MB. IP Intelligence's on-premise +`GettingStartedOnPrem` needs a download of about 230 MB that unpacks to about +460 MB, so run it second. + +The examples use the packages published to Maven Central unless the `dev` +profile is active (see +[One reactor, six package trees](#one-reactor-six-package-trees)). Those +packages carry the native engines, so building only an example folder, as +below, needs no CMake and no C++ toolchain. `mvn install` at the root builds +the native engines from `cxx` and does need them, as the README says. + +### Prerequisites + +- Git. +- A JDK, 8 or later. CI tests 8, 11, 17 and 21 (`ci/options.json`). +- Maven 3.8 or later, or the `mvnw` wrapper at the root. +- Network access to GitHub, to Maven Central for the packages the examples + use, and to `distributor.51degrees.com` for the data files. +- For the data files, `curl`, `gzip` and `md5sum` (`md5` on macOS) on Linux + and macOS, where the examples download with the shell script, and Windows + PowerShell 5.1 or PowerShell 7 on Windows, where they use the PowerShell + script. + +Clone the repository with its submodules. + +``` +git clone --recurse-submodules https://github.com/51Degrees/51degrees-java.git +cd 51degrees-java +``` + +In a clone made without `--recurse-submodules`, fetch the submodules. + +``` +git submodule update --init --recursive +``` + +### Data files + +The examples use the free Lite data files, which need no licence key. They are +not stored in this repository. The `cxx` submodule holds the +[data](https://github.com/51Degrees/data) repository as `cxx/data`, which has a +link to each file, and a script downloads them into `cxx/data/device-detection` +and `cxx/data/ip-intelligence`. + +On Linux and macOS, run the shell script. + +``` +cxx/data/get-data-files.sh +``` + +On Windows, run the PowerShell script. Windows PowerShell can refuse to run a +script file under the policy Windows sets by default, so run it the way the +examples do. + +``` +powershell -NoProfile -ExecutionPolicy Bypass -File cxx/data/get-data-files.ps1 +``` + +Where PowerShell 7 is installed, `pwsh cxx/data/get-data-files.ps1` does the +same on any system. Either script downloads every free file, and keeps a file +that is already there. To download one file, give its name, as in +`cxx/data/get-data-files.sh 51Degrees-LiteV4.1.hash`. An example whose free +file is missing runs the script for that file itself. + +#### Check the files + +- `cxx/data/device-detection/51Degrees-LiteV4.1.hash` is more than 10 MB. +- `cxx/data/ip-intelligence/51Degrees-LiteV41.ipi` is more than 100 MB. + +### Build + +Build the console examples of a product, with the examples' shared module, in +that product's examples folder. Each build makes a jar that holds the example +and everything it needs. + +``` +cd examples/device-detection +mvn -pl console -am package -DskipTests +cd ../.. +cd examples/ip-intelligence +mvn -pl console -am package -DskipTests +cd ../.. +``` + +With the wrapper, run `../../mvnw` in place of `mvn`, or `..\..\mvnw.cmd` on +Windows. The jars are +`examples/device-detection/console/target/device-detection-java-examples.console--jar-with-dependencies.jar` +and +`examples/ip-intelligence/console/target/ip-intelligence-java-examples.console--jar-with-dependencies.jar`, +where `` is the `` in that examples folder's `pom.xml`, +`4.5.127-SNAPSHOT` on `rc` today. + +### Run + +Run the examples from the repository root. They find `logback.xml`, the data +file and their options file with `FileFinder`, which searches only below the +folder Java runs in (`pipeline/pipeline.core/src/main/java/fiftyone/pipeline/util/FileFinder.java`). +When a free data file is not there, the example finds its `.link` the same way +and runs the download script for it (`findDataFile` in each examples folder's +`shared` module, `DataFileHelper.java`), with Windows PowerShell on Windows and +`sh` on Linux and macOS. The same commands work on Windows, Linux and macOS. + +``` +java -cp examples/device-detection/console/target/device-detection-java-examples.console-4.5.127-SNAPSHOT-jar-with-dependencies.jar fiftyone.devicedetection.examples.console.GettingStartedOnPrem +``` + +The IP Intelligence example looks for the paid Enterprise file, +`ip-intelligence/51Degrees-EnterpriseIpiV41.ipi`, so give it the Lite file as +its first argument. + +``` +java -cp examples/ip-intelligence/console/target/ip-intelligence-java-examples.console-4.5.127-SNAPSHOT-jar-with-dependencies.jar fiftyone.ipintelligence.examples.console.GettingStartedOnPrem cxx/data/ip-intelligence/51Degrees-LiteV41.ipi +``` + +The example downloads the Lite file first when it is not there. In place of +the argument, the `51DEGREES_IPI_PATH` environment variable or system property +can give the absolute path of a data file that is already there. + +On Java 24 or later, add `--enable-native-access=ALL-UNNAMED` before the class +name, as +[examples/device-detection/README.md](examples/device-detection/README.md#native-library-access) +explains, or Java warns about the native library. + +### Correct output + +When an example downloads its data file, a Logback line first reports +`'' is missing, so it is being downloaded by running` and the command, +and the script's `Downloading` and `Saved` lines follow. + +The Device Detection example processes five sets of evidence. For each it +prints `Input values:` and the evidence, then `Results:` with +`Mobile Device`, `Platform Name`, `Platform Version`, `Browser Name` and +`Browser Version`. Logback lines come before and after. The last ones report +the data file and end the run, with these messages. + +``` +Used a 'Lite' data file, created , days ago, from location '' +This example is using the 'Lite ' data file. This is used for illustration, ... +All done +``` + +A warning that the data file is more than 28 days old may come too, which is +expected with the Lite file. The exit code is 0. + +The IP Intelligence example looks up `116.154.188.222`, `45.236.48.61` and an +IPv6 address. For each it prints `Input values:` with `query.client-ip`, then +`Results:` with a line for each property, from `RegisteredName` to +`TimeZoneOffset`. The Lite file has RegisteredCountry, RegisteredName and +RegisteredOwner, so a line for a property it does not have reads +`Unknown. The property '' is not available in this data file.` followed +by a link, and the first such line also logs a warning that some properties +need a paid subscription. It then logs the data file in the same way, and +`All done`. + +The paid data files have many more properties, as the +[pricing page](https://51degrees.com/pricing?utm_source=github&utm_medium=docs&utm_campaign=51degrees-java&utm_content=agents.md&utm_term=correct-output) +describes. + +### Cloud examples + +The cloud examples need a resource key, which a person creates in the +configurator, so ask for one if none is set. A key with the properties most of +the Device Detection examples use can be made +[here](https://configure.51degrees.com/jqz435Nc?utm_source=github&utm_medium=docs&utm_campaign=51degrees-java&utm_content=agents.md&utm_term=cloud-examples). +The Device Detection cloud example takes the key as its first argument, or +from a `TestResourceKey` environment variable or system property. + +``` +java -cp examples/device-detection/console/target/device-detection-java-examples.console-4.5.127-SNAPSHOT-jar-with-dependencies.jar fiftyone.devicedetection.examples.console.GettingStartedCloud +``` + +It prints the same `Input values:` and `Results:` blocks as the on-premise +example. The IP Intelligence cloud examples read their key from +`51DEGREES_RESOURCE_KEY` first and `TestResourceKey` second, and +`examples/ip-intelligence/README.md` gives the keys to create for them. + +### Pitfalls + +- **A data file that is not there.** `cxx/data` holds a `.link` for each file + rather than the file. An example whose file is missing runs the download + script for it, so it needs the network and the tools under "Prerequisites". + Check the file sizes as above. +- **Missing submodules.** The example then logs `Could not find the data file + 'device-detection/51Degrees-LiteV4.1.hash' which must be somewhere in the + project space to be found.` Run `git submodule update --init --recursive`. A + zip download of the repository has no submodules, so clone with Git. +- **Running from another folder.** `FileFinder` looks only below the folder + Java runs in, so `java` started in `examples/device-detection`, as the + examples' README shows, cannot find the data file or its link in `cxx`. + Start it from the repository root. The same applies to `mvn exec:java`, + which runs in the folder Maven starts in. +- **The version in the jar name.** The examples' READMEs show jar names with + `4.4.20` and `4.4.19`, which are out of date. Use the name in `target`. +- **The IP Intelligence example without an argument.** It searches for the + Enterprise file, logs `Failed to find IP Intelligence data file` and stops, + even when the Lite file is in place. +- **The `dev` profile.** `-Pdev`, which CI uses, builds the examples against + the libraries here, which needs `mvn install` at the root first and so the + native build, with CMake 3.10 or later and a C++ toolchain. It is not needed + to run the examples. +- **Platforms the packages cover.** The published + `device-detection.hash.engine.on-premise` 4.5.128 jar has native libraries + for Windows x64 and x86, Linux x64 and aarch64, and macOS x64 and aarch64. + The `ip-intelligence.engine.on-premise` 4.5.128 jar has the same, without + Windows x86. On a platform a jar lacks, the example cannot load the engine. +- **What CI does not test.** On a pull request CI builds and tests the + examples with the `dev` profile (`ci/build-project.ps1` and + `ci/run-unit-tests.ps1`), and against the published packages only when it + publishes (`ci/run-integration-tests.ps1`). It never runs the console + example jars as above, and the example tests use the paid data files when + CI has them. ## One reactor, six package trees @@ -22,9 +241,18 @@ Tests and examples find data files with `FileFinder`, which searches below `project.root`. The surefire configuration sets that to `maven.multiModuleProjectDirectory`, which Maven takes from the nearest `.mvn` folder above the working directory. The only `.mvn` folder is at the root, so -the search covers `cxx/device-detection-data` and `cxx/ip-intelligence-data` +the search covers `cxx/data/device-detection` and `cxx/data/ip-intelligence` from wherever Maven is run. Do not add a `.mvn` folder below the root. +Those folders are the data repository, a submodule of cxx, where each free +data file is a `.link` until `get-data-files.ps1` or `get-data-files.sh` +downloads it. `ci/fetch-assets.ps1` runs the script, and the examples' +`DataFileHelper.findDataFile` runs it for a file that is missing. Look for a +free file by its folder and name, such as +`device-detection/51Degrees-LiteV4.1.hash`, so the search does not stop at a +file of the same name elsewhere, such as a Git LFS pointer in an old +`cxx/device-detection-data` folder. + ## Native builds Each on-premise engine compiles its own copy of the common C code from `cxx`, @@ -44,6 +272,14 @@ The Windows pre-build scripts take a toolset and SDK version from the pom. JavaScript templates and the OWID library are submodules inside `pipeline`, where the Pipeline API expects them. +## Releases + +Pull requests go to `rc`, not `main`, and stay drafts until they have been +reviewed, because the nightly merges any pull request into `rc` that is not a +draft and has no review requested. When a release is wanted, one pull request +merges `rc` into `main`, and the "Only rc into main" check fails a pull request +into `main` from any other branch. + ## Writing Plain words, no em dashes, sentence case headings. Nothing in this public diff --git a/README.md b/README.md index 4ba31cf22..479f1ff36 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ Created from `pipeline-java`, `device-detection-java`, `ip-intelligence-java`, ` | `location` | The Location packages, `com.51degrees:geo-location*` | | `examples/device-detection` | Console and web examples for Device Detection | | `examples/ip-intelligence` | Console and web examples for IP Intelligence, including the mixed examples that run both products | -| `cxx` | Submodule of [51degrees-cxx](https://github.com/51Degrees/51degrees-cxx), with the C and C++ code the on-premise engines are built from and the data file submodules | +| `cxx` | Submodule of [51degrees-cxx](https://github.com/51Degrees/51degrees-cxx), with the C and C++ code the on-premise engines are built from and the [data](https://github.com/51Degrees/data) submodule in `cxx/data` | | `pom.xml` | The reactor that builds everything here. It is never published | | `ci` | The scripts the shared 51Degrees CI workflows call | @@ -55,10 +55,20 @@ git clone --recurse-submodules https://github.com/51Degrees/51degrees-java.git cd 51degrees-java ``` -The free Device Detection Lite data file in `cxx/device-detection-data` is -stored with [Git LFS](https://git-lfs.github.com/). Run `git lfs pull` in that -folder to fetch it. The IP Intelligence ASN file in `cxx/ip-intelligence-data` -is plain git. +The free data files come from the [data](https://github.com/51Degrees/data) +repository, which is the `cxx/data` submodule. Each one is listed there as a +`.link` file, and a script downloads them all into place: + +``` +cxx/data/get-data-files.sh +``` + +On Windows, run `cxx\data\get-data-files.ps1` in PowerShell instead. The +examples run the script themselves for a free file they need that is missing. + +In a clone made before the data repository was added, delete the +`cxx/device-detection-data` and `cxx/ip-intelligence-data` folders once cxx no +longer includes them, because Git leaves a removed submodule's folder on disk. ## Building @@ -89,8 +99,9 @@ mvn surefire:test ``` The cloud tests need a Resource Key, passed as `-DTestResourceKey=`. The -on-premise IP Intelligence tests need the Lite or Enterprise data file, which -is not in the repository. See `ip-intelligence/README.md` for how to get it. +on-premise tests need the free data files, which the script in +[Getting the code](#getting-the-code) downloads. For the Enterprise IP +Intelligence data file, see `ip-intelligence/README.md`. Tests find data files by searching from the root of this repository, so run Maven from here rather than from inside a package folder. diff --git a/ci/README.md b/ci/README.md index 1041809a8..bf6b2a62b 100644 --- a/ci/README.md +++ b/ci/README.md @@ -21,8 +21,12 @@ repositories used to do for one package tree each. workflow instead tests the examples against the version it built, through [run-integration-tests.ps1](run-integration-tests.ps1). - [fetch-assets.ps1](fetch-assets.ps1) links every data file it downloads into - `cxx/device-detection-data` or `cxx/ip-intelligence-data`, where the tests - and examples find them. + `cxx/data/device-detection` or `cxx/data/ip-intelligence`, where the tests + and examples find them. Those are the folders of the + [data](https://github.com/51Degrees/data) repository, which is a submodule + of cxx. The script then runs the data repository's `get-data-files.ps1`, + which keeps the linked files and downloads the free files that are not + assets, which is the Asn file. - [run-performance-tests.ps1](run-performance-tests.ps1) measures the Pipeline API, Device Detection and IP Intelligence and writes one results file per configuration. The Device Detection and IP Intelligence metrics are prefixed @@ -51,7 +55,7 @@ To run the on-premise tests: To run the cloud tests: -- `_51DEGREES_RESOURCE_KEY_BESPOKE`: [resource key](https://51degrees.com/documentation/_info__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=ci-readme.md&utm_term=secrets) +- `_51DEGREES_RESOURCE_KEY_BESPOKE`: [resource key](https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=ci-readme.md&utm_term=secrets) passed to the tests as `TestResourceKey` and `SuperResourceKey`. - `_51DEGREES_LICENSE_KEY_51DID`: license key for the 51Did cloud tests, together with the `_51DEGREES_RESOURCE_KEY_BASIC` and diff --git a/ci/fetch-assets.ps1 b/ci/fetch-assets.ps1 index 23f509ea1..4062061b1 100644 --- a/ci/fetch-assets.ps1 +++ b/ci/fetch-assets.ps1 @@ -7,9 +7,11 @@ param ( $ErrorActionPreference = "Stop" # Tests and examples find data files by searching below the repository root, -# so every asset is linked into the data folders of the cxx submodule. -$deviceDetectionData = "$PSScriptRoot/../cxx/device-detection-data" -$ipIntelligenceData = "$PSScriptRoot/../cxx/ip-intelligence-data" +# so every asset is linked into the folders of the data repository, which is +# the data submodule of cxx. +$data = "$PSScriptRoot/../cxx/data" +$deviceDetectionData = "$data/device-detection" +$ipIntelligenceData = "$data/ip-intelligence" $deviceDetectionAssets = 'TAC-HashV41.hash', '51Degrees-LiteV4.1.hash', '20000 Evidence Records.yml', '20000 User Agents.csv' ./steps/fetch-assets.ps1 -DeviceDetection:$DeviceDetection -DeviceDetectionUrl:$DeviceDetectionUrl -Assets $deviceDetectionAssets @@ -24,6 +26,14 @@ New-Item -ItemType SymbolicLink -Force -Target "$PWD/assets/51Degrees-Enterprise # link it as that. New-Item -ItemType SymbolicLink -Force -Target "$PWD/assets/51Degrees-LiteIpiV41.ipi" -Path "$ipIntelligenceData/51Degrees-LiteV41.ipi" +# The data repository's script downloads every linked file that is not +# already there. The files linked above are kept, so this gets the free files +# that are not assets, which is the Asn file. +& "$data/get-data-files.ps1" +if (-not $?) { + throw "The data repository's get-data-files.ps1 could not download every file" +} + Write-Host "Assets hashes:" Get-FileHash -Algorithm MD5 -Path assets/* diff --git a/ci/pmd-resource-ruleset.xml b/ci/pmd-resource-ruleset.xml new file mode 100644 index 000000000..5f99fad8e --- /dev/null +++ b/ci/pmd-resource-ruleset.xml @@ -0,0 +1,99 @@ + + + + + Fails the build when a FlowData, a Pipeline or a collection of + metadata is not closed on every path. The native results in a FlowData + and the native metadata collections each hold the engine's data set, + so while one is open, closing the Pipeline cannot free the data set. + Java never frees the native results by itself. + + Where something is closed by its owner elsewhere, such as the FlowData + the web integration closes at the end of a request, the code says so + with @SuppressWarnings and a comment naming the owner. + + + + + Every FlowData created is closed by a try-with-resources statement, + unless it is returned to the caller or kept in a field. + + 3 + + + + + + + + + + + + Every CloseableIterable a method returns, such as the profiles or + the components of an engine, is closed by a try-with-resources + statement, unless it is returned to the caller. + + 3 + + + + + + + + + + + + A local variable holding a Pipeline that a builder has just built + is a try-with-resources resource, is closed in a finally block of + the same method, or is returned to the caller. + + 3 + + + + + + + + + + diff --git a/cxx b/cxx index 1bec7b570..e866f6d3c 160000 --- a/cxx +++ b/cxx @@ -1 +1 @@ -Subproject commit 1bec7b570575e2e180b52435599bef9b03e35ad8 +Subproject commit e866f6d3cea936587790f87967eff7863778b30e diff --git a/device-detection/README.md b/device-detection/README.md index c393a561e..5441e364a 100644 --- a/device-detection/README.md +++ b/device-detection/README.md @@ -11,7 +11,7 @@ This repository contains Java implementation of the device detection [specificat ## Dependencies For runtime dependencies, see our [dependencies](https://51degrees.com/documentation/_info__dependencies.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=dependencies) page. -The [tested versions](https://51degrees.com/documentation/_info__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=dependencies) page shows +The [tested versions](https://51degrees.com/documentation/_product_summaries__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=dependencies) page shows the JDK versions that we currently test against. The software may run fine against other versions, but additional caution should be applied. @@ -21,30 +21,31 @@ The Java API can either use our cloud service to get its data or it can use a lo #### Cloud -You will require [resource keys](https://51degrees.com/documentation/_info__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud) +You will require [resource keys](https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud) to use the Cloud API, as described on our website. Get resource keys from -our [configurator](https://configure.51degrees.com/?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud), see our [documentation](https://51degrees.com/documentation/_concepts__configurator.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud) on +our [configurator](https://configure.51degrees.com/?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud), see our [documentation](https://51degrees.com/documentation/_services__configurator.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud) on how to use this. #### On-Premise -If you are using on-premise detection, a "Lite" version of the data required is packaged -in this repository. It contains only a limited set of "essential" device detection properties. +If you are using on-premise detection, a free "Lite" version of the data required is +available from the [data](https://github.com/51Degrees/data) repository, which is the +`cxx/data` folder of this repository. It contains only a limited set of "essential" +device detection properties. You may want to license our complete data file containing all properties. [Details of our licenses](https://51degrees.com/pricing?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=on-premise) are available on our website. -If you want to use the lite file, you will need to install [GitLFS](https://git-lfs.github.com/), then: +To get the Lite file, run the data repository's download script from the root of +this repository: ``` -git lfs install +cxx/data/get-data-files.sh 51Degrees-LiteV4.1.hash ``` -Then, navigate to 'device-detection.hash.engine.on-premise/src/main/cxx/device-detection-cxx/device-detection-data' and execute: - -``` -git lfs pull -``` +On Windows, run `cxx\data\get-data-files.ps1 51Degrees-LiteV4.1.hash` in PowerShell +instead. The file is saved in `cxx/data/device-detection`, where the tests and +examples find it. ## Installation @@ -224,7 +225,7 @@ repository. ## Tests -You will need [resource keys](https://51degrees.com/documentation/_info__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=tests) +You will need [resource keys](https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=tests) (see above) to complete the tests and run examples which include exercising the cloud API. To verify the code: diff --git a/device-detection/device-detection.cloud/pom.xml b/device-detection/device-detection.cloud/pom.xml index 4e6c839b7..ddfbaf069 100644 --- a/device-detection/device-detection.cloud/pom.xml +++ b/device-detection/device-detection.cloud/pom.xml @@ -12,7 +12,7 @@ device-detection.cloud 51Degrees :: Device Detection :: Cloud Cloud-based device detection for Java that uses the 51Degrees cloud service via the Pipeline API. Parses HTTP headers to identify device hardware, operating systems, browsers, and crawlers. - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=device-detection.cloud-pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java diff --git a/device-detection/device-detection.cloud/src/test/java/fiftyone/devicedetection/cloud/TestsBase.java b/device-detection/device-detection.cloud/src/test/java/fiftyone/devicedetection/cloud/TestsBase.java index ac8007cd1..44d233d6a 100644 --- a/device-detection/device-detection.cloud/src/test/java/fiftyone/devicedetection/cloud/TestsBase.java +++ b/device-detection/device-detection.cloud/src/test/java/fiftyone/devicedetection/cloud/TestsBase.java @@ -25,7 +25,7 @@ import fiftyone.devicedetection.shared.testhelpers.UserAgentGenerator; import fiftyone.pipeline.util.FileFinder; -import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_NAME; +import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_PATH; public class TestsBase { @@ -43,7 +43,7 @@ protected UserAgentGenerator getUserAgents() { protected void testInitialize() throws Exception { wrapper = new WrapperCloud(); userAgents = new UserAgentGenerator( - FileFinder.getFilePath(UA_FILE_NAME)); + FileFinder.getFilePath(UA_FILE_PATH)); } public void testCleanup() { diff --git a/device-detection/device-detection.hash.engine.on-premise/pom.xml b/device-detection/device-detection.hash.engine.on-premise/pom.xml index 87250bfe8..7d141d302 100644 --- a/device-detection/device-detection.hash.engine.on-premise/pom.xml +++ b/device-detection/device-detection.hash.engine.on-premise/pom.xml @@ -12,7 +12,7 @@ device-detection.hash.engine.on-premise 51Degrees :: Device Detection :: Hash :: On Premise High-performance on-premise hash-based device detection engine for Java, built for the 51Degrees Pipeline API. Parses HTTP headers locally to identify device hardware, operating systems, browsers, and crawlers, with no external service calls. - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=device-detection.hash.engine.on-premise-pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java diff --git a/device-detection/device-detection.hash.engine.on-premise/src/main/java/fiftyone/devicedetection/hash/engine/onpremise/data/ComponentMetaDataHash.java b/device-detection/device-detection.hash.engine.on-premise/src/main/java/fiftyone/devicedetection/hash/engine/onpremise/data/ComponentMetaDataHash.java index 30d18118b..63cfdbf6f 100644 --- a/device-detection/device-detection.hash.engine.on-premise/src/main/java/fiftyone/devicedetection/hash/engine/onpremise/data/ComponentMetaDataHash.java +++ b/device-detection/device-detection.hash.engine.on-premise/src/main/java/fiftyone/devicedetection/hash/engine/onpremise/data/ComponentMetaDataHash.java @@ -22,6 +22,7 @@ package fiftyone.devicedetection.hash.engine.onpremise.data; +import fiftyone.pipeline.engines.fiftyone.data.CloseableIterable; import fiftyone.devicedetection.hash.engine.onpremise.flowelements.DeviceDetectionHashEngine; import fiftyone.devicedetection.hash.engine.onpremise.interop.PropertyIterable; import fiftyone.devicedetection.hash.engine.onpremise.interop.swig.ComponentMetaDataSwig; @@ -77,7 +78,7 @@ public ProfileMetaData getDefaultProfile() { } @Override - public Iterable getProperties() { + public CloseableIterable getProperties() { return new PropertyIterable( engine, properties, diff --git a/device-detection/device-detection.hash.engine.on-premise/src/main/java/fiftyone/devicedetection/hash/engine/onpremise/data/ProfileMetaDataHash.java b/device-detection/device-detection.hash.engine.on-premise/src/main/java/fiftyone/devicedetection/hash/engine/onpremise/data/ProfileMetaDataHash.java index 7603773ae..afadcd5b2 100644 --- a/device-detection/device-detection.hash.engine.on-premise/src/main/java/fiftyone/devicedetection/hash/engine/onpremise/data/ProfileMetaDataHash.java +++ b/device-detection/device-detection.hash.engine.on-premise/src/main/java/fiftyone/devicedetection/hash/engine/onpremise/data/ProfileMetaDataHash.java @@ -114,11 +114,19 @@ public ComponentMetaData getComponent() { @Override public String getName() { List values = new ArrayList<>(); - for (ValueMetaData value : getValues()) { - if (value.getProperty().getDisplayOrder() > 0 && - value.getName().equalsIgnoreCase("N/A") == false) { - values.add(value); + // The native collection holds the data set until it is closed. The + // values taken from it are separate objects, so they outlive it. + try (CloseableIterable profileValues = getValues()) { + for (ValueMetaData value : profileValues) { + if (value.getProperty().getDisplayOrder() > 0 && + value.getName().equalsIgnoreCase("N/A") == false) { + values.add(value); + } } + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new IllegalStateException(e); } ValueMetaData[] valuesArray = new ValueMetaData[0]; valuesArray = values.toArray(valuesArray); diff --git a/device-detection/device-detection.hash.engine.on-premise/src/main/java/fiftyone/devicedetection/hash/engine/onpremise/data/PropertyMetaDataHash.java b/device-detection/device-detection.hash.engine.on-premise/src/main/java/fiftyone/devicedetection/hash/engine/onpremise/data/PropertyMetaDataHash.java index 3dbf2b5d5..6a6b5a92a 100644 --- a/device-detection/device-detection.hash.engine.on-premise/src/main/java/fiftyone/devicedetection/hash/engine/onpremise/data/PropertyMetaDataHash.java +++ b/device-detection/device-detection.hash.engine.on-premise/src/main/java/fiftyone/devicedetection/hash/engine/onpremise/data/PropertyMetaDataHash.java @@ -22,6 +22,7 @@ package fiftyone.devicedetection.hash.engine.onpremise.data; +import fiftyone.pipeline.engines.fiftyone.data.CloseableIterable; import fiftyone.devicedetection.hash.engine.onpremise.flowelements.DeviceDetectionHashEngine; import fiftyone.devicedetection.hash.engine.onpremise.interop.Swig; import fiftyone.devicedetection.hash.engine.onpremise.interop.ValueIterable; @@ -136,7 +137,7 @@ public ComponentMetaData getComponent() { } @Override - public Iterable getValues() { + public CloseableIterable getValues() { return new ValueIterable( engine, engine.getMetaData().getValuesForProperty(source)); diff --git a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/TestsBase.java b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/TestsBase.java index 0351da463..f3826ee0e 100644 --- a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/TestsBase.java +++ b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/TestsBase.java @@ -26,7 +26,7 @@ import fiftyone.devicedetection.shared.testhelpers.FileUtils; import fiftyone.pipeline.engines.Constants; -import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_NAME; +import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_PATH; import static fiftyone.pipeline.util.FileFinder.getFilePath; public class TestsBase { @@ -47,7 +47,7 @@ protected void testInitialize(Constants.PerformanceProfiles profile) throws Exce FileUtils.getHashFile(), profile); userAgents = new UserAgentGenerator( - getFilePath(UA_FILE_NAME)); + getFilePath(UA_FILE_PATH)); } public void testCleanup() { diff --git a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/WrapperHash.java b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/WrapperHash.java index 2d034f6ff..21bdb6fb7 100644 --- a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/WrapperHash.java +++ b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/WrapperHash.java @@ -28,6 +28,7 @@ import fiftyone.pipeline.core.flowelements.Pipeline; import fiftyone.pipeline.core.flowelements.PipelineBuilder; import fiftyone.pipeline.engines.Constants; +import fiftyone.pipeline.engines.fiftyone.data.CloseableIterable; import fiftyone.pipeline.engines.fiftyone.data.ComponentMetaData; import fiftyone.pipeline.engines.fiftyone.data.FiftyOneAspectPropertyMetaData; import fiftyone.pipeline.engines.fiftyone.data.ProfileMetaData; @@ -70,15 +71,15 @@ public Iterable getProperties() { } @Override - public Iterable getValues() { + public CloseableIterable getValues() { return engine.getValues(); } - public Iterable getProfiles() { + public CloseableIterable getProfiles() { return engine.getProfiles(); } - public Iterable getComponents() { + public CloseableIterable getComponents() { return engine.getComponents(); } diff --git a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/data/DataValidatorHash.java b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/data/DataValidatorHash.java index 336a52141..ddd55d599 100644 --- a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/data/DataValidatorHash.java +++ b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/data/DataValidatorHash.java @@ -29,6 +29,7 @@ import fiftyone.pipeline.core.data.FlowData; import fiftyone.pipeline.engines.data.AspectPropertyValue; import fiftyone.pipeline.engines.exceptions.NoValueException; +import fiftyone.pipeline.engines.fiftyone.data.CloseableIterable; import fiftyone.pipeline.engines.fiftyone.data.ComponentMetaData; import fiftyone.pipeline.engines.fiftyone.data.FiftyOneAspectPropertyMetaData; @@ -91,8 +92,16 @@ public void validateData(FlowData data, boolean validEvidence) throws NoValueExc // rather than a fixed value, and that valid evidence produces at least // one result. int componentCount = 0; - for (ComponentMetaData component : engine.getComponents()) { - componentCount++; + // The collection holds the engine's data set until it is closed. + try (CloseableIterable components = + engine.getComponents()) { + for (ComponentMetaData component : components) { + componentCount++; + } + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new IllegalStateException(e); } int matchedUserAgents = elementData.getUserAgents().getValue().size(); assertTrue( diff --git a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/data/MetaDataHashTests.java b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/data/MetaDataHashTests.java index 4d0a9e220..8e4fa034a 100644 --- a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/data/MetaDataHashTests.java +++ b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/data/MetaDataHashTests.java @@ -65,7 +65,7 @@ public void MetaData_Hash_Reload() throws ExecutionException, InterruptedExcepti * assumption rather than failing. */ @Test - public void MetaData_Performance_Properties() { + public void MetaData_Performance_Properties() throws Exception { MetaDataHasherHash hasher = new MetaDataHasherHash(); long start = System.currentTimeMillis(); hasher.hashProperties(0, getWrapper()); @@ -83,7 +83,7 @@ public void MetaData_Performance_Properties() { * assumption rather than failing. */ @Test - public void MetaData_Performance_Components() { + public void MetaData_Performance_Components() throws Exception { MetaDataHasherHash hasher = new MetaDataHasherHash(); long start = System.currentTimeMillis(); hasher.hashComponents(0, getWrapper()); @@ -101,7 +101,7 @@ public void MetaData_Performance_Components() { * assumption rather than failing. */ @Test - public void MetaData_Performance_Values() { + public void MetaData_Performance_Values() throws Exception { MetaDataHasherHash hasher = new MetaDataHasherHash(); long start = System.currentTimeMillis(); hasher.hashValues(0, getWrapper()); @@ -119,7 +119,7 @@ public void MetaData_Performance_Values() { * assumption rather than failing. */ @Test - public void MetaData_Performance_Profiles() { + public void MetaData_Performance_Profiles() throws Exception { MetaDataHasherHash hasher = new MetaDataHasherHash(); long start = System.currentTimeMillis(); hasher.hashProfiles(0, getWrapper()); diff --git a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/data/MetaDataHasherHash.java b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/data/MetaDataHasherHash.java index f793a46fc..aa5a619ba 100644 --- a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/data/MetaDataHasherHash.java +++ b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/data/MetaDataHasherHash.java @@ -25,6 +25,7 @@ import fiftyone.devicedetection.hash.engine.onpremise.flowelements.Constants; import fiftyone.devicedetection.shared.testhelpers.Wrapper; import fiftyone.devicedetection.shared.testhelpers.data.MetaDataHasher; +import fiftyone.pipeline.engines.fiftyone.data.CloseableIterable; import fiftyone.pipeline.engines.fiftyone.data.ComponentMetaData; import fiftyone.pipeline.engines.fiftyone.data.FiftyOneAspectPropertyMetaData; import fiftyone.pipeline.engines.fiftyone.data.ProfileMetaData; @@ -34,7 +35,7 @@ public class MetaDataHasherHash implements MetaDataHasher { @Override - public int hashProperties(int hash, Wrapper wrapper) { + public int hashProperties(int hash, Wrapper wrapper) throws Exception { int i = 0; Iterable properties = wrapper.getProperties(); for (FiftyOneAspectPropertyMetaData property : properties) { @@ -42,11 +43,13 @@ public int hashProperties(int hash, Wrapper wrapper) { hash ^= property.hashCode(); int j = 0; - for (ValueMetaData value : property.getValues()) { - if (j % 10 == 0) { - hash ^= value.hashCode(); + try (CloseableIterable propertyValues = property.getValues()) { + for (ValueMetaData value : propertyValues) { + if (j % 10 == 0) { + hash ^= value.hashCode(); + } + j++; } - j++; } // Match metric property does not have component so don't include if (!property.getCategory().equals(Constants.MatchMetrics.CATEGORY)) { @@ -62,50 +65,60 @@ public int hashProperties(int hash, Wrapper wrapper) { } @Override - public int hashValues(int hash, Wrapper wrapper) { + public int hashValues(int hash, Wrapper wrapper) throws Exception { int i = 0; - for (ValueMetaData value : wrapper.getValues()) { - if (i % 100 == 0) { - hash ^= value.hashCode(); - hash ^= value.getProperty().hashCode(); + try (CloseableIterable values = wrapper.getValues()) { + for (ValueMetaData value : values) { + if (i % 100 == 0) { + hash ^= value.hashCode(); + hash ^= value.getProperty().hashCode(); + } + i++; } - i++; } return hash; } @Override - public int hashComponents(int hash, Wrapper wrapper) { - for (ComponentMetaData component : wrapper.getComponents()) { - hash ^= component.hashCode(); - int i = 0; - for (FiftyOneAspectPropertyMetaData property : component.getProperties()) { - if (i % 10 == 0) { - hash ^= property.hashCode(); + public int hashComponents(int hash, Wrapper wrapper) throws Exception { + try (CloseableIterable components = wrapper.getComponents()) { + for (ComponentMetaData component : components) { + hash ^= component.hashCode(); + int i = 0; + try (CloseableIterable componentProperties = component.getProperties()) { + for (FiftyOneAspectPropertyMetaData property : componentProperties) { + if (i % 10 == 0) { + hash ^= property.hashCode(); + } + i++; + } } - i++; + hash ^= component.getDefaultProfile().hashCode(); } - hash ^= component.getDefaultProfile().hashCode(); } return hash; } @Override - public int hashProfiles(int hash, Wrapper wrapper) { + public int hashProfiles(int hash, Wrapper wrapper) throws Exception { int i = 0; - for (ProfileMetaData profile : wrapper.getProfiles()) { - if (i % 100 == 0) { - hash ^= profile.hashCode(); - hash ^= profile.getComponent().hashCode(); - int j = 0; - for (ValueMetaData value : profile.getValues()) { - if (j % 10 == 0) { - hash ^= value.hashCode(); + try (CloseableIterable profiles = wrapper.getProfiles()) { + for (ProfileMetaData profile : profiles) { + if (i % 100 == 0) { + hash ^= profile.hashCode(); + hash ^= profile.getComponent().hashCode(); + int j = 0; + try (CloseableIterable profileValues = profile.getValues()) { + for (ValueMetaData value : profileValues) { + if (j % 10 == 0) { + hash ^= value.hashCode(); + } + j++; + } } - j++; } + i++; } - i++; } return hash; } diff --git a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/flowelements/EngineTests.java b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/flowelements/EngineTests.java index b73fbd83c..14b30de62 100644 --- a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/flowelements/EngineTests.java +++ b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/flowelements/EngineTests.java @@ -33,7 +33,7 @@ import fiftyone.devicedetection.shared.testhelpers.FileUtils; import fiftyone.pipeline.engines.Constants; -import static fiftyone.devicedetection.shared.testhelpers.FileUtils.LITE_HASH_DATA_FILE_NAME; +import static fiftyone.devicedetection.shared.testhelpers.FileUtils.LITE_HASH_DATA_FILE_PATH; import static fiftyone.devicedetection.shared.testhelpers.FileUtils.TAC_HASH_DATA_FILE_NAME; import static fiftyone.pipeline.util.FileFinder.getFilePath; @@ -62,11 +62,12 @@ private void testDataSourceTier(String fileName) throws Exception { assumeTrue(e.getMessage(), false); } - DeviceDetectionHashEngine engine = createEngine(dataFile); - String tier = engine.getDataSourceTier(); - engine.close(); + String tier; + try (DeviceDetectionHashEngine engine = createEngine(dataFile)) { + tier = engine.getDataSourceTier(); + } - if (fileName.equals(LITE_HASH_DATA_FILE_NAME)) { + if (fileName.equals(LITE_HASH_DATA_FILE_PATH)) { assertEquals("Lite", tier); } else { @@ -81,7 +82,7 @@ private void testDataSourceTier(String fileName) throws Exception { */ @Test public void Engine_Hash_GetDataSourceTier_Lite() throws Exception { - testDataSourceTier(LITE_HASH_DATA_FILE_NAME); + testDataSourceTier(LITE_HASH_DATA_FILE_PATH); } /** diff --git a/device-detection/device-detection.shared/pom.xml b/device-detection/device-detection.shared/pom.xml index 202797eb3..7d2618c67 100644 --- a/device-detection/device-detection.shared/pom.xml +++ b/device-detection/device-detection.shared/pom.xml @@ -12,7 +12,7 @@ device-detection.shared 51Degrees :: Device Detection :: Shared Shared components and utilities for building 51Degrees Pipeline API device detection engines in Java. Supports parsing HTTP headers to identify device type, model, OS, browser, and crawler information—an alternative approach to UAParser, DeviceAtlas, and WURFL-based solutions. - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=device-detection.shared-pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java diff --git a/device-detection/device-detection.shared/src/main/java/fiftyone/devicedetection/shared/testhelpers/FileUtils.java b/device-detection/device-detection.shared/src/main/java/fiftyone/devicedetection/shared/testhelpers/FileUtils.java index 8eff15a70..fe9e13a9e 100644 --- a/device-detection/device-detection.shared/src/main/java/fiftyone/devicedetection/shared/testhelpers/FileUtils.java +++ b/device-detection/device-detection.shared/src/main/java/fiftyone/devicedetection/shared/testhelpers/FileUtils.java @@ -47,6 +47,14 @@ public class FileUtils { private static File UA_FILE; public static final String EVIDENCE_FILE_NAME = "20000 Evidence Records.yml"; private static Optional EVIDENCE_FILE; + // Where each free file is in the data repository. A search for the path + // rather than the name does not stop at another file of the same name, + // such as a Git LFS pointer in an old device-detection-data folder. + public static final String LITE_HASH_DATA_FILE_PATH = + "device-detection/" + LITE_HASH_DATA_FILE_NAME; + public static final String UA_FILE_PATH = "device-detection/" + UA_FILE_NAME; + public static final String EVIDENCE_FILE_PATH = + "device-detection/" + EVIDENCE_FILE_NAME; /** * Helper to find the location of an Enterprise or Lite Hash file in the default search scope @@ -78,7 +86,7 @@ public static File getHashFile() { } catch (Exception ex) { try { // following throws exception if not found - HASH_DATA_FILE = Optional.of(getFilePath(LITE_HASH_DATA_FILE_NAME)); + HASH_DATA_FILE = Optional.of(getFilePath(LITE_HASH_DATA_FILE_PATH)); return HASH_DATA_FILE.get(); } catch (Exception exc) { HASH_DATA_FILE = Optional.empty(); @@ -93,7 +101,7 @@ public static File getEvidenceFile() { return EVIDENCE_FILE.orElse(null); } try { - EVIDENCE_FILE = Optional.of(getFilePath(EVIDENCE_FILE_NAME)); + EVIDENCE_FILE = Optional.of(getFilePath(EVIDENCE_FILE_PATH)); return EVIDENCE_FILE.get(); } catch (Exception e) { EVIDENCE_FILE = Optional.empty(); diff --git a/device-detection/device-detection.shared/src/test/java/fiftyone/devicedetection/shared/testhelpers/Wrapper.java b/device-detection/device-detection.shared/src/test/java/fiftyone/devicedetection/shared/testhelpers/Wrapper.java index 45fc54abe..82a7b6454 100644 --- a/device-detection/device-detection.shared/src/test/java/fiftyone/devicedetection/shared/testhelpers/Wrapper.java +++ b/device-detection/device-detection.shared/src/test/java/fiftyone/devicedetection/shared/testhelpers/Wrapper.java @@ -24,6 +24,7 @@ import fiftyone.pipeline.core.flowelements.Pipeline; import fiftyone.pipeline.engines.data.AspectData; +import fiftyone.pipeline.engines.fiftyone.data.CloseableIterable; import fiftyone.pipeline.engines.fiftyone.data.ComponentMetaData; import fiftyone.pipeline.engines.fiftyone.data.FiftyOneAspectPropertyMetaData; import fiftyone.pipeline.engines.fiftyone.data.ProfileMetaData; @@ -38,11 +39,11 @@ public interface Wrapper extends Closeable { Iterable getProperties(); - Iterable getValues(); + CloseableIterable getValues(); - Iterable getProfiles(); + CloseableIterable getProfiles(); - Iterable getComponents(); + CloseableIterable getComponents(); FiftyOneAspectEngine getEngine(); } \ No newline at end of file diff --git a/device-detection/device-detection.shared/src/test/java/fiftyone/devicedetection/shared/testhelpers/data/MetaDataHasher.java b/device-detection/device-detection.shared/src/test/java/fiftyone/devicedetection/shared/testhelpers/data/MetaDataHasher.java index 49240aecd..17fd7796e 100644 --- a/device-detection/device-detection.shared/src/test/java/fiftyone/devicedetection/shared/testhelpers/data/MetaDataHasher.java +++ b/device-detection/device-detection.shared/src/test/java/fiftyone/devicedetection/shared/testhelpers/data/MetaDataHasher.java @@ -25,11 +25,11 @@ import fiftyone.devicedetection.shared.testhelpers.Wrapper; public interface MetaDataHasher { - int hashProperties(int hash, Wrapper wrapper); + int hashProperties(int hash, Wrapper wrapper) throws Exception; - int hashValues(int hash, Wrapper wrapper); + int hashValues(int hash, Wrapper wrapper) throws Exception; - int hashComponents(int hash, Wrapper wrapper); + int hashComponents(int hash, Wrapper wrapper) throws Exception; - int hashProfiles(int hash, Wrapper wrapper); + int hashProfiles(int hash, Wrapper wrapper) throws Exception; } diff --git a/device-detection/device-detection.shared/src/test/java/fiftyone/devicedetection/shared/testhelpers/data/ValueTests.java b/device-detection/device-detection.shared/src/test/java/fiftyone/devicedetection/shared/testhelpers/data/ValueTests.java index 148e3a6e0..6826fb4d2 100644 --- a/device-detection/device-detection.shared/src/test/java/fiftyone/devicedetection/shared/testhelpers/data/ValueTests.java +++ b/device-detection/device-detection.shared/src/test/java/fiftyone/devicedetection/shared/testhelpers/data/ValueTests.java @@ -30,6 +30,7 @@ import fiftyone.pipeline.core.data.FlowData; import fiftyone.pipeline.engines.data.AspectPropertyValue; import fiftyone.pipeline.engines.exceptions.PropertyMissingException; +import fiftyone.pipeline.engines.fiftyone.data.CloseableIterable; import fiftyone.pipeline.engines.fiftyone.data.ComponentMetaData; import fiftyone.pipeline.engines.fiftyone.data.FiftyOneAspectPropertyMetaData; @@ -67,8 +68,10 @@ public static void matchedUserAgents(Wrapper wrapper) throws Exception { // depends on the data file, so assert the bound here and validate // every matched substring below. int componentCount = 0; - for (ComponentMetaData component : wrapper.getComponents()) { - componentCount++; + try (CloseableIterable components = wrapper.getComponents()) { + for (ComponentMetaData component : components) { + componentCount++; + } } int matchedCount = device.getUserAgents().getValue().size(); assertTrue( diff --git a/device-detection/device-detection/pom.xml b/device-detection/device-detection/pom.xml index 7ffa7b7d3..96eefb171 100644 --- a/device-detection/device-detection/pom.xml +++ b/device-detection/device-detection/pom.xml @@ -13,7 +13,7 @@ device-detection 51Degrees :: Device Detection Parse HTTP headers to detect the device type, model, OS, browser, and crawler information. This is an alternative to popular UAParser, DeviceAtlas, and WURFL packages. - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=device-detection-pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java diff --git a/device-detection/device-detection/src/test/java/fiftyone/devicedetection/CloudRequestOriginTests.java b/device-detection/device-detection/src/test/java/fiftyone/devicedetection/CloudRequestOriginTests.java index 72e4e2335..bb311b1b3 100644 --- a/device-detection/device-detection/src/test/java/fiftyone/devicedetection/CloudRequestOriginTests.java +++ b/device-detection/device-detection/src/test/java/fiftyone/devicedetection/CloudRequestOriginTests.java @@ -86,12 +86,11 @@ public void ResourceKeyWithOrigin() throws Exception { when(loggerFactory.getLogger(anyString())).thenReturn(logger); try { - Pipeline pipeline = new DeviceDetectionPipelineBuilder(loggerFactory, http) + try (Pipeline pipeline = new DeviceDetectionPipelineBuilder(loggerFactory, http) .useCloud(resourceKey) .setCloudRequestOrigin(this.origin) .build(); - - try(FlowData data = pipeline.createFlowData()) { + FlowData data = pipeline.createFlowData()) { data.addEvidence("query.user-agent", "Mozilla/5.0 (Linux; Android 8.0.0; Pixel 2 XL " + "Build/OPD1.170816.004) AppleWebKit/537.36 " + diff --git a/device-detection/device-detection/src/test/java/fiftyone/devicedetection/DeviceDetectionOnPremisePipelineBuilderTest.java b/device-detection/device-detection/src/test/java/fiftyone/devicedetection/DeviceDetectionOnPremisePipelineBuilderTest.java index c9430a527..21b9e5bd4 100644 --- a/device-detection/device-detection/src/test/java/fiftyone/devicedetection/DeviceDetectionOnPremisePipelineBuilderTest.java +++ b/device-detection/device-detection/src/test/java/fiftyone/devicedetection/DeviceDetectionOnPremisePipelineBuilderTest.java @@ -87,12 +87,13 @@ private ConfigHashSwig getConfigHashSwig(Pipeline pipeline) throws NoSuchFieldEx @Test public void setFilename_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setFilename(HASH_DATA_FILE_NAME, true) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(HASH_DATA_FILE_NAME, dataFileConfiguration.getDataFilePath()); - assertTrue(dataFileConfiguration.getCreateTempDataCopy()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(HASH_DATA_FILE_NAME, dataFileConfiguration.getDataFilePath()); + assertTrue(dataFileConfiguration.getCreateTempDataCopy()); + } } @Test @@ -118,210 +119,232 @@ public void setFilename_OtherExtension_ExceptionThrown() { @Test public void setEngineData_PropertyPropagatedToDataFileConfiguration() throws Exception { byte[] data = Files.readAllBytes(datafile.toPath()); - Pipeline pipeline = new DeviceDetectionPipelineBuilder(loggerFactory) + try (Pipeline pipeline = new DeviceDetectionPipelineBuilder(loggerFactory) .useOnPremise(new byte[]{}, Enums.DeviceDetectionAlgorithm.Hash) .setEngineData(data) .setShareUsage(false) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(data, dataFileConfiguration.getData()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(data, dataFileConfiguration.getData()); + } } @Test public void setEngineData_FilenameIsSet_PropertyNotPropagated() throws Exception { byte[] data = Files.readAllBytes(datafile.toPath()); - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setEngineData(data) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertNull(dataFileConfiguration.getData()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertNull(dataFileConfiguration.getData()); + } } @Test public void setShareUsage_False_SharedFlowElementNotAdded() throws Exception { - Pipeline pipeline = new DeviceDetectionPipelineBuilder(loggerFactory) + try (Pipeline pipeline = new DeviceDetectionPipelineBuilder(loggerFactory) .useOnPremise(datafile.getAbsolutePath(), false) .setShareUsage(false) - .build(); - List flowElements = pipeline.getFlowElements(); - assertEquals(1, flowElements.size()); - assertFalse(flowElements.get(0) instanceof ShareUsageElement); + .build()) { + List flowElements = pipeline.getFlowElements(); + assertEquals(1, flowElements.size()); + assertFalse(flowElements.get(0) instanceof ShareUsageElement); + } } @Test public void setShareUsage_True_SharedFlowElementAdded() throws Exception { - Pipeline pipeline = new DeviceDetectionPipelineBuilder(loggerFactory) + try (Pipeline pipeline = new DeviceDetectionPipelineBuilder(loggerFactory) .useOnPremise(datafile.getAbsolutePath(), false) .setShareUsage(true) - .build(); - List flowElements = pipeline.getFlowElements(); - assertEquals(2, flowElements.size()); - assertTrue(flowElements.get(0) instanceof ShareUsageElement); + .build()) { + List flowElements = pipeline.getFlowElements(); + assertEquals(2, flowElements.size()); + assertTrue(flowElements.get(0) instanceof ShareUsageElement); + } } @Test public void setAutoUpdate_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setAutoUpdate(true) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertTrue(dataFileConfiguration.getAutomaticUpdatesEnabled()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertTrue(dataFileConfiguration.getAutomaticUpdatesEnabled()); + } } @Test public void setDataFileSystemWatcher_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setDataFileSystemWatcher(true) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertTrue(dataFileConfiguration.getFileSystemWatcherEnabled()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertTrue(dataFileConfiguration.getFileSystemWatcherEnabled()); + } } @Test public void setDataUpdateService_RegisterDataFileTriggered() throws Exception { DataUpdateServiceDefault dataUpdateServiceDefault = mock(DataUpdateServiceDefault.class); - Pipeline pipeline = builder + try (Pipeline ignored = builder .setAutoUpdate(true) .setDataUpdateService(dataUpdateServiceDefault) - .build(); - verify(dataUpdateServiceDefault).registerDataFile(any()); + .build()) { + verify(dataUpdateServiceDefault).registerDataFile(any()); + } } @Test public void setDataUpdateOnStartup_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setDataUpdateOnStartup(true) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertTrue(dataFileConfiguration.getUpdateOnStartup()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertTrue(dataFileConfiguration.getUpdateOnStartup()); + } } @Test public void setUpdatePollingInterval_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setUpdatePollingInterval(10) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(10, dataFileConfiguration.getPollingIntervalSeconds()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(10, dataFileConfiguration.getPollingIntervalSeconds()); + } } @Test public void setUpdatePollingIntervalMillis_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setUpdatePollingIntervalMillis(10000) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(10, dataFileConfiguration.getPollingIntervalSeconds()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(10, dataFileConfiguration.getPollingIntervalSeconds()); + } } @Test public void setUpdateRandomisationMax_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setUpdateRandomisationMax(123) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(123, dataFileConfiguration.getMaxRandomisationSeconds()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(123, dataFileConfiguration.getMaxRandomisationSeconds()); + } } @Test public void setUpdateRandomisationMaxMillis_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setUpdateRandomisationMaxMillis(123000) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(123, dataFileConfiguration.getMaxRandomisationSeconds()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(123, dataFileConfiguration.getMaxRandomisationSeconds()); + } } @Test public void setDataUpdateLicenseKey_PropertyPropagatedToDataFileConfiguration() throws Exception { String key = "licence_key"; - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setDataUpdateLicenseKey(key) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertTrue(dataFileConfiguration.getDataUpdateLicenseKeys().contains(key)); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertTrue(dataFileConfiguration.getDataUpdateLicenseKeys().contains(key)); + } } @Test public void setConcurrency_PropertyPropagatedSwigConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setConcurrency(4) - .build(); - ConfigHashSwig swig = getConfigHashSwig(pipeline); - assertEquals(4, swig.getConcurrency()); + .build()) { + ConfigHashSwig swig = getConfigHashSwig(pipeline); + assertEquals(4, swig.getConcurrency()); + } } @Test public void setDifference_PropertyPropagatedSwigConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setDifference(123) - .build(); - ConfigHashSwig swig = getConfigHashSwig(pipeline); - assertEquals(123, swig.getDifference()); + .build()) { + ConfigHashSwig swig = getConfigHashSwig(pipeline); + assertEquals(123, swig.getDifference()); + } } @Test public void setAllowUnmatched_PropertyPropagatedSwigConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setAllowUnmatched(true) - .build(); - ConfigHashSwig swig = getConfigHashSwig(pipeline); - assertTrue(swig.getAllowUnmatched()); + .build()) { + ConfigHashSwig swig = getConfigHashSwig(pipeline); + assertTrue(swig.getAllowUnmatched()); + } } @Test public void setDrift_PropertyPropagatedSwigConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setDrift(123) - .build(); - ConfigHashSwig swig = getConfigHashSwig(pipeline); - assertEquals(123, swig.getDrift()); + .build()) { + ConfigHashSwig swig = getConfigHashSwig(pipeline); + assertEquals(123, swig.getDrift()); + } } @Test public void setUsePerformanceGraph_PropertyPropagatedSwigConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setUsePerformanceGraph(true) - .build(); - ConfigHashSwig swig = getConfigHashSwig(pipeline); - assertTrue(swig.getUsePerformanceGraph()); + .build()) { + ConfigHashSwig swig = getConfigHashSwig(pipeline); + assertTrue(swig.getUsePerformanceGraph()); + } } @Test public void setUsePredictiveGraph_PropertyPropagatedSwigConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setUsePredictiveGraph(true) - .build(); - ConfigHashSwig swig = getConfigHashSwig(pipeline); - assertTrue(swig.getUsePredictiveGraph()); + .build()) { + ConfigHashSwig swig = getConfigHashSwig(pipeline); + assertTrue(swig.getUsePredictiveGraph()); + } } @Test public void setDataUpdateUrl_PropertyPropagatedToDataFileConfiguration() throws Exception { String url = "https://update.me.com"; - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setDataUpdateUrl(url) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(url, dataFileConfiguration.getDataUpdateUrl()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(url, dataFileConfiguration.getDataUpdateUrl()); + } } @Test public void setDataUpdateVerifyMd5_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setDataUpdateVerifyMd5(true) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertTrue(dataFileConfiguration.getVerifyMd5()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertTrue(dataFileConfiguration.getVerifyMd5()); + } } @Test public void setDataUpdateUrlFormatter_PropertyPropagatedToDataFileConfiguration() throws Exception { DataUpdateUrlFormatter urlFormatter = mock(DataUpdateUrlFormatter.class); - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setDataUpdateUrlFormatter(urlFormatter) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(urlFormatter, dataFileConfiguration.getUrlFormatter()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(urlFormatter, dataFileConfiguration.getUrlFormatter()); + } } } \ No newline at end of file diff --git a/device-detection/device-detection/src/test/java/fiftyone/devicedetection/DeviceDetectionTests.java b/device-detection/device-detection/src/test/java/fiftyone/devicedetection/DeviceDetectionTests.java index 7d1bac24d..a59a0a6d6 100644 --- a/device-detection/device-detection/src/test/java/fiftyone/devicedetection/DeviceDetectionTests.java +++ b/device-detection/device-detection/src/test/java/fiftyone/devicedetection/DeviceDetectionTests.java @@ -52,7 +52,7 @@ import java.util.concurrent.Executors; import java.util.concurrent.Future; -import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_NAME; +import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_PATH; import static fiftyone.pipeline.core.Constants.EVIDENCE_HTTPHEADER_PREFIX; import static fiftyone.pipeline.core.Constants.EVIDENCE_SEPERATOR; import static fiftyone.pipeline.engines.Constants.PerformanceProfiles.*; @@ -99,7 +99,7 @@ public class DeviceDetectionTests { @BeforeClass public static void initClass() throws IOException { userAgents = new UserAgentGenerator( - FileFinder.getFilePath(UA_FILE_NAME)); + FileFinder.getFilePath(UA_FILE_PATH)); } private static String reportErrors(Collection errors) { diff --git a/device-detection/pom.xml b/device-detection/pom.xml index 8aef20aea..b255d1e53 100644 --- a/device-detection/pom.xml +++ b/device-detection/pom.xml @@ -1,4 +1,4 @@ - 4.0.0 @@ -9,7 +9,11 @@ pom 51Degrees :: Pipeline :: Device Detection - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java + + 51Degrees + https://51degrees.com/device-detection?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pipeline.device-detection&utm_term=organization + Device detection engines for the 51Degrees Pipeline API @@ -52,6 +56,7 @@ 3.7.1 6.5.3 0.7.0 + 3.26.0 true false @@ -304,6 +309,40 @@ + + + org.apache.maven.plugins + maven-pmd-plugin + ${maven-pmd-plugin.version} + + + + ${maven.multiModuleProjectDirectory}/ci/pmd-resource-ruleset.xml + + true + true + true + false + + **/interop/swig/*.java + + + + + check-resources-closed + process-test-classes + + check + + + + diff --git a/examples/device-detection/README.md b/examples/device-detection/README.md index 5d2edd339..4c80193e5 100644 --- a/examples/device-detection/README.md +++ b/examples/device-detection/README.md @@ -21,11 +21,19 @@ Among other things, the examples illustrate: - use of device detection pipeline for off-line processing tasks - configuring device detection trade-offs between speed and conserving memory +## Data files + +The on-premise examples use the free Lite data file and the evidence files in +`cxx/data/device-detection`, a folder of the +[data](https://github.com/51Degrees/data) repository. An example that cannot +find one of these files runs the data repository's download script for it, +and logs the command it runs. + ## Cloud resource keys -You will require [resource keys](https://51degrees.com/documentation/_info__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud-resource-keys) +You will require [resource keys](https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud-resource-keys) to use the Cloud API, as described on our website. Get resource keys from -our [configurator](https://configure.51degrees.com/?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud-resource-keys), see our [documentation](https://51degrees.com/documentation/_concepts__configurator.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud-resource-keys) on +our [configurator](https://configure.51degrees.com/?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud-resource-keys), see our [documentation](https://51degrees.com/documentation/_services__configurator.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud-resource-keys) on how to use this. A resource key configured with the properties needed @@ -41,23 +49,18 @@ resource keys and license keys at our [pricing page](https://51degrees.com/prici ## Running examples with changes to Pipeline packages -A common use case is to make a change to the Pipeline logic in -device-detection-java and then use these examples to observe the results of the -change. - -By default, the examples are configured to use the packages from Maven central. -In order to produce and use local packages instead: +A common use case is to make a change to the Device Detection or Pipeline +packages in this repository and then use these examples to observe the +results of the change. -- Clone and make your changes to device-detection-java -- Set the version of the device detection packages that we're going to create to 0.0.0: - `mvn versions:set-property -Dproperty="project.version" "-DnewVersion=0.0.0"` -- Create and install the packages locally (skipping tests is needed): - `mvn clean install [-DskipTests]` -- Modify the POM for the examples to reference these new local packages. This can - be done by editing the POM directly or by using the command line: - `mvn versions:set-property -Dproperty="device-detection.version" "-DnewVersion=0.0.0"` +By default, the examples use the packages from Maven Central. To build them +against the packages in this repository instead, run Maven from the root of +the repository with the `dev` profile, as the +[root README](../../README.md#examples) describes: -The same principle can be applied to incorporate changes in pipeline-java if needed. +``` +mvn install -DskipTests -Pdev +``` ## Examples @@ -114,7 +117,8 @@ PORT=8100 java -jar ./web/getting-started.onprem/target/device-detection-java-ex The on-premise web example takes its data file from the `51DEGREES_DD_PATH` environment variable, then from `TestDataFile`, and otherwise uses the free Lite file -in `device-detection-data`. +in `cxx/data/device-detection`, which it downloads before the server starts if it is +missing. ### Native library access diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/GettingStartedCloud.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/GettingStartedCloud.java index c97a1868a..846e1fcb4 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/GettingStartedCloud.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/GettingStartedCloud.java @@ -84,7 +84,7 @@ public static void run(String resourceKey, /* In this example, we use the DeviceDetectionPipelineBuilder and configure it in code. For more information about pipelines in general see the documentation at - https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-gettingstartedcloud.java&utm_term=run + https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-gettingstartedcloud.java&utm_term=run Note that we wrap the creation of a pipeline in a try/resources to control its lifecycle */ try (Pipeline pipeline = new DeviceDetectionPipelineBuilder() diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/GettingStartedOnPrem.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/GettingStartedOnPrem.java index 9d8819565..44b7227fb 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/GettingStartedOnPrem.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/GettingStartedOnPrem.java @@ -58,12 +58,13 @@ public class GettingStartedOnPrem { private static final Logger logger = LoggerFactory.getLogger(GettingStartedOnPrem.class); - /* In this example, by default, the 51degrees "Lite" file needs to be somewhere in the project - space, or you may specify another file as a command line parameter. + /* In this example, by default, the 51degrees "Lite" file is used from the data repository in + cxx/data, and is downloaded if it is missing, or you may specify another file as a command line + parameter. Note that the Lite data file is only used for illustration, and has limited accuracy and capabilities. Find out about the Enterprise data file here: https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-gettingstartedonprem.java&utm_term=lite_v_4_1_hash */ - public static String LITE_V_4_1_HASH = "51Degrees-LiteV4.1.hash"; + public static String LITE_V_4_1_HASH = "device-detection/51Degrees-LiteV4.1.hash"; public static String ENTERPRISE_HASH = "TAC-HashV41.hash"; public static void main(String[] args) throws Exception { @@ -87,7 +88,8 @@ public static void run(String dataFile, OutputStream outputStream) throws Exception { logger.info("Running GettingStarted example"); try { - String dataFileLocation = getFilePath(dataFile).getAbsolutePath(); + String dataFileLocation = + DataFileHelper.findDataFile(dataFile).getAbsolutePath(); // the location of the test data file is interpolated in the pipeline // configuration file System.setProperty("TestDataFile", dataFileLocation); @@ -100,7 +102,7 @@ public static void run(String dataFile, options contained in the file "gettingStartedOnPrem.xml". For more information about pipelines in general see the documentation at - https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-gettingstartedonprem.java&utm_term=run + https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-gettingstartedonprem.java&utm_term=run Note that we wrap the creation of a pipeline in a try/resources to control its lifecycle */ // the configuration file is in the resources directory diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MatchMetrics.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MatchMetrics.java index a54bfe71c..4c0aa55a1 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MatchMetrics.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MatchMetrics.java @@ -27,7 +27,6 @@ import fiftyone.devicedetection.examples.shared.EvidenceHelper; import fiftyone.devicedetection.hash.engine.onpremise.data.DeviceDataHash; import fiftyone.devicedetection.hash.engine.onpremise.flowelements.DeviceDetectionHashEngine; -import fiftyone.devicedetection.shared.testhelpers.FileUtils; import fiftyone.pipeline.core.data.ElementPropertyMetaData; import fiftyone.pipeline.core.data.FlowData; import fiftyone.pipeline.core.flowelements.Pipeline; @@ -80,7 +79,7 @@ static void run(String dataFile, List> evidenceList, boolean showDescs, OutputStream out) throws Exception { PrintWriter writer = new PrintWriter(out, true); if (Objects.isNull(dataFile)) { - dataFile = FileUtils.getHashFileName(); + dataFile = DataFileHelper.getDataFileLocation(null); } logger.info("Constructing pipeline from file " + dataFile); diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MetadataOnPrem.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MetadataOnPrem.java index a58e739ab..350172405 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MetadataOnPrem.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MetadataOnPrem.java @@ -21,11 +21,13 @@ * ********************************************************************* */ package fiftyone.devicedetection.examples.console; +import fiftyone.pipeline.engines.fiftyone.data.FiftyOneAspectPropertyMetaData; import fiftyone.devicedetection.examples.shared.DataFileHelper; import fiftyone.devicedetection.hash.engine.onpremise.flowelements.DeviceDetectionHashEngine; import fiftyone.devicedetection.hash.engine.onpremise.flowelements.DeviceDetectionHashEngineBuilder; import fiftyone.pipeline.core.data.EvidenceKeyFilterWhitelist; import fiftyone.pipeline.engines.Constants; +import fiftyone.pipeline.engines.fiftyone.data.CloseableIterable; import fiftyone.pipeline.engines.fiftyone.data.ComponentMetaData; import fiftyone.pipeline.engines.fiftyone.data.ProfileMetaData; import fiftyone.pipeline.engines.fiftyone.data.ValueMetaData; @@ -45,12 +47,13 @@ public class MetadataOnPrem { private static final Logger logger = LoggerFactory.getLogger(GettingStartedOnPrem.class); - /* In this example, by default, the 51degrees "Lite" file needs to be somewhere in the project - space, or you may specify another file as a command line parameter. + /* In this example, by default, the 51degrees "Lite" file is used from the data repository in + cxx/data, and is downloaded if it is missing, or you may specify another file as a command line + parameter. Note that the Lite data file is only used for illustration, and has limited accuracy and capabilities. Find out about the Enterprise data file here: https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-metadataonprem.java&utm_term=lite_v_4_1_hash */ - public static String LITE_V_4_1_HASH = "51Degrees-LiteV4.1.hash"; + public static String LITE_V_4_1_HASH = "device-detection/51Degrees-LiteV4.1.hash"; public static void main(String[] args) throws Exception { configureLogback(getFilePath("logback.xml")); @@ -63,7 +66,8 @@ public static void run(String dataFile, OutputStream output) throws Exception { logger.info("Running MetadataOnPrem example"); String dataFileLocation; try { - dataFileLocation = getFilePath(dataFile).getAbsolutePath(); + dataFileLocation = + DataFileHelper.findDataFile(dataFile).getAbsolutePath(); } catch (Exception e) { DataFileHelper.cantFindDataFile(dataFile); throw e; @@ -79,7 +83,7 @@ public static void run(String dataFile, OutputStream output) throws Exception { // example. See the documentation for more detail on this and other // configuration options: // https://51degrees.com/documentation/_device_detection__features__performance_options.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-metadataonprem.java&utm_term=run - // https://51degrees.com/documentation/_features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-metadataonprem.java&utm_term=run + // https://51degrees.com/documentation/_pipeline_api__features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-metadataonprem.java&utm_term=run .setPerformanceProfile(Constants.PerformanceProfiles.LowMemory) // inhibit auto-update of the data file for this test .setAutoUpdate(false) @@ -132,71 +136,92 @@ private static void outputEvidenceKeyDetails(DeviceDetectionHashEngine ddEngine, } private static void outputProfileDetails(DeviceDetectionHashEngine ddEngine, - PrintWriter output) { - // Group the profiles by component and then output the number of profiles + PrintWriter output) throws Exception { + // Group the profiles by component and then output the number of profiles // for each component. - Map> groups = - StreamSupport.stream(ddEngine.getProfiles().spliterator(), false) - .collect(Collectors.groupingBy(p -> p.getComponent().getName())); - groups.forEach((k,v)->output.format("%s Profiles: %d\n", k , v.size())); + // + // A try-with-resource block MUST be used for the collections of + // metadata returned by the engine. Each one holds a reference to the + // engine's data set until it is closed, so if it is left open the + // data set cannot be freed when the engine is closed. + Map groups; + try (CloseableIterable profiles = ddEngine.getProfiles()) { + groups = StreamSupport.stream(profiles.spliterator(), false) + .collect(Collectors.groupingBy( + p -> p.getComponent().getName(), + Collectors.counting())); + } + groups.forEach((k,v)->output.format("%s Profiles: %d\n", k , v)); } // Output the component name as well as a list of all the associated properties. // If we're outputting to console then we also add some formatting to make it // more readable. - private static void outputComponents(DeviceDetectionHashEngine ddEngine, PrintWriter output){ - ddEngine.getComponents().forEach(c -> { - output.println("Component - "+ c.getName()); - outputProperties(c, output); - }); + private static void outputComponents(DeviceDetectionHashEngine ddEngine, PrintWriter output) + throws Exception { + // See outputProfileDetails for why this is a try-with-resource block. + try (CloseableIterable components = ddEngine.getComponents()) { + for (ComponentMetaData c : components) { + output.println("Component - "+ c.getName()); + outputProperties(c, output); + } + } } - private static void outputProperties(ComponentMetaData component, PrintWriter output) { - if (component.getProperties().iterator().hasNext() == false) { - output.println(" ... no properties"); - return; + private static void outputProperties(ComponentMetaData component, PrintWriter output) + throws Exception { + // The properties of a component, and the values of a property, are + // collections that are closed in the same way. + try (CloseableIterable properties = + component.getProperties()) { + if (properties.iterator().hasNext() == false) { + output.println(" ... no properties"); + return; + } + for (FiftyOneAspectPropertyMetaData property : properties) { + // Output some details about the property. + // If we're outputting to console then we also add some formatting to make it + // more readable. + output.format(" Property - %s [Category: %s] (%s)\n " + + "Description: %s\n", + property.getName(), + property.getCategory(), + property.getType().getName(), + property.getDescription()); + + // Next, output a list of the possible values this property can have. + // Most properties in the Device Metrics category do not have defined + // values so exclude them. + if (property.getCategory().equals("Device Metrics")==false) { + StringBuilder values = new StringBuilder(" Possible " + + "values: "); + try (CloseableIterable propertyValues = + property.getValues()) { + Spliterator spliterator2 = + propertyValues.spliterator(); + StreamSupport.stream(spliterator2, false) + .limit(20) + .forEach(value -> { + // add value + values.append(truncateToNl(value.getName())); + // add description if exists + String d = value.getDescription(); + if (Objects.nonNull(d) && d.isEmpty() == false) { + values.append("(") + .append(d) + .append(")"); + } + values.append(","); + }); + + if (spliterator2.estimateSize() > 20) { + values.append(" + more ..."); + } + } + output.println(values); + } + } } - component.getProperties() - .forEach(property-> { - // Output some details about the property. - // If we're outputting to console then we also add some formatting to make it - // more readable. - output.format(" Property - %s [Category: %s] (%s)\n " + - "Description: %s\n", - property.getName(), - property.getCategory(), - property.getType().getName(), - property.getDescription()); - - // Next, output a list of the possible values this property can have. - // Most properties in the Device Metrics category do not have defined - // values so exclude them. - if (property.getCategory().equals("Device Metrics")==false) { - StringBuilder values = new StringBuilder(" Possible " + - "values: "); - Spliterator spliterator2 = - property.getValues().spliterator(); - StreamSupport.stream(spliterator2, false) - .limit(20) - .forEach(value -> { - // add value - values.append(truncateToNl(value.getName())); - // add description if exists - String d = value.getDescription(); - if (Objects.nonNull(d) && d.isEmpty() == false) { - values.append("(") - .append(d) - .append(")"); - } - values.append(","); - }); - - if (spliterator2.estimateSize() > 20) { - values.append(" + more ..."); - } - output.println(values); - } - }); } diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MinimalExample.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MinimalExample.java index 4b996daba..8c5fb5206 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MinimalExample.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MinimalExample.java @@ -29,7 +29,7 @@ /** * This is the code for the minimal example displayed in the configurator. Please see - * + * * Getting Started Console Cloud for a fuller example. */ diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/NativeModelCloud.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/NativeModelCloud.java index 754f0596a..3c6c724ac 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/NativeModelCloud.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/NativeModelCloud.java @@ -56,7 +56,7 @@ public static void run(String resourceKey, OutputStream os) throws Exception { // This example creates the pipeline and engines in code. For a demonstration // of how to do this using a configuration file instead, see the TacCloud example. // For more information about builders in general see the documentation at - // https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-nativemodelcloud.java&utm_term=run + // https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-nativemodelcloud.java&utm_term=run resourceKey = getOrSetSuperResourceKey(resourceKey, NATIVE_MODEL_EXAMPLE_RESOURCE_KEY_NAME); try (PrintWriter output = new PrintWriter(os)) { diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/OfflineProcessing.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/OfflineProcessing.java index 031c63368..8fc928a56 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/OfflineProcessing.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/OfflineProcessing.java @@ -67,22 +67,22 @@ */ public class OfflineProcessing { static final Logger logger = LoggerFactory.getLogger(OfflineProcessing.class); - // This 51degrees "Lite" file (distributed with the source) needs to - // be somewhere in the project space + // This 51degrees "Lite" file is used from the data repository in + // cxx/data, and is downloaded if it is missing // // Note that the Lite data file is only used for illustration, and has // limited accuracy and capabilities. Find out about the Enterprise data // file here: https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-offlineprocessing.java&utm_term=lite_v_4_1_hash public static final String LITE_V_4_1_HASH = - "device-detection-data/51Degrees-LiteV4.1.hash"; - // This 51degrees file of 20,000 examples (distributed with the source) - // needs to be somewhere in the project space + "device-detection/51Degrees-LiteV4.1.hash"; + // This 51degrees file of 20,000 examples is used from the same folder, + // and is downloaded if it is missing public static final String HEADER_EVIDENCE_YML = - "device-detection-data/20000 Evidence Records.yml"; + "device-detection/20000 Evidence Records.yml"; public static void main(String[] args) throws Exception { configureLogback(getFilePath("logback.xml")); - File evidenceFile = getFilePath(HEADER_EVIDENCE_YML); + File evidenceFile = DataFileHelper.findDataFile(HEADER_EVIDENCE_YML); run(LITE_V_4_1_HASH, Files.newInputStream(evidenceFile.toPath()), System.out); } @@ -100,7 +100,8 @@ public static void run(String dataFile, InputStream is, OutputStream os) throws String detectionFile; try { - detectionFile = getFilePath(dataFile).getAbsolutePath(); + detectionFile = + DataFileHelper.findDataFile(dataFile).getAbsolutePath(); } catch (Exception e) { DataFileHelper.cantFindDataFile(dataFile); throw e; diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/PerformanceBenchmark.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/PerformanceBenchmark.java index aeaf3180c..8c6f2b0fe 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/PerformanceBenchmark.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/PerformanceBenchmark.java @@ -388,8 +388,9 @@ public PerformanceConfiguration(Constants.PerformanceProfiles profile, * @example PerformanceBenchmark.java * The example illustrates a "clock-time" benchmark for assessing detection speed. * - * Using a YAML formatted evidence file - "20000 Evidence Records.yml" - supplied with the - * distribution or can be obtained from the [data repository on Github](https://github.com/51Degrees/device-detection-data/blob/master/20000%20Evidence%20Records.yml). + * Using a YAML formatted evidence file - "20000 Evidence Records.yml" - which is downloaded by + * the scripts in the [data repository on GitHub](https://github.com/51Degrees/data), and by the + * example itself when the file is missing. * * It's important to understand the trade-offs between performance, memory usage and accuracy, that * the 51Degrees pipeline configuration makes available, and this example shows a range of diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/ShareUsageExample.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/ShareUsageExample.java index 9e9ba2320..50d44c364 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/ShareUsageExample.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/ShareUsageExample.java @@ -22,6 +22,7 @@ package fiftyone.devicedetection.examples.console; +import fiftyone.devicedetection.examples.shared.DataFileHelper; import fiftyone.pipeline.core.data.FlowData; import fiftyone.pipeline.core.flowelements.Pipeline; import fiftyone.pipeline.core.flowelements.PipelineBuilder; @@ -75,9 +76,10 @@ public class ShareUsageExample { static final Logger logger = LoggerFactory.getLogger(ShareUsageExample.class); - // Evidence file with 20,000 examples + // Evidence file with 20,000 examples, from the data repository in + // cxx/data. The example downloads the file if it is missing. public static final String HEADER_EVIDENCE_YML = - "device-detection-data/20000 Evidence Records.yml"; + "device-detection/20000 Evidence Records.yml"; // Configurable settings private static final int MINIMUM_ENTRIES_PER_MESSAGE = 10; // Send after 10 entries (default is 50) @@ -100,7 +102,7 @@ public class ShareUsageExample { public static void main(String[] args) throws Exception { configureLogback(getFilePath("logback.xml")); - File evidenceFile = getFilePath(HEADER_EVIDENCE_YML); + File evidenceFile = DataFileHelper.findDataFile(HEADER_EVIDENCE_YML); run(Files.newInputStream(evidenceFile.toPath())); } @@ -135,9 +137,12 @@ public static void run(InputStream is) throws Exception { // .setIncludedQueryStringParameter("campaign") .build(); - // Build a pipeline with only the ShareUsageElement + // Build a pipeline with only the ShareUsageElement. The pipeline is + // asked to close the element, which sends any data still queued, as + // it is not closed anywhere else. try (Pipeline pipeline = new PipelineBuilder() .addFlowElement(shareUsageElement) + .setAutoCloseElements(true) .build()) { logger.info("Pipeline built successfully with ShareUsageElement"); diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/TacCloud.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/TacCloud.java index 91213032b..ee5ee1e0b 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/TacCloud.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/TacCloud.java @@ -72,7 +72,7 @@ public static void run(String resourceKey, OutputStream os) throws Exception { // For a demonstration of how to do this in code instead, see the // NativeModelLookup example. // For more information about builders in general see the documentation at - // https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-taccloud.java&utm_term=run + // https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-taccloud.java&utm_term=run // the configuration file is in the resources directory File optionsFile = getFilePath("tacCloud.xml"); diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/UpdateDataFile.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/UpdateDataFile.java index 6dae1a226..d3babcd27 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/UpdateDataFile.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/UpdateDataFile.java @@ -73,9 +73,9 @@ * ``` * ## Lite Data File * Lite data files (free-to-use, limited capabilities, no license key required) are created roughly - * once a month and cannot be updated using auto-update, they may be downloaded from - * [Github](https://github.com/51Degrees/device-detection-data) and are included with - * source distributions of this software. + * once a month and cannot be updated using auto-update, they may be downloaded with the + * scripts in the [data repository](https://github.com/51Degrees/data) on GitHub, which is + * the cxx/data folder of the source. * # Update on Start-Up * You can configure the pipeline builder to download an Enterprise data file on start-up. * ## Pre-Requisites diff --git a/examples/device-detection/console/src/main/resources/gettingStartedOnPrem.xml b/examples/device-detection/console/src/main/resources/gettingStartedOnPrem.xml index fed362bfd..ee634b786 100644 --- a/examples/device-detection/console/src/main/resources/gettingStartedOnPrem.xml +++ b/examples/device-detection/console/src/main/resources/gettingStartedOnPrem.xml @@ -5,7 +5,7 @@ false false - ${TestDataFile:-cxx/device-detection-data/51Degrees-LiteV4.1.hash} + ${TestDataFile:-cxx/data/device-detection/51Degrees-LiteV4.1.hash} LowMemory DeviceDetectionHashEngine diff --git a/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/GettingStartedOnPremTest.java b/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/GettingStartedOnPremTest.java index 1c272b52e..779a70fc2 100644 --- a/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/GettingStartedOnPremTest.java +++ b/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/GettingStartedOnPremTest.java @@ -31,7 +31,7 @@ public class GettingStartedOnPremTest { @Test public void gettingStartedOnPremTest() throws Exception { GettingStartedOnPrem.run(FileUtils.getHashFileName() == null - ? FileUtils.LITE_HASH_DATA_FILE_NAME + ? FileUtils.LITE_HASH_DATA_FILE_PATH : FileUtils.getHashFileName(), EvidenceHelper.setUpEvidence(), System.out); } diff --git a/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/MetadataOnPremTest.java b/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/MetadataOnPremTest.java index 32d3b8882..a128b1393 100644 --- a/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/MetadataOnPremTest.java +++ b/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/MetadataOnPremTest.java @@ -29,7 +29,7 @@ public class MetadataOnPremTest { @Test public void gettingMetaDataOnPremTest() throws Exception { MetadataOnPrem.run(FileUtils.getHashFileName() == null - ? FileUtils.LITE_HASH_DATA_FILE_NAME + ? FileUtils.LITE_HASH_DATA_FILE_PATH : FileUtils.getHashFileName(), System.out); } } \ No newline at end of file diff --git a/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/OfflineProcessingTest.java b/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/OfflineProcessingTest.java index 80d767c58..4861f14d6 100644 --- a/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/OfflineProcessingTest.java +++ b/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/OfflineProcessingTest.java @@ -32,7 +32,7 @@ public class OfflineProcessingTest { @Test public void offlineProcessingTest() throws Exception { OfflineProcessing.run(FileUtils.getHashFileName() == null - ? FileUtils.LITE_HASH_DATA_FILE_NAME + ? FileUtils.LITE_HASH_DATA_FILE_PATH : FileUtils.getHashFileName(), new FileInputStream(Objects.requireNonNull(FileUtils.getEvidenceFile())), System.out); diff --git a/examples/device-detection/pom.xml b/examples/device-detection/pom.xml index d7617e6e6..fa55e5534 100644 --- a/examples/device-detection/pom.xml +++ b/examples/device-detection/pom.xml @@ -62,6 +62,7 @@ 2.0.20 1.3.16 + 3.26.0 @@ -165,6 +166,40 @@ + + + org.apache.maven.plugins + maven-pmd-plugin + ${maven-pmd-plugin.version} + + + + ${maven.multiModuleProjectDirectory}/ci/pmd-resource-ruleset.xml + + true + true + true + false + + **/interop/swig/*.java + + + + + check-resources-closed + process-test-classes + + check + + + + diff --git a/examples/device-detection/shared/src/main/java/fiftyone/devicedetection/examples/shared/DataFileHelper.java b/examples/device-detection/shared/src/main/java/fiftyone/devicedetection/examples/shared/DataFileHelper.java index c5aab3928..337f6e165 100644 --- a/examples/device-detection/shared/src/main/java/fiftyone/devicedetection/examples/shared/DataFileHelper.java +++ b/examples/device-detection/shared/src/main/java/fiftyone/devicedetection/examples/shared/DataFileHelper.java @@ -32,10 +32,15 @@ import org.slf4j.LoggerFactory; import java.io.File; +import java.io.IOException; import java.text.SimpleDateFormat; import java.time.Instant; import java.time.temporal.ChronoUnit; +import java.util.ArrayList; +import java.util.Arrays; import java.util.Date; +import java.util.List; +import java.util.Locale; import java.util.Objects; import static fiftyone.pipeline.util.FileFinder.getFilePath; @@ -43,6 +48,12 @@ public class DataFileHelper { static Logger logger = LoggerFactory.getLogger(DataFileHelper.class); + /** + * Sent with each download, so the Distributor can count the downloads + * made by these examples. + */ + private static final String DOWNLOAD_SOURCE = "examples-51degrees-java"; + public static class DatafileInfo { FiftyOneDataFile fileInfo; String tier; @@ -90,9 +101,10 @@ public static void logDataFileInfo(DeviceDetectionHashEngine engine) { "than {} days old. A more recent data file " + "may be needed to correctly detect the latest " + "devices, browsers, etc.", daysOld); - logger.info("The latest 'Lite' data file is available " + - "from the device-detection-data repository on GitHub " + - "https://github.com/51Degrees/device-detection-data. " + + logger.info("Run get-data-files.ps1 -Update or " + + "get-data-files.sh -u in cxx/data, which is the data " + + "repository https://github.com/51Degrees/data, to get " + + "the latest 'Lite' data file. " + "Find out about the Enterprise data file, which " + "includes automatic daily updates, on our pricing " + "page: https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=shared-src-main-java-fiftyone-devicedetection-examples-shared-datafilehelper.java&utm_term=data-file-age-warning"); @@ -102,9 +114,10 @@ public static void logDataFileInfo(DeviceDetectionHashEngine engine) { public static void cantFindDataFile(String dataFile) { logger.error("Could not find the data file '{}' which must be " + "somewhere in the project space to be found.", dataFile); - logger.info("The latest 'Lite' data file is available " + - "from the device-detection-data repository on GitHub " + - "https://github.com/51Degrees/device-detection-data. " + + logger.info("The free 'Lite' data file and the evidence files " + + "are downloaded by get-data-files.ps1 or get-data-files.sh " + + "in cxx/data, which is the data repository " + + "https://github.com/51Degrees/data. " + "Find out about the Enterprise data file, which includes " + "automatic daily updates, on our pricing " + "page: https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=shared-src-main-java-fiftyone-devicedetection-examples-shared-datafilehelper.java&utm_term=data-file-not-found"); @@ -120,12 +133,12 @@ public static void cantFindDataFile(String dataFile) { @SuppressWarnings("RedundantThrows") public static File getEvidenceFile(String evidenceFilename) throws Exception { if (Objects.isNull(evidenceFilename)) { - evidenceFilename = FileUtils.EVIDENCE_FILE_NAME; + evidenceFilename = FileUtils.EVIDENCE_FILE_PATH; } File evidenceFile; try { - evidenceFile = FileFinder.getFilePath(evidenceFilename); + evidenceFile = findDataFile(evidenceFilename); } catch (Exception e) { logger.error("Could not find evidence file {}", evidenceFilename); throw e; @@ -134,7 +147,8 @@ public static File getEvidenceFile(String evidenceFilename) throws Exception { } /** - * Tries to find the passed file, or if null a default file + * Tries to find the passed file, or if null the first data file found, + * or the free Lite file when there is none * @param dataFilename a filename to find * @return a full pathname * @throws Exception if the file was not found @@ -144,13 +158,90 @@ public static String getDataFileLocation(String dataFilename) throws Exception { if (Objects.isNull(dataFilename)) { dataFilename = FileUtils.getHashFileName(); } + if (Objects.isNull(dataFilename)) { + dataFilename = FileUtils.LITE_HASH_DATA_FILE_PATH; + } String dataFileLocation; try { - dataFileLocation = getFilePath(dataFilename).getAbsolutePath(); + dataFileLocation = findDataFile(dataFilename).getAbsolutePath(); } catch (Exception e) { DataFileHelper.cantFindDataFile(dataFilename); throw e; } return dataFileLocation; } + + /** + * Finds a file below the project root, as + * {@link FileFinder#getFilePath(String)} does. A free data file that is + * missing has a {@code .link} in the data repository, so the data + * repository's script is run to download it, and the file is looked for + * again. + * @param file the name of the file, or the end of its path + * @return the file + * @throws IllegalArgumentException if the file was not found, naming the + * command that downloads it when there is one + */ + public static File findDataFile(String file) { + try { + return getFilePath(file); + } catch (IllegalArgumentException notFound) { + File link; + try { + link = getFilePath(file + ".link"); + } catch (IllegalArgumentException noLink) { + throw notFound; + } + String command = runDownload(link); + try { + return getFilePath(file); + } catch (IllegalArgumentException stillNotFound) { + String message = "Could not download '" + file + + "'. Run this command to download it: " + command; + logger.error(message); + throw new IllegalArgumentException(message, stillNotFound); + } + } + } + + /** + * Runs the data repository's script for the file that a link is for, + * and waits for it to finish. The data repository is the folder above + * the one the link is in. + * @param link the {@code .link} file + * @return the command, as it would be typed + */ + private static String runDownload(File link) { + String name = link.getName().substring( + 0, link.getName().length() - ".link".length()); + File data = link.getAbsoluteFile().getParentFile().getParentFile(); + List command; + if (System.getProperty("os.name", "") + .toLowerCase(Locale.ROOT).startsWith("windows")) { + command = Arrays.asList("powershell", "-NoProfile", + "-ExecutionPolicy", "Bypass", "-File", + new File(data, "get-data-files.ps1").getPath(), name, + "-Source", DOWNLOAD_SOURCE); + } else { + command = Arrays.asList("sh", + new File(data, "get-data-files.sh").getPath(), + "-s", DOWNLOAD_SOURCE, name); + } + List typed = new ArrayList<>(); + for (String argument : command) { + typed.add(argument.contains(" ") ? + "\"" + argument + "\"" : argument); + } + String line = String.join(" ", typed); + logger.info("'{}' is missing, so it is being downloaded by running {}", + name, line); + try { + new ProcessBuilder(command).inheritIO().start().waitFor(); + } catch (IOException e) { + logger.error("Could not run {}. {}", line, e.getMessage()); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + return line; + } } diff --git a/examples/device-detection/web/getting-started.onprem/src/main/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPrem.java b/examples/device-detection/web/getting-started.onprem/src/main/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPrem.java index 22b31c746..8387720c6 100644 --- a/examples/device-detection/web/getting-started.onprem/src/main/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPrem.java +++ b/examples/device-detection/web/getting-started.onprem/src/main/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPrem.java @@ -22,6 +22,7 @@ package fiftyone.devicedetection.examples.web; +import fiftyone.devicedetection.examples.shared.DataFileHelper; import fiftyone.devicedetection.shared.DeviceData; import fiftyone.pipeline.core.configuration.PipelineOptions; import fiftyone.pipeline.core.configuration.PipelineOptionsFactory; @@ -34,6 +35,7 @@ import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; +import java.io.File; import java.io.PrintWriter; import static fiftyone.common.testhelpers.LogbackHelper.configureLogback; @@ -48,7 +50,8 @@ *

* The data file is taken from the 51DEGREES_DD_PATH environment variable, then * from TestDataFile (an environment variable or system property), and otherwise - * the free Lite file in device-detection-data is used. + * the free Lite file in cxx/data/device-detection is used, which is downloaded + * before the server starts if it is missing. *

* The server listens on port 8081 and stops when Enter is pressed. If the PORT * environment variable is set, the server listens on that port instead and runs @@ -63,6 +66,16 @@ public static void main(String[] args) throws Exception { configureLogback(getFilePath("logback.xml")); logger.info("Running Example {}", GettingStartedWebOnPrem.class); + // If the data file the configuration names is not at that path, it + // is searched for, and the free Lite file is downloaded if missing. + String dataFile = PipelineOptionsFactory.getOptionsFromFile( + getFilePath(resourceBase) + "/WEB-INF/51Degrees-OnPrem.xml") + .findAndSubstitute("DeviceDetectionHashEngine", "DataFile"); + if (!new File(dataFile).exists()) { + System.setProperty("TestDataFile", + DataFileHelper.getDataFileLocation(dataFile)); + } + String portEnv = System.getenv("PORT"); if (portEnv != null) { // Automated runs (e.g. the unified Selenium suite) inject a port and have no diff --git a/examples/device-detection/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml b/examples/device-detection/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml index f785a6607..bba6dd60e 100644 --- a/examples/device-detection/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml +++ b/examples/device-detection/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml @@ -32,7 +32,7 @@ variable, then from TestDataFile (an environment variable or system property), and otherwise the free Lite file is used. --> - ${51DEGREES_DD_PATH:-${TestDataFile:-cxx/device-detection-data/51Degrees-LiteV4.1.hash}} + ${51DEGREES_DD_PATH:-${TestDataFile:-cxx/data/device-detection/51Degrees-LiteV4.1.hash}} LowMemory DeviceDetectionHashEngine diff --git a/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPremTest.java b/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPremTest.java index 70636a724..d6a94896f 100644 --- a/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPremTest.java +++ b/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPremTest.java @@ -43,7 +43,7 @@ public class GettingStartedWebOnPremTest { @BeforeClass public static void startJetty() throws Exception { System.setProperty("TestDataFile", FileUtils.getHashFile() == null - ? FileUtils.LITE_HASH_DATA_FILE_NAME + ? FileUtils.LITE_HASH_DATA_FILE_PATH : FileUtils.getHashFile().getAbsolutePath()); SERVER = EmbedJetty.startWebApp(getFilePath(resourceBase).getAbsolutePath(), 8081); } diff --git a/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/OnPremDataFileOptionTest.java b/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/OnPremDataFileOptionTest.java index 7c898e848..108fbd615 100644 --- a/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/OnPremDataFileOptionTest.java +++ b/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/OnPremDataFileOptionTest.java @@ -43,7 +43,7 @@ public class OnPremDataFileOptionTest { private static final String DD_PATH = "51DEGREES_DD_PATH"; private static final String TEST_DATA_FILE = "TestDataFile"; private static final String LITE = - "cxx/device-detection-data/51Degrees-LiteV4.1.hash"; + "cxx/data/device-detection/51Degrees-LiteV4.1.hash"; private String savedDdPath; private String savedTestDataFile; diff --git a/examples/ip-intelligence/.gitignore b/examples/ip-intelligence/.gitignore index 092783861..23e4fdfef 100644 --- a/examples/ip-intelligence/.gitignore +++ b/examples/ip-intelligence/.gitignore @@ -48,5 +48,3 @@ SESSIONS.ser 51Degrees-IPIV4EnterpriseIpiV41.ipi.gz 51Degrees-IPIV4AsnIpiV41.ipi 51Degrees-IPIV4AsnIpiV41.ipi.gz -# device detection data file used by the mixed examples -device-detection-data/ diff --git a/examples/ip-intelligence/README.md b/examples/ip-intelligence/README.md index 022af4b4a..9344df4e0 100644 --- a/examples/ip-intelligence/README.md +++ b/examples/ip-intelligence/README.md @@ -10,16 +10,20 @@ These examples are not distributed as maven jars and need to be built by you. ## Required Files -See [ip-intelligence-data](https://github.com/51Degrees/ip-intelligence-data/) -repository for instructions on obtaining the necessary data files for on-premise IP intelligence. - -**IMPORTANT**: The enterprise IP Intelligence data file must be placed in the `ip-intelligence-data` -directory at the root of this repository. The expected filename is `51Degrees-EnterpriseIpiV41.ipi`. - -Two free data files are also usable for many of the examples. The ASN file -(`51Degrees-IPIV4AsnIpiV41.ipi`) is included in the `ip-intelligence-data` submodule, and the -Lite file (`51Degrees-LiteV41.ipi`) can be downloaded by running -`ip-intelligence-data/get-lite-file-from-azure.ps1` (or `.sh`). When the enterprise file is +The data files for on-premise IP intelligence are kept in `cxx/data/ip-intelligence`, a +folder of the [data](https://github.com/51Degrees/data) repository, which is a submodule of +cxx. + +**IMPORTANT**: The enterprise IP Intelligence data file must be placed in the +`cxx/data/ip-intelligence` directory of this repository. The expected filename is +`51Degrees-EnterpriseIpiV41.ipi`. + +Two free data files are also usable for many of the examples, the ASN file +(`51Degrees-IPIV4AsnIpiV41.ipi`) and the Lite file (`51Degrees-LiteV41.ipi`). Run +`cxx/data/get-data-files.sh ip-intelligence` from the root of this repository to download +them, or `cxx\data\get-data-files.ps1 ip-intelligence` in PowerShell on Windows. An example +that needs one of them and cannot find it runs the script for that file, and logs the +command it runs. When the enterprise file is absent, the tests for examples that work with any data file fall back to the ASN file and then the Lite file. The OfflineProcessing and PerformanceBenchmark examples need the RegisteredName property, which the ASN file does not contain, so their tests fall back to @@ -27,12 +31,16 @@ the Lite file only. Tests for examples needing enterprise-only properties (Metri and Suspicious) are skipped instead. ``` -ip-intelligence-java-examples/ -├── ip-intelligence-data/ -│ └── 51Degrees-EnterpriseIpiV41.ipi -├── console/ -├── web/ -└── shared/ +51degrees-java/ +├── cxx/ +│ └── data/ +│ └── ip-intelligence/ +│ └── 51Degrees-EnterpriseIpiV41.ipi +└── examples/ + └── ip-intelligence/ + ├── console/ + ├── web/ + └── shared/ ``` The examples locate the data file in the following order: @@ -40,9 +48,9 @@ The examples locate the data file in the following order: 1. The `51DEGREES_IPI_PATH` environment variable or system property, which can be set to an explicit path to the data file. 2. A search of the project folder hierarchy for the expected file name. -3. The expected location `ip-intelligence-data/51Degrees-EnterpriseIpiV41.ipi` +3. The expected location `cxx/data/ip-intelligence/51Degrees-EnterpriseIpiV41.ipi` (the configuration files for the getting started examples fall back to the - free Lite file `ip-intelligence-data/51Degrees-LiteV41.ipi`). + free Lite file `cxx/data/ip-intelligence/51Degrees-LiteV41.ipi`). This project contains sub-modules - **console**, giving examples that are intended to be run from the command line/console and **web**, illustrating use @@ -121,14 +129,14 @@ to get a paid subscription with more properties. ## Running Examples and Tests -**IMPORTANT**: All examples and tests must be run from the repository root directory (`ip-intelligence-java-examples/`). +**IMPORTANT**: All examples and tests must be run from this folder, `examples/ip-intelligence`. ### Running Tests To run all tests: ```bash -# From the repository root directory +# From examples/ip-intelligence mvn test ``` @@ -147,14 +155,14 @@ mvn test -pl web/getting-started.onprem Console examples can be run directly using Maven exec plugin: ```bash -# From the repository root directory +# From examples/ip-intelligence mvn compile exec:java -pl console -Dexec.mainClass="fiftyone.ipintelligence.examples.console.GettingStartedOnPrem" ``` Web examples: ```bash -# From the repository root directory +# From examples/ip-intelligence mvn compile exec:java -pl web/getting-started.onprem -Dexec.mainClass="fiftyone.ipintelligence.examples.web.GettingStartedWebOnPrem" ``` diff --git a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/CompareOnPrem.java b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/CompareOnPrem.java index 06538b059..4b994e3b9 100644 --- a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/CompareOnPrem.java +++ b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/CompareOnPrem.java @@ -112,10 +112,10 @@ public class CompareOnPrem { /** * Relative path to the CSV file of truth records distributed with the - * ip-intelligence-data submodule. + * data repository, in cxx/data/ip-intelligence. */ public static final String GEOIP_COMPARISON_EVIDENCE_REL_PATH = - "ip-intelligence-data/geoip_comparison_evidence.csv"; + "ip-intelligence/geoip_comparison_evidence.csv"; /** * Default name of the output file written to the working directory. @@ -202,7 +202,7 @@ public static void run(String dataFile, // We use the max performance profile for optimal lookup // speed in this example. See the documentation for more detail // on this and other configuration options. - // https://51degrees.com/documentation/_features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-compareonprem.java&utm_term=automatic-datafile-updates + // https://51degrees.com/documentation/_pipeline_api__features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-compareonprem.java&utm_term=automatic-datafile-updates .setPerformanceProfile(Constants.PerformanceProfiles.MaxPerformance) // inhibit sharing usage for this example .setShareUsage(false) diff --git a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedCloud.java b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedCloud.java index fb63e8546..684990a91 100644 --- a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedCloud.java +++ b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedCloud.java @@ -125,7 +125,7 @@ public static void run(String resourceKey, /* In this example, we use the IPIntelligencePipelineBuilder and configure it in code. For more information about pipelines in general see the documentation at - https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-gettingstartedcloud.java&utm_term=configuration-builders + https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-gettingstartedcloud.java&utm_term=configuration-builders Note that we wrap the creation of a pipeline in a try/resources to control its lifecycle */ IPIntelligenceCloudPipelineBuilder builder = new IPIntelligencePipelineBuilder() diff --git a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedCloudMixed.java b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedCloudMixed.java index e5bf11069..efca03fdc 100644 --- a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedCloudMixed.java +++ b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedCloudMixed.java @@ -144,10 +144,13 @@ public static void run(String resourceKey, IPIntelligenceCloudEngine ipIntelligenceEngine = new IPIntelligenceCloudEngineBuilder(loggerFactory).build(); + // The engines are not closed anywhere else, so the pipeline is asked + // to close them when it is closed. try (Pipeline pipeline = new PipelineBuilder(loggerFactory) .addFlowElement(cloudRequestEngine) .addFlowElement(deviceDetectionEngine) .addFlowElement(ipIntelligenceEngine) + .setAutoCloseElements(true) .build()) { // carry out some sample detections diff --git a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedMixed.java b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedMixed.java index 9bab40ab5..07f33c130 100644 --- a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedMixed.java +++ b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedMixed.java @@ -83,7 +83,7 @@ public class GettingStartedMixed { private static final Logger logger = LoggerFactory.getLogger(GettingStartedMixed.class); - public static final String DD_DATA_FILE_DEFAULT = "device-detection-data/51Degrees-LiteV4.1.hash"; + public static final String DD_DATA_FILE_DEFAULT = "device-detection/51Degrees-LiteV4.1.hash"; public static final String IPI_DATA_FILE_DEFAULT = DataFileHelper.ENTERPRISE_DATA_FILE_REL_PATH; public static void main(String[] args) throws Exception { @@ -271,11 +271,12 @@ private static String resolveDataFile(String dataFile) throws IOException { if (Files.exists(relativePath)) { return relativePath.toAbsolutePath().toString(); } - // Try FileFinder (searches within project directory) + // Search within the project directory, downloading the free Lite + // file if it is missing try { - return getFilePath(dataFile).getAbsolutePath(); + return DataFileHelper.findDataFile(dataFile).getAbsolutePath(); } catch (Exception e) { - throw new IOException("Data file not found: " + dataFile); + throw new IOException("Data file not found: " + dataFile, e); } } } diff --git a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedOnPrem.java b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedOnPrem.java index 8c6e4d0b9..41a7712a5 100644 --- a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedOnPrem.java +++ b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedOnPrem.java @@ -139,7 +139,7 @@ public static void run(String dataFile, options contained in the file "gettingStartedOnPrem.xml". For more information about pipelines in general see the documentation at - https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-gettingstartedonprem.java&utm_term=run + https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-gettingstartedonprem.java&utm_term=run Note that we wrap the creation of a pipeline in a try/resources to control its lifecycle */ // the configuration file is in the resources directory diff --git a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/MetadataOnPrem.java b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/MetadataOnPrem.java index 7a96b562f..6e127722d 100644 --- a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/MetadataOnPrem.java +++ b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/MetadataOnPrem.java @@ -21,6 +21,7 @@ * ********************************************************************* */ package fiftyone.ipintelligence.examples.console; +import fiftyone.pipeline.engines.fiftyone.data.FiftyOneAspectPropertyMetaData; import fiftyone.ipintelligence.examples.shared.DataFileHelper; import fiftyone.ipintelligence.engine.onpremise.flowelements.IPIntelligenceOnPremiseEngine; import fiftyone.ipintelligence.engine.onpremise.flowelements.IPIntelligenceOnPremiseEngineBuilder; @@ -175,59 +176,67 @@ private static void outputComponents(IPIntelligenceOnPremiseEngine engine, Print throws Exception { // See outputProfileDetails for why this is a try-with-resource block. try (CloseableIterable components = engine.getComponents()) { - components.forEach(c -> { + for (ComponentMetaData c : components) { output.println("Component - "+ c.getName()); outputProperties(c, output); - }); + } } } - private static void outputProperties(ComponentMetaData component, PrintWriter output) { - if (component.getProperties().iterator().hasNext() == false) { - output.println(" ... no properties"); - return; + private static void outputProperties(ComponentMetaData component, PrintWriter output) + throws Exception { + // The properties of a component, and the values of a property, are + // collections that are closed in the same way. + try (CloseableIterable properties = + component.getProperties()) { + if (properties.iterator().hasNext() == false) { + output.println(" ... no properties"); + return; + } + for (FiftyOneAspectPropertyMetaData property : properties) { + // Output some details about the property. + // If we're outputting to console then we also add some formatting to make it + // more readable. + output.format(" Property - %s [Category: %s] (%s)\n " + + "Description: %s\n", + property.getName(), + property.getCategory(), + property.getType().getName(), + property.getDescription()); + + // Next, output a list of the possible values this property can have. + // Most properties in the Metrics category do not have defined + // values so exclude them. + if (property.getCategory().equals("Metrics")==false) { + StringBuilder values = new StringBuilder(" Possible " + + "values: "); + try (CloseableIterable propertyValues = + property.getValues()) { + Spliterator spliterator2 = + propertyValues.spliterator(); + StreamSupport.stream(spliterator2, false) + .limit(20) + .forEach(value -> { + // add value + values.append(truncateToNl(value.getName())); + // add description if exists + String d = value.getDescription(); + if (Objects.nonNull(d) && d.isEmpty() == false) { + values.append("(") + .append(d) + .append(")"); + } + values.append(","); + }); + + if (spliterator2.estimateSize() > 20) { + values.append(" + more ..."); + } + } + output.println(values); + } + } } - component.getProperties() - .forEach(property-> { - // Output some details about the property. - // If we're outputting to console then we also add some formatting to make it - // more readable. - output.format(" Property - %s [Category: %s] (%s)\n " + - "Description: %s\n", - property.getName(), - property.getCategory(), - property.getType().getName(), - property.getDescription()); - - // Next, output a list of the possible values this property can have. - // Most properties in the Metrics category do not have defined - // values so exclude them. - if (property.getCategory().equals("Metrics")==false) { - StringBuilder values = new StringBuilder(" Possible " + - "values: "); - Spliterator spliterator2 = - property.getValues().spliterator(); - StreamSupport.stream(spliterator2, false) - .limit(20) - .forEach(value -> { - // add value - values.append(truncateToNl(value.getName())); - // add description if exists - String d = value.getDescription(); - if (Objects.nonNull(d) && d.isEmpty() == false) { - values.append("(") - .append(d) - .append(")"); - } - values.append(","); - }); - - if (spliterator2.estimateSize() > 20) { - values.append(" + more ..."); - } - output.println(values); - } - }); } diff --git a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/MetricsOnPrem.java b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/MetricsOnPrem.java index aaf6ddac4..347c35b60 100644 --- a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/MetricsOnPrem.java +++ b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/MetricsOnPrem.java @@ -343,7 +343,7 @@ public static void run(String dataFile, // detection speed in this example. See the // documentation for more detail on this and other // configuration options. - // https://51degrees.com/documentation/_features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-metricsonprem.java&utm_term=automatic-datafile-updates + // https://51degrees.com/documentation/_pipeline_api__features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-metricsonprem.java&utm_term=automatic-datafile-updates .setPerformanceProfile(Constants.PerformanceProfiles.MaxPerformance) // inhibit sharing usage for this example .setShareUsage(false) @@ -507,11 +507,13 @@ private static String getValue(ProfileMetaData profile, String name) String result = null; // Every value is closed, not only the first, as each one wraps a // native object of its own. - for (ValueMetaData value : profile.getValues(name)) { - if (result == null) { - result = value.getName(); + try (CloseableIterable values = profile.getValues(name)) { + for (ValueMetaData value : values) { + if (result == null) { + result = value.getName(); + } + value.close(); } - value.close(); } return result; } diff --git a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/OfflineProcessing.java b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/OfflineProcessing.java index 983d25997..608989b73 100644 --- a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/OfflineProcessing.java +++ b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/OfflineProcessing.java @@ -40,7 +40,7 @@ * advanced operation this example can be used to experiment with tuning IP Intelligence for * performance using Performance Profile settings. * - * Evidence files can be obtained from the [ip-intelligence-data repository](https://github.com/51Degrees/ip-intelligence-data). + * Evidence files can be obtained from the [data repository](https://github.com/51Degrees/data). * * This example is available in full on [GitHub](https://github.com/51Degrees/51degrees-java/blob/main/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/OfflineProcessing.java). * @@ -99,10 +99,10 @@ public class OfflineProcessing { // be somewhere in the project space // // For testing, contact us to obtain an enterprise data file: https://51degrees.com/contact-us?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-offlineprocessing.java&utm_term=datadir - private static final String dataDir = "ip-intelligence-data"; + private static final String dataDir = "ip-intelligence"; // This 51degrees file of 20,000 examples (distributed with the source) // needs to be somewhere in the project space. Additional evidence files - // can be obtained from the [ip-intelligence-data repository](https://github.com/51Degrees/ip-intelligence-data) + // can be obtained from the [data repository](https://github.com/51Degrees/data) public static final String HEADER_EVIDENCE_YML = dataDir + "/evidence.yml"; diff --git a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/SuspiciousOnPrem.java b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/SuspiciousOnPrem.java index dd46f9cc5..3df68b732 100644 --- a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/SuspiciousOnPrem.java +++ b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/SuspiciousOnPrem.java @@ -120,7 +120,7 @@ public static void run(String dataFile, /* In this example, we use the IPIntelligencePipelineBuilder and configure it in code. For more information about builders in general see the documentation at - https://51degrees.com/documentation/_concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-suspiciousonprem.java&utm_term=configuration-builders + https://51degrees.com/documentation/_pipeline_api__concepts__configuration__builders__index.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-suspiciousonprem.java&utm_term=configuration-builders Note that we wrap the creation of a pipeline in a try/resources to control its lifecycle */ try (Pipeline pipeline = new IPIntelligencePipelineBuilder() @@ -128,8 +128,8 @@ public static void run(String dataFile, // We use the max performance profile for optimal lookup speed in this // example. See the documentation for more detail on this and other // configuration options. - // https://51degrees.com/documentation/_features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-suspiciousonprem.java&utm_term=automatic-datafile-updates - // https://51degrees.com/documentation/_features__usage_sharing.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-suspiciousonprem.java&utm_term=usage-sharing + // https://51degrees.com/documentation/_pipeline_api__features__automatic_datafile_updates.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-suspiciousonprem.java&utm_term=automatic-datafile-updates + // https://51degrees.com/documentation/_pipeline_api__features__usage_sharing.html?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-suspiciousonprem.java&utm_term=usage-sharing .setPerformanceProfile(Constants.PerformanceProfiles.MaxPerformance) // inhibit sharing usage for this example, usually this should be set to "true" .setShareUsage(false) diff --git a/examples/ip-intelligence/console/src/main/resources/gettingStartedMixed.xml b/examples/ip-intelligence/console/src/main/resources/gettingStartedMixed.xml index c6dba02ab..cfb3acbcf 100644 --- a/examples/ip-intelligence/console/src/main/resources/gettingStartedMixed.xml +++ b/examples/ip-intelligence/console/src/main/resources/gettingStartedMixed.xml @@ -5,7 +5,7 @@ false false - ${TestDeviceDetectionDataFile:-cxx/device-detection-data/51Degrees-LiteV4.1.hash} + ${TestDeviceDetectionDataFile:-cxx/data/device-detection/51Degrees-LiteV4.1.hash} LowMemory DeviceDetectionHashEngine @@ -14,7 +14,7 @@ false false - ${TestDataFile:-cxx/ip-intelligence-data/51Degrees-LiteV41.ipi} + ${TestDataFile:-cxx/data/ip-intelligence/51Degrees-LiteV41.ipi} LowMemory IPIntelligenceOnPremiseEngine diff --git a/examples/ip-intelligence/console/src/main/resources/gettingStartedOnPrem.xml b/examples/ip-intelligence/console/src/main/resources/gettingStartedOnPrem.xml index 7f1171389..f156a40b5 100644 --- a/examples/ip-intelligence/console/src/main/resources/gettingStartedOnPrem.xml +++ b/examples/ip-intelligence/console/src/main/resources/gettingStartedOnPrem.xml @@ -6,7 +6,7 @@ false false - ${TestDataFile:-cxx/ip-intelligence-data/51Degrees-LiteV41.ipi} + ${TestDataFile:-cxx/data/ip-intelligence/51Degrees-LiteV41.ipi} LowMemory diff --git a/examples/ip-intelligence/pom.xml b/examples/ip-intelligence/pom.xml index 945f34223..2e3624a91 100644 --- a/examples/ip-intelligence/pom.xml +++ b/examples/ip-intelligence/pom.xml @@ -63,6 +63,7 @@ 2.0.20 1.3.16 + 3.26.0 @@ -165,6 +166,40 @@ + + + org.apache.maven.plugins + maven-pmd-plugin + ${maven-pmd-plugin.version} + + + + ${maven.multiModuleProjectDirectory}/ci/pmd-resource-ruleset.xml + + true + true + true + false + + **/interop/swig/*.java + + + + + check-resources-closed + process-test-classes + + check + + + + diff --git a/examples/ip-intelligence/shared/src/main/java/fiftyone/ipintelligence/examples/shared/DataFileHelper.java b/examples/ip-intelligence/shared/src/main/java/fiftyone/ipintelligence/examples/shared/DataFileHelper.java index d448834d4..4cfa22723 100644 --- a/examples/ip-intelligence/shared/src/main/java/fiftyone/ipintelligence/examples/shared/DataFileHelper.java +++ b/examples/ip-intelligence/shared/src/main/java/fiftyone/ipintelligence/examples/shared/DataFileHelper.java @@ -39,7 +39,11 @@ import java.text.SimpleDateFormat; import java.time.Instant; import java.time.temporal.ChronoUnit; +import java.util.ArrayList; +import java.util.Arrays; import java.util.Date; +import java.util.List; +import java.util.Locale; import java.util.Objects; import static fiftyone.pipeline.util.FileFinder.getFilePath; @@ -48,31 +52,38 @@ public class DataFileHelper { static Logger logger = LoggerFactory.getLogger(DataFileHelper.class); /** - * Relative path to the enterprise IP Intelligence data file from the examples repository root. + * Sent with each download, so the Distributor can count the downloads + * made by these examples. + */ + private static final String DOWNLOAD_SOURCE = "examples-51degrees-java"; + + /** + * Folder and name of the enterprise IP Intelligence data file, which + * belongs in cxx/data/ip-intelligence, a folder of the data repository. * This path is used by both console and web examples for testing. - * The data file is downloaded to this location by CI scripts. + * The data file is linked to this location by CI scripts. */ - public static final String ENTERPRISE_DATA_FILE_REL_PATH = "ip-intelligence-data/51Degrees-EnterpriseIpiV41.ipi"; + public static final String ENTERPRISE_DATA_FILE_REL_PATH = "ip-intelligence/51Degrees-EnterpriseIpiV41.ipi"; /** - * Relative path to the freely downloadable Lite IP Intelligence data file. - * Run ip-intelligence-data/get-lite-file-from-azure.ps1 (or .sh) to obtain it. + * Folder and name of the free Lite IP Intelligence data file, which the + * data repository's get-data-files script downloads to that folder. * It contains the RegisteredCountry, RegisteredName and RegisteredOwner properties. */ - public static final String LITE_DATA_FILE_REL_PATH = "ip-intelligence-data/51Degrees-LiteV41.ipi"; + public static final String LITE_DATA_FILE_REL_PATH = "ip-intelligence/51Degrees-LiteV41.ipi"; /** - * Relative path to the ASN IP Intelligence data file which is part of the - * ip-intelligence-data submodule. It contains the Asn and AsnName properties. + * Folder and name of the free ASN IP Intelligence data file, which the + * same script downloads. It contains the Asn and AsnName properties. */ - public static final String ASN_DATA_FILE_REL_PATH = "ip-intelligence-data/51Degrees-IPIV4AsnIpiV41.ipi"; + public static final String ASN_DATA_FILE_REL_PATH = "ip-intelligence/51Degrees-IPIV4AsnIpiV41.ipi"; /** * Find the best IP Intelligence data file available, preferring the - * enterprise file, then the ASN file from the ip-intelligence-data - * submodule, then the Lite file. Suitable for examples that work with - * whatever properties the data file contains, where any data file - * allows the example to run. + * enterprise file, then the free ASN file, then the free Lite file, + * and downloading a free file that is missing. Suitable for examples + * that work with whatever properties the data file contains, where any + * data file allows the example to run. * @return a resolved data file path, or null when none is available */ public static String findAvailableDataFile() { @@ -180,7 +191,7 @@ public static File getEvidenceFile(String evidenceFilename) throws Exception { File evidenceFile; try { - evidenceFile = FileFinder.getFilePath(evidenceFilename); + evidenceFile = findDataFile(evidenceFilename); } catch (Exception e) { logger.error("Could not find evidence file {}", evidenceFilename); throw e; @@ -244,13 +255,92 @@ public static String getDataFileLocation(String dataFilename) throws Exception { if (Files.exists(dataPath)) { return dataPath.toAbsolutePath().toString(); } - DataFileHelper.cantFindDataFile(dataFilename); - throw new IOException("Data file not found at relative path: " + dataFilename + - ". Searched in project directories and current working directory." + - " An explicit path can be supplied via the " + IPI_PATH_ENV_VAR + - " environment variable."); + // Download it, if it is a free data file + try { + return findDataFile(dataFilename).getAbsolutePath(); + } catch (IllegalArgumentException notFound) { + DataFileHelper.cantFindDataFile(dataFilename); + throw new IOException("Data file not found at relative path: " + dataFilename + + ". Searched in project directories and current working directory." + + " An explicit path can be supplied via the " + IPI_PATH_ENV_VAR + + " environment variable.", notFound); + } } } } } + + /** + * Finds a file below the project root, as + * {@link FileFinder#getFilePath(String)} does. A free data file that is + * missing has a {@code .link} in the data repository, so the data + * repository's script is run to download it, and the file is looked for + * again. + * @param file the name of the file, or the end of its path + * @return the file + * @throws IllegalArgumentException if the file was not found, naming the + * command that downloads it when there is one + */ + public static File findDataFile(String file) { + try { + return getFilePath(file); + } catch (IllegalArgumentException notFound) { + File link; + try { + link = getFilePath(file + ".link"); + } catch (IllegalArgumentException noLink) { + throw notFound; + } + String command = runDownload(link); + try { + return getFilePath(file); + } catch (IllegalArgumentException stillNotFound) { + String message = "Could not download '" + file + + "'. Run this command to download it: " + command; + logger.error(message); + throw new IllegalArgumentException(message, stillNotFound); + } + } + } + + /** + * Runs the data repository's script for the file that a link is for, + * and waits for it to finish. The data repository is the folder above + * the one the link is in. + * @param link the {@code .link} file + * @return the command, as it would be typed + */ + private static String runDownload(File link) { + String name = link.getName().substring( + 0, link.getName().length() - ".link".length()); + File data = link.getAbsoluteFile().getParentFile().getParentFile(); + List command; + if (System.getProperty("os.name", "") + .toLowerCase(Locale.ROOT).startsWith("windows")) { + command = Arrays.asList("powershell", "-NoProfile", + "-ExecutionPolicy", "Bypass", "-File", + new File(data, "get-data-files.ps1").getPath(), name, + "-Source", DOWNLOAD_SOURCE); + } else { + command = Arrays.asList("sh", + new File(data, "get-data-files.sh").getPath(), + "-s", DOWNLOAD_SOURCE, name); + } + List typed = new ArrayList<>(); + for (String argument : command) { + typed.add(argument.contains(" ") ? + "\"" + argument + "\"" : argument); + } + String line = String.join(" ", typed); + logger.info("'{}' is missing, so it is being downloaded by running {}", + name, line); + try { + new ProcessBuilder(command).inheritIO().start().waitFor(); + } catch (IOException e) { + logger.error("Could not run {}. {}", line, e.getMessage()); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + return line; + } } diff --git a/examples/ip-intelligence/web/getting-started.api/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedApi.java b/examples/ip-intelligence/web/getting-started.api/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedApi.java index 2de381969..9887afc57 100644 --- a/examples/ip-intelligence/web/getting-started.api/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedApi.java +++ b/examples/ip-intelligence/web/getting-started.api/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedApi.java @@ -94,7 +94,7 @@ * and a JSON builder element which serialises the results. *

* The data file can be supplied as a command line argument. When no argument is given - * the enterprise data file is looked for in the ip-intelligence-data directory. + * the enterprise data file is looked for in the cxx/data/ip-intelligence directory. *

* Note that the .NET version of this example also includes a device detection engine * in the same pipeline. This Java version is IP Intelligence only. @@ -127,16 +127,29 @@ public void init() throws ServletException { .setAutoUpdate(false) .build(dataFileLocation, false); // The SequenceElement provides the sequence evidence required by - // the JsonBuilderElement. + // the JsonBuilderElement. The pipeline closes all three elements, + // the engine and its data file included, when destroy closes it. pipeline = new PipelineBuilder(LoggerFactory.getILoggerFactory()) .addFlowElement(new SequenceElementBuilder( LoggerFactory.getILoggerFactory()).build()) .addFlowElement(engine) .addFlowElement(new JsonBuilderElementBuilder( LoggerFactory.getILoggerFactory()).build()) + .setAutoCloseElements(true) .build(); DataFileHelper.logDataFileInfo(engine); } catch (Exception e) { + // A servlet whose init fails is never destroyed, so whatever was + // built is closed here. + try { + if (pipeline != null) { + pipeline.close(); + } else if (engine != null) { + engine.close(); + } + } catch (Exception closeException) { + e.addSuppressed(closeException); + } throw new ServletException("Failed to create pipeline from data file '" + dataFileLocation + "'", e); } diff --git a/examples/ip-intelligence/web/getting-started.cloud.mixed/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebCloudMixed.java b/examples/ip-intelligence/web/getting-started.cloud.mixed/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebCloudMixed.java index 4090fc110..977e3b3fa 100644 --- a/examples/ip-intelligence/web/getting-started.cloud.mixed/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebCloudMixed.java +++ b/examples/ip-intelligence/web/getting-started.cloud.mixed/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebCloudMixed.java @@ -118,7 +118,9 @@ protected void processRequest(HttpServletRequest request, HttpServletResponse re // Get the IP address parameter from the request for custom lookup String inputIpAddress = request.getParameter("client-ip"); - // The detection has already been carried out by the PipelineFilter + // The detection has already been carried out by the PipelineFilter, + // which closes the FlowData at the end of the request, so it is not + // closed here. FlowData flowData = flowDataProvider.getFlowData(request); // Determine target IP for display. @@ -149,14 +151,6 @@ protected void processRequest(HttpServletRequest request, HttpServletResponse re String htmlTemplate = loadTemplate(resourceBase + "/WEB-INF/html/index.html"); String processedHtml = substituteTemplateValues(htmlTemplate, deviceData, ipiData, targetIp, flowData); out.println(processedHtml); - } finally { - if (flowData != null) { - try { - flowData.close(); - } catch (Exception e) { - logger.warn("Error closing flow data", e); - } - } } } diff --git a/examples/ip-intelligence/web/getting-started.cloud/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebCloud.java b/examples/ip-intelligence/web/getting-started.cloud/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebCloud.java index dbcb50e7f..2e0fbb58f 100644 --- a/examples/ip-intelligence/web/getting-started.cloud/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebCloud.java +++ b/examples/ip-intelligence/web/getting-started.cloud/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebCloud.java @@ -47,8 +47,9 @@ * The `PipelineFilter` is used to intercept requests and perform IP Intelligence. The results * will be stored in the HttpServletRequest object. * - * The results of the lookup can be accessed by using a FlowDataProvider which - * is responsible for managing the lifecycle of the flowData - do NOT dispose + * The results of the lookup can be accessed by using a FlowDataProvider. The + * PipelineFilter closes the FlowData at the end of the request, so it is not + * closed by the example. * ```{java} * FlowData flowData = flowDataProvider.getFlowData(request); * IPIntelligenceData ipData = flowData.get(IPIntelligenceData.class); @@ -137,7 +138,8 @@ protected void processRequest(HttpServletRequest request, HttpServletResponse re String inputIpAddress = request.getParameter("client-ip"); // The lookup has already been carried out by the PipelineFilter - // which is responsible for the lifecycle of the flowData - do NOT dispose + // which closes the FlowData at the end of the request, so it is not + // closed here. FlowData flowData = flowDataProvider.getFlowData(request); // Determine target IP for display. @@ -165,14 +167,6 @@ protected void processRequest(HttpServletRequest request, HttpServletResponse re String htmlTemplate = loadTemplate(resourceBase + "/WEB-INF/html/index.html"); String processedHtml = substituteTemplateValues(htmlTemplate, ipiData, targetIp, flowData); out.println(processedHtml); - } finally { - if (flowData != null) { - try { - flowData.close(); - } catch (Exception e) { - logger.warn("Error closing flow data", e); - } - } } } diff --git a/examples/ip-intelligence/web/getting-started.mixed/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebMixed.java b/examples/ip-intelligence/web/getting-started.mixed/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebMixed.java index 9969e3262..b6addc8d7 100644 --- a/examples/ip-intelligence/web/getting-started.mixed/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebMixed.java +++ b/examples/ip-intelligence/web/getting-started.mixed/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebMixed.java @@ -87,7 +87,7 @@ public class GettingStartedWebMixed extends HttpServlet { private static final long serialVersionUID = 1734154705981153541L; public static Logger logger = LoggerFactory.getLogger(GettingStartedWebMixed.class); - public static final String DD_DATA_FILE_DEFAULT = "device-detection-data/51Degrees-LiteV4.1.hash"; + public static final String DD_DATA_FILE_DEFAULT = "device-detection/51Degrees-LiteV4.1.hash"; public static void main(String[] args) throws Exception { configureLogback(getFilePath("logback.xml")); @@ -140,7 +140,9 @@ protected void processRequest(HttpServletRequest request, HttpServletResponse re // Get the IP address parameter from the request for custom lookup String inputIpAddress = request.getParameter("client-ip"); - // The detection has already been carried out by the PipelineFilter + // The detection has already been carried out by the PipelineFilter, + // which closes the FlowData at the end of the request, so it is not + // closed here. FlowData flowData = flowDataProvider.getFlowData(request); // Determine target IP for display. @@ -171,14 +173,6 @@ protected void processRequest(HttpServletRequest request, HttpServletResponse re String htmlTemplate = loadTemplate(resourceBase + "/WEB-INF/html/index.html"); String processedHtml = substituteTemplateValues(htmlTemplate, deviceData, ipiData, targetIp, flowData); out.println(processedHtml); - } finally { - if (flowData != null) { - try { - flowData.close(); - } catch (Exception e) { - logger.warn("Error closing flow data", e); - } - } } } @@ -310,10 +304,13 @@ private static String resolveDataFile(String dataFile) throws IOException { if (Files.exists(relativePath)) { return relativePath.toAbsolutePath().toString(); } + // Search within the project directory, downloading the free Lite + // file if it is missing try { - return getFilePath(dataFile).getAbsolutePath(); + return fiftyone.ipintelligence.examples.shared.DataFileHelper + .findDataFile(dataFile).getAbsolutePath(); } catch (Exception e) { - throw new IOException("Data file not found: " + dataFile); + throw new IOException("Data file not found: " + dataFile, e); } } } diff --git a/examples/ip-intelligence/web/getting-started.mixed/src/main/webapp/WEB-INF/51Degrees-Mixed.xml b/examples/ip-intelligence/web/getting-started.mixed/src/main/webapp/WEB-INF/51Degrees-Mixed.xml index 95f859502..127b4665d 100644 --- a/examples/ip-intelligence/web/getting-started.mixed/src/main/webapp/WEB-INF/51Degrees-Mixed.xml +++ b/examples/ip-intelligence/web/getting-started.mixed/src/main/webapp/WEB-INF/51Degrees-Mixed.xml @@ -27,7 +27,7 @@ false false - ${TestDeviceDetectionDataFile:-cxx/device-detection-data/51Degrees-LiteV4.1.hash} + ${TestDeviceDetectionDataFile:-cxx/data/device-detection/51Degrees-LiteV4.1.hash} LowMemory DeviceDetectionHashEngine @@ -36,7 +36,7 @@ false false - ${TestDataFile:-cxx/ip-intelligence-data/51Degrees-EnterpriseIpiV41.ipi} + ${TestDataFile:-cxx/data/ip-intelligence/51Degrees-EnterpriseIpiV41.ipi} LowMemory IPIntelligenceOnPremiseEngine diff --git a/examples/ip-intelligence/web/getting-started.onprem/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebOnPrem.java b/examples/ip-intelligence/web/getting-started.onprem/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebOnPrem.java index d73bafdf8..ec48e2df9 100644 --- a/examples/ip-intelligence/web/getting-started.onprem/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebOnPrem.java +++ b/examples/ip-intelligence/web/getting-started.onprem/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebOnPrem.java @@ -45,8 +45,9 @@ * The `PipelineFilter` is used to intercept requests and perform IP Intelligence. The results * will be stored in the HttpServletRequest object. * - * The results of the lookup can be accessed by using a FlowDataProvider which - * is responsible for managing the lifecycle of the flowData - do NOT dispose + * The results of the lookup can be accessed by using a FlowDataProvider. The + * PipelineFilter closes the FlowData at the end of the request, so it is not + * closed by the example. * ```{java} * FlowData flowData = flowDataProvider.getFlowData(request); * IPIntelligenceData ipData = flowData.get(IPIntelligenceData.class); @@ -155,7 +156,8 @@ protected void processRequest(HttpServletRequest request, HttpServletResponse re String inputIpAddress = request.getParameter("client-ip"); // The lookup has already been carried out by the PipelineFilter - // which is responsible for the lifecycle of the flowData - do NOT dispose + // which closes the FlowData at the end of the request, so it is not + // closed here. FlowData flowData = flowDataProvider.getFlowData(request); // Determine target IP for display. @@ -183,14 +185,6 @@ protected void processRequest(HttpServletRequest request, HttpServletResponse re String htmlTemplate = loadTemplate(resourceBase + "/WEB-INF/html/index.html"); String processedHtml = substituteTemplateValues(htmlTemplate, ipiData, targetIp, flowData); out.println(processedHtml); - } finally { - if (flowData != null) { - try { - flowData.close(); - } catch (Exception e) { - logger.warn("Error closing flow data", e); - } - } } } diff --git a/examples/ip-intelligence/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml b/examples/ip-intelligence/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml index d1bc62de7..c8e8d515b 100644 --- a/examples/ip-intelligence/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml +++ b/examples/ip-intelligence/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml @@ -29,7 +29,7 @@ false - ${TestDataFile:-cxx/ip-intelligence-data/51Degrees-EnterpriseIpiV41.ipi} + ${TestDataFile:-cxx/data/ip-intelligence/51Degrees-EnterpriseIpiV41.ipi} LowMemory IPIntelligenceOnPremiseEngine diff --git a/examples/ip-intelligence/web/shared/src/main/java/fiftyone/ipintelligence/examples/web/HtmlContentHelper.java b/examples/ip-intelligence/web/shared/src/main/java/fiftyone/ipintelligence/examples/web/HtmlContentHelper.java index 2425e1be5..0d70be0f1 100644 --- a/examples/ip-intelligence/web/shared/src/main/java/fiftyone/ipintelligence/examples/web/HtmlContentHelper.java +++ b/examples/ip-intelligence/web/shared/src/main/java/fiftyone/ipintelligence/examples/web/HtmlContentHelper.java @@ -78,8 +78,8 @@ public static String getDataFileAgeWarning(FlowData flowData) { return "

\n" + " The IP intelligence data file is " + daysOld + " days old. " + "A more recent data file may be needed for the most accurate IP " + - "intelligence results. The latest Lite data file is available from the " + - "ip-intelligence-data " + + "intelligence results. The latest Lite data file is downloaded by the " + + "scripts in the data " + "repository on GitHub. Find out about the full Enterprise data file, " + "which includes automatic daily updates, on our " + "pricing page.\n" + diff --git a/ip-intelligence/README.md b/ip-intelligence/README.md index 2b087fe20..8f4534ed9 100644 --- a/ip-intelligence/README.md +++ b/ip-intelligence/README.md @@ -11,7 +11,7 @@ This repository contains Java implementation of the IP Intelligence [specificati ## Dependencies For runtime dependencies, see our [dependencies](https://51degrees.com/documentation/_info__dependencies.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=dependencies) page. -The [tested versions](https://51degrees.com/documentation/_info__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=dependencies) page shows +The [tested versions](https://51degrees.com/documentation/_product_summaries__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=dependencies) page shows the JDK versions that we currently test against. The software may run fine against other versions, but additional caution should be applied. @@ -23,9 +23,9 @@ The Java API can either use our cloud service to get its data or it can use a lo The cloud service for IP Intelligence is currently in development. -You will require [resource keys](https://51degrees.com/documentation/_info__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud-coming-soon) +You will require [resource keys](https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud-coming-soon) to use the Cloud API, as described on our website. Get resource keys from -our [configurator](https://configure.51degrees.com/?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud-coming-soon), see our [documentation](https://51degrees.com/documentation/_concepts__configurator.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud-coming-soon) on +our [configurator](https://configure.51degrees.com/?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud-coming-soon), see our [documentation](https://51degrees.com/documentation/_services__configurator.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud-coming-soon) on how to use this. #### On-Premise @@ -33,14 +33,13 @@ how to use this. In order to perform IP intelligence on-premise, you will need to use a data file. -[ip-intelligence-data](https://github.com/51Degrees/ip-intelligence-data/) repository instructs how to obtain a 'Lite' data file, otherwise [contact us](https://51degrees.com/contact-us?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=on-premise) to obtain an 'Enterprise' data file. +The free 'Lite' and Asn data files come from the [data](https://github.com/51Degrees/data) repository, which is the `cxx/data` folder of this repository. Run `cxx/data/get-data-files.sh ip-intelligence` from the root of this repository to download them, or `cxx\data\get-data-files.ps1 ip-intelligence` in PowerShell on Windows. Otherwise [contact us](https://51degrees.com/contact-us?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=on-premise) to obtain an 'Enterprise' data file. For tests, the data file can be supplied explicitly by setting the `51DEGREES_IPI_PATH` environment variable or system property to the path of the .ipi file. When it is not set, the project space is searched for the -expected file name, with the free 'Lite' file expected in the -[ip-intelligence-data](https://github.com/51Degrees/ip-intelligence-data/) -location. +expected file name, with the free 'Lite' file expected in +`cxx/data/ip-intelligence`. ## Installation @@ -109,7 +108,7 @@ repository. ## Tests -You will need [resource keys](https://51degrees.com/documentation/_info__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=tests) +You will need [resource keys](https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=tests) (see above) to complete the tests and run examples which include exercising the cloud API. The tests look for a resource key in the following order: diff --git a/ip-intelligence/ip-intelligence.cloud/pom.xml b/ip-intelligence/ip-intelligence.cloud/pom.xml index 9a1d434f4..cfc6ffa32 100644 --- a/ip-intelligence/ip-intelligence.cloud/pom.xml +++ b/ip-intelligence/ip-intelligence.cloud/pom.xml @@ -12,7 +12,7 @@ ip-intelligence.cloud 51Degrees :: IP Intelligence :: Cloud Provides IP Intelligence for the 51Degrees Pipeline API using the 51Degrees Cloud service. This can be used as an alternative to legacy IP Intelligence services such as MaxMind or IPinfo. - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=ip-intelligence.cloud-pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java diff --git a/ip-intelligence/ip-intelligence.engine.on-premise/pom.xml b/ip-intelligence/ip-intelligence.engine.on-premise/pom.xml index 00b37b646..0c48477de 100644 --- a/ip-intelligence/ip-intelligence.engine.on-premise/pom.xml +++ b/ip-intelligence/ip-intelligence.engine.on-premise/pom.xml @@ -12,7 +12,7 @@ ip-intelligence.engine.on-premise 51Degrees :: IP Intelligence :: On Premise Provides the On-premise IP Intelligence engine for the 51Degrees Pipeline API. This can be used as an alternative to legacy IP Intelligence services such as MaxMind or IPinfo. - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=ip-intelligence.engine.on-premise-pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java diff --git a/ip-intelligence/ip-intelligence.engine.on-premise/src/main/java/fiftyone/ipintelligence/engine/onpremise/data/ComponentMetaDataIPI.java b/ip-intelligence/ip-intelligence.engine.on-premise/src/main/java/fiftyone/ipintelligence/engine/onpremise/data/ComponentMetaDataIPI.java index 7ee5100d7..528e1d737 100644 --- a/ip-intelligence/ip-intelligence.engine.on-premise/src/main/java/fiftyone/ipintelligence/engine/onpremise/data/ComponentMetaDataIPI.java +++ b/ip-intelligence/ip-intelligence.engine.on-premise/src/main/java/fiftyone/ipintelligence/engine/onpremise/data/ComponentMetaDataIPI.java @@ -22,6 +22,7 @@ package fiftyone.ipintelligence.engine.onpremise.data; +import fiftyone.pipeline.engines.fiftyone.data.CloseableIterable; import fiftyone.ipintelligence.engine.onpremise.flowelements.IPIntelligenceOnPremiseEngine; import fiftyone.ipintelligence.engine.onpremise.interop.PropertyIterable; import fiftyone.ipintelligence.engine.onpremise.interop.swig.ComponentMetaDataSwig; @@ -77,7 +78,7 @@ public ProfileMetaData getDefaultProfile() { } @Override - public Iterable getProperties() { + public CloseableIterable getProperties() { return new PropertyIterable( engine, properties, diff --git a/ip-intelligence/ip-intelligence.engine.on-premise/src/main/java/fiftyone/ipintelligence/engine/onpremise/data/ProfileMetaDataIPI.java b/ip-intelligence/ip-intelligence.engine.on-premise/src/main/java/fiftyone/ipintelligence/engine/onpremise/data/ProfileMetaDataIPI.java index b9ee322df..3f2645723 100644 --- a/ip-intelligence/ip-intelligence.engine.on-premise/src/main/java/fiftyone/ipintelligence/engine/onpremise/data/ProfileMetaDataIPI.java +++ b/ip-intelligence/ip-intelligence.engine.on-premise/src/main/java/fiftyone/ipintelligence/engine/onpremise/data/ProfileMetaDataIPI.java @@ -114,11 +114,19 @@ public ComponentMetaData getComponent() { @Override public String getName() { List values = new ArrayList<>(); - for (ValueMetaData value : getValues()) { - if (value.getProperty().getDisplayOrder() > 0 && - value.getName().equalsIgnoreCase("N/A") == false) { - values.add(value); + // The native collection holds the data set until it is closed. The + // values taken from it are separate objects, so they outlive it. + try (CloseableIterable profileValues = getValues()) { + for (ValueMetaData value : profileValues) { + if (value.getProperty().getDisplayOrder() > 0 && + value.getName().equalsIgnoreCase("N/A") == false) { + values.add(value); + } } + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new IllegalStateException(e); } ValueMetaData[] valuesArray = new ValueMetaData[0]; valuesArray = values.toArray(valuesArray); diff --git a/ip-intelligence/ip-intelligence.engine.on-premise/src/main/java/fiftyone/ipintelligence/engine/onpremise/data/PropertyMetaDataIPI.java b/ip-intelligence/ip-intelligence.engine.on-premise/src/main/java/fiftyone/ipintelligence/engine/onpremise/data/PropertyMetaDataIPI.java index 527df5b74..eabb2aecb 100644 --- a/ip-intelligence/ip-intelligence.engine.on-premise/src/main/java/fiftyone/ipintelligence/engine/onpremise/data/PropertyMetaDataIPI.java +++ b/ip-intelligence/ip-intelligence.engine.on-premise/src/main/java/fiftyone/ipintelligence/engine/onpremise/data/PropertyMetaDataIPI.java @@ -22,6 +22,7 @@ package fiftyone.ipintelligence.engine.onpremise.data; +import fiftyone.pipeline.engines.fiftyone.data.CloseableIterable; import fiftyone.ipintelligence.engine.onpremise.flowelements.IPIntelligenceOnPremiseEngine; import fiftyone.ipintelligence.engine.onpremise.interop.Swig; import fiftyone.ipintelligence.engine.onpremise.interop.ValueIterable; @@ -136,7 +137,7 @@ public ComponentMetaData getComponent() { } @Override - public Iterable getValues() { + public CloseableIterable getValues() { return new ValueIterable( engine, engine.getMetaData().getValuesForProperty(source)); diff --git a/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/WrapperHash.java b/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/WrapperHash.java index d9e861c0f..a001f6e34 100644 --- a/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/WrapperHash.java +++ b/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/WrapperHash.java @@ -28,6 +28,7 @@ import fiftyone.pipeline.core.flowelements.Pipeline; import fiftyone.pipeline.core.flowelements.PipelineBuilder; import fiftyone.pipeline.engines.Constants; +import fiftyone.pipeline.engines.fiftyone.data.CloseableIterable; import fiftyone.pipeline.engines.fiftyone.data.ComponentMetaData; import fiftyone.pipeline.engines.fiftyone.data.FiftyOneAspectPropertyMetaData; import fiftyone.pipeline.engines.fiftyone.data.ProfileMetaData; @@ -69,15 +70,15 @@ public Iterable getProperties() { } @Override - public Iterable getValues() { + public CloseableIterable getValues() { return engine.getValues(); } - public Iterable getProfiles() { + public CloseableIterable getProfiles() { return engine.getProfiles(); } - public Iterable getComponents() { + public CloseableIterable getComponents() { return engine.getComponents(); } diff --git a/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/data/MetaDataHashTests.java b/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/data/MetaDataHashTests.java index d229af08b..f9f9a03eb 100644 --- a/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/data/MetaDataHashTests.java +++ b/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/data/MetaDataHashTests.java @@ -67,7 +67,7 @@ public void MetaData_Hash_Reload() throws ExecutionException, InterruptedExcepti * assumption rather than failing. */ @Test - public void MetaData_Performance_Properties() { + public void MetaData_Performance_Properties() throws Exception { MetaDataHasherHash hasher = new MetaDataHasherHash(); long start = System.currentTimeMillis(); hasher.hashProperties(0, getWrapper()); @@ -85,7 +85,7 @@ public void MetaData_Performance_Properties() { * assumption rather than failing. */ @Test - public void MetaData_Performance_Components() { + public void MetaData_Performance_Components() throws Exception { MetaDataHasherHash hasher = new MetaDataHasherHash(); long start = System.currentTimeMillis(); hasher.hashComponents(0, getWrapper()); @@ -103,7 +103,7 @@ public void MetaData_Performance_Components() { * assumption rather than failing. */ @Test - public void MetaData_Performance_Values() { + public void MetaData_Performance_Values() throws Exception { MetaDataHasherHash hasher = new MetaDataHasherHash(); long start = System.currentTimeMillis(); hasher.hashValues(0, getWrapper()); @@ -121,7 +121,7 @@ public void MetaData_Performance_Values() { * assumption rather than failing. */ @Test - public void MetaData_Performance_Profiles() { + public void MetaData_Performance_Profiles() throws Exception { MetaDataHasherHash hasher = new MetaDataHasherHash(); long start = System.currentTimeMillis(); hasher.hashProfiles(0, getWrapper()); diff --git a/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/data/MetaDataHasherHash.java b/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/data/MetaDataHasherHash.java index 4683157a5..2bd50f8b6 100644 --- a/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/data/MetaDataHasherHash.java +++ b/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/data/MetaDataHasherHash.java @@ -24,6 +24,7 @@ import fiftyone.ipintelligence.shared.testhelpers.Wrapper; import fiftyone.ipintelligence.shared.testhelpers.data.MetaDataHasher; +import fiftyone.pipeline.engines.fiftyone.data.CloseableIterable; import fiftyone.pipeline.engines.fiftyone.data.ComponentMetaData; import fiftyone.pipeline.engines.fiftyone.data.FiftyOneAspectPropertyMetaData; import fiftyone.pipeline.engines.fiftyone.data.ProfileMetaData; @@ -33,7 +34,7 @@ public class MetaDataHasherHash implements MetaDataHasher { @Override - public int hashProperties(int hash, Wrapper wrapper) { + public int hashProperties(int hash, Wrapper wrapper) throws Exception { int i = 0; Iterable properties = wrapper.getProperties(); for (FiftyOneAspectPropertyMetaData property : properties) { @@ -41,11 +42,13 @@ public int hashProperties(int hash, Wrapper wrapper) { hash ^= property.hashCode(); int j = 0; - for (ValueMetaData value : property.getValues()) { - if (j % 10 == 0) { - hash ^= value.hashCode(); + try (CloseableIterable propertyValues = property.getValues()) { + for (ValueMetaData value : propertyValues) { + if (j % 10 == 0) { + hash ^= value.hashCode(); + } + j++; } - j++; } if (property.getDefaultValue() != null) { hash ^= property.getDefaultValue().hashCode(); @@ -57,50 +60,60 @@ public int hashProperties(int hash, Wrapper wrapper) { } @Override - public int hashValues(int hash, Wrapper wrapper) { + public int hashValues(int hash, Wrapper wrapper) throws Exception { int i = 0; - for (ValueMetaData value : wrapper.getValues()) { - if (i % 100 == 0) { - hash ^= value.hashCode(); - hash ^= value.getProperty().hashCode(); + try (CloseableIterable values = wrapper.getValues()) { + for (ValueMetaData value : values) { + if (i % 100 == 0) { + hash ^= value.hashCode(); + hash ^= value.getProperty().hashCode(); + } + i++; } - i++; } return hash; } @Override - public int hashComponents(int hash, Wrapper wrapper) { - for (ComponentMetaData component : wrapper.getComponents()) { - hash ^= component.hashCode(); - int i = 0; - for (FiftyOneAspectPropertyMetaData property : component.getProperties()) { - if (i % 10 == 0) { - hash ^= property.hashCode(); + public int hashComponents(int hash, Wrapper wrapper) throws Exception { + try (CloseableIterable components = wrapper.getComponents()) { + for (ComponentMetaData component : components) { + hash ^= component.hashCode(); + int i = 0; + try (CloseableIterable componentProperties = component.getProperties()) { + for (FiftyOneAspectPropertyMetaData property : componentProperties) { + if (i % 10 == 0) { + hash ^= property.hashCode(); + } + i++; + } } - i++; + hash ^= component.getDefaultProfile().hashCode(); } - hash ^= component.getDefaultProfile().hashCode(); } return hash; } @Override - public int hashProfiles(int hash, Wrapper wrapper) { + public int hashProfiles(int hash, Wrapper wrapper) throws Exception { int i = 0; - for (ProfileMetaData profile : wrapper.getProfiles()) { - if (i % 100 == 0) { - hash ^= profile.hashCode(); - hash ^= profile.getComponent().hashCode(); - int j = 0; - for (ValueMetaData value : profile.getValues()) { - if (j % 10 == 0) { - hash ^= value.hashCode(); + try (CloseableIterable profiles = wrapper.getProfiles()) { + for (ProfileMetaData profile : profiles) { + if (i % 100 == 0) { + hash ^= profile.hashCode(); + hash ^= profile.getComponent().hashCode(); + int j = 0; + try (CloseableIterable profileValues = profile.getValues()) { + for (ValueMetaData value : profileValues) { + if (j % 10 == 0) { + hash ^= value.hashCode(); + } + j++; + } } - j++; } + i++; } - i++; } return hash; } diff --git a/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/flowelements/EngineTests.java b/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/flowelements/EngineTests.java index 4f0d9005b..e12f6cc10 100644 --- a/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/flowelements/EngineTests.java +++ b/ip-intelligence/ip-intelligence.engine.on-premise/src/test/java/fiftyone/ipintelligence/engine/onpremise/flowelements/EngineTests.java @@ -62,9 +62,10 @@ private void testDataSourceTier(String fileName) throws Exception { assumeTrue(e.getMessage(), false); } - IPIntelligenceOnPremiseEngine engine = createEngine(dataFile); - String tier = engine.getDataSourceTier(); - engine.close(); + String tier; + try (IPIntelligenceOnPremiseEngine engine = createEngine(dataFile)) { + tier = engine.getDataSourceTier(); + } if (fileName.equals(LITE_IPI_DATA_FILE_NAME)) { assertEquals("Lite", tier); diff --git a/ip-intelligence/ip-intelligence.shared/pom.xml b/ip-intelligence/ip-intelligence.shared/pom.xml index 71ab0f830..3f2271d0a 100644 --- a/ip-intelligence/ip-intelligence.shared/pom.xml +++ b/ip-intelligence/ip-intelligence.shared/pom.xml @@ -12,7 +12,7 @@ ip-intelligence.shared 51Degrees :: IP Intelligence :: Shared Shared code used by 51Degrees IP Intelligence modules for the Pipeline API. - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=ip-intelligence.shared-pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java diff --git a/ip-intelligence/ip-intelligence.shared/src/test/java/fiftyone/ipintelligence/shared/testhelpers/Wrapper.java b/ip-intelligence/ip-intelligence.shared/src/test/java/fiftyone/ipintelligence/shared/testhelpers/Wrapper.java index 39378f9fa..d74e56ff7 100644 --- a/ip-intelligence/ip-intelligence.shared/src/test/java/fiftyone/ipintelligence/shared/testhelpers/Wrapper.java +++ b/ip-intelligence/ip-intelligence.shared/src/test/java/fiftyone/ipintelligence/shared/testhelpers/Wrapper.java @@ -24,6 +24,7 @@ import fiftyone.pipeline.core.flowelements.Pipeline; import fiftyone.pipeline.engines.data.AspectData; +import fiftyone.pipeline.engines.fiftyone.data.CloseableIterable; import fiftyone.pipeline.engines.fiftyone.data.ComponentMetaData; import fiftyone.pipeline.engines.fiftyone.data.FiftyOneAspectPropertyMetaData; import fiftyone.pipeline.engines.fiftyone.data.ProfileMetaData; @@ -38,11 +39,11 @@ public interface Wrapper extends Closeable { Iterable getProperties(); - Iterable getValues(); + CloseableIterable getValues(); - Iterable getProfiles(); + CloseableIterable getProfiles(); - Iterable getComponents(); + CloseableIterable getComponents(); FiftyOneAspectEngine getEngine(); } \ No newline at end of file diff --git a/ip-intelligence/ip-intelligence.shared/src/test/java/fiftyone/ipintelligence/shared/testhelpers/data/MetaDataHasher.java b/ip-intelligence/ip-intelligence.shared/src/test/java/fiftyone/ipintelligence/shared/testhelpers/data/MetaDataHasher.java index 43ffbbfa7..da6e9b7bd 100644 --- a/ip-intelligence/ip-intelligence.shared/src/test/java/fiftyone/ipintelligence/shared/testhelpers/data/MetaDataHasher.java +++ b/ip-intelligence/ip-intelligence.shared/src/test/java/fiftyone/ipintelligence/shared/testhelpers/data/MetaDataHasher.java @@ -25,11 +25,11 @@ import fiftyone.ipintelligence.shared.testhelpers.Wrapper; public interface MetaDataHasher { - int hashProperties(int hash, Wrapper wrapper); + int hashProperties(int hash, Wrapper wrapper) throws Exception; - int hashValues(int hash, Wrapper wrapper); + int hashValues(int hash, Wrapper wrapper) throws Exception; - int hashComponents(int hash, Wrapper wrapper); + int hashComponents(int hash, Wrapper wrapper) throws Exception; - int hashProfiles(int hash, Wrapper wrapper); + int hashProfiles(int hash, Wrapper wrapper) throws Exception; } diff --git a/ip-intelligence/ip-intelligence.translation/pom.xml b/ip-intelligence/ip-intelligence.translation/pom.xml index 83cbc8865..98c629255 100644 --- a/ip-intelligence/ip-intelligence.translation/pom.xml +++ b/ip-intelligence/ip-intelligence.translation/pom.xml @@ -14,7 +14,7 @@ Translation engines that turn the weighted ISO country codes from 51Degrees IP Intelligence into localized, ordered country name and code lists. - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=ip-intelligence.translation-pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java @@ -60,9 +60,11 @@ + they are copied in at build time from the data repository in + cxx/data/ip-intelligence/Translations, as the .NET project + embeds the same files. A folder that is not there is skipped + without an error, so TranslationTests, which read the files + from the classpath, are what show they were copied. --> org.apache.maven.plugins maven-resources-plugin @@ -78,13 +80,13 @@ ${project.build.outputDirectory}/fiftyone/ipintelligence/translation - ${project.basedir}/../../cxx/ip-intelligence-data/Translations + ${project.basedir}/../../cxx/data/ip-intelligence/Translations countrycodes.en_GB.yml - ${project.basedir}/../../cxx/ip-intelligence-data/Translations/OSM + ${project.basedir}/../../cxx/data/ip-intelligence/Translations/OSM countries.*.yml diff --git a/ip-intelligence/ip-intelligence/pom.xml b/ip-intelligence/ip-intelligence/pom.xml index b22ba399b..2baf5e0fd 100644 --- a/ip-intelligence/ip-intelligence/pom.xml +++ b/ip-intelligence/ip-intelligence/pom.xml @@ -13,7 +13,7 @@ ip-intelligence 51Degrees :: IP Intelligence Provides IP Intelligence for the 51Degrees Pipeline API. This can be used as an alternative to legacy IP Intelligence services such as MaxMind or IPinfo. - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=ip-intelligence-pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java diff --git a/ip-intelligence/ip-intelligence/src/test/java/fiftyone/ipintelligence/CloudRequestOriginTests.java b/ip-intelligence/ip-intelligence/src/test/java/fiftyone/ipintelligence/CloudRequestOriginTests.java index c4d13d293..54b262730 100644 --- a/ip-intelligence/ip-intelligence/src/test/java/fiftyone/ipintelligence/CloudRequestOriginTests.java +++ b/ip-intelligence/ip-intelligence/src/test/java/fiftyone/ipintelligence/CloudRequestOriginTests.java @@ -88,12 +88,11 @@ public void ResourceKeyWithOrigin() throws Exception { when(loggerFactory.getLogger(anyString())).thenReturn(logger); try { - Pipeline pipeline = new IPIntelligencePipelineBuilder(loggerFactory, http) + try (Pipeline pipeline = new IPIntelligencePipelineBuilder(loggerFactory, http) .useCloud(resourceKey) .setCloudRequestOrigin(this.origin) .build(); - - try(FlowData data = pipeline.createFlowData()) { + FlowData data = pipeline.createFlowData()) { data.addEvidence("query.client-ip", "8.8.8.8") .process(); } diff --git a/ip-intelligence/ip-intelligence/src/test/java/fiftyone/ipintelligence/IPIntelligenceOnPremisePipelineBuilderTest.java b/ip-intelligence/ip-intelligence/src/test/java/fiftyone/ipintelligence/IPIntelligenceOnPremisePipelineBuilderTest.java index 98b386e97..d220eb776 100644 --- a/ip-intelligence/ip-intelligence/src/test/java/fiftyone/ipintelligence/IPIntelligenceOnPremisePipelineBuilderTest.java +++ b/ip-intelligence/ip-intelligence/src/test/java/fiftyone/ipintelligence/IPIntelligenceOnPremisePipelineBuilderTest.java @@ -88,12 +88,13 @@ private ConfigIpiSwig getConfigIpiSwig(Pipeline pipeline) throws NoSuchFieldExce @Test public void setFilename_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setFilename(IPI_DATA_FILE_NAME, true) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(IPI_DATA_FILE_NAME, dataFileConfiguration.getDataFilePath()); - assertTrue(dataFileConfiguration.getCreateTempDataCopy()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(IPI_DATA_FILE_NAME, dataFileConfiguration.getDataFilePath()); + assertTrue(dataFileConfiguration.getCreateTempDataCopy()); + } } @Test @@ -118,144 +119,159 @@ public void setFilename_OtherExtension_ExceptionThrown() { @Test public void setShareUsage_False_SharedFlowElementNotAdded() throws Exception { - Pipeline pipeline = new IPIntelligencePipelineBuilder(loggerFactory) + try (Pipeline pipeline = new IPIntelligencePipelineBuilder(loggerFactory) .useOnPremise(datafile.getAbsolutePath(), false) .setShareUsage(false) - .build(); - List flowElements = pipeline.getFlowElements(); - assertEquals(1, flowElements.size()); - assertFalse(flowElements.get(0) instanceof ShareUsageElement); + .build()) { + List flowElements = pipeline.getFlowElements(); + assertEquals(1, flowElements.size()); + assertFalse(flowElements.get(0) instanceof ShareUsageElement); + } } @Test public void setShareUsage_True_SharedFlowElementAdded() throws Exception { - Pipeline pipeline = new IPIntelligencePipelineBuilder(loggerFactory) + try (Pipeline pipeline = new IPIntelligencePipelineBuilder(loggerFactory) .useOnPremise(datafile.getAbsolutePath(), false) .setShareUsage(true) - .build(); - List flowElements = pipeline.getFlowElements(); - assertEquals(2, flowElements.size()); - assertTrue(flowElements.get(0) instanceof ShareUsageElement); + .build()) { + List flowElements = pipeline.getFlowElements(); + assertEquals(2, flowElements.size()); + assertTrue(flowElements.get(0) instanceof ShareUsageElement); + } } @Test public void setAutoUpdate_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setAutoUpdate(true) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertTrue(dataFileConfiguration.getAutomaticUpdatesEnabled()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertTrue(dataFileConfiguration.getAutomaticUpdatesEnabled()); + } } @Test public void setDataFileSystemWatcher_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setDataFileSystemWatcher(true) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertTrue(dataFileConfiguration.getFileSystemWatcherEnabled()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertTrue(dataFileConfiguration.getFileSystemWatcherEnabled()); + } } @Test public void setDataUpdateService_RegisterDataFileTriggered() throws Exception { DataUpdateServiceDefault dataUpdateServiceDefault = mock(DataUpdateServiceDefault.class); - builder + try (Pipeline ignored = builder .setAutoUpdate(true) .setDataUpdateService(dataUpdateServiceDefault) - .build(); - verify(dataUpdateServiceDefault).registerDataFile(any()); + .build()) { + verify(dataUpdateServiceDefault).registerDataFile(any()); + } } @Test public void setDataUpdateOnStartup_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setDataUpdateOnStartup(true) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertTrue(dataFileConfiguration.getUpdateOnStartup()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertTrue(dataFileConfiguration.getUpdateOnStartup()); + } } @Test public void setUpdatePollingInterval_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setUpdatePollingInterval(10) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(10, dataFileConfiguration.getPollingIntervalSeconds()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(10, dataFileConfiguration.getPollingIntervalSeconds()); + } } @Test public void setUpdatePollingIntervalMillis_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setUpdatePollingIntervalMillis(10000) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(10, dataFileConfiguration.getPollingIntervalSeconds()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(10, dataFileConfiguration.getPollingIntervalSeconds()); + } } @Test public void setUpdateRandomisationMax_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setUpdateRandomisationMax(123) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(123, dataFileConfiguration.getMaxRandomisationSeconds()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(123, dataFileConfiguration.getMaxRandomisationSeconds()); + } } @Test public void setUpdateRandomisationMaxMillis_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setUpdateRandomisationMaxMillis(123000) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(123, dataFileConfiguration.getMaxRandomisationSeconds()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(123, dataFileConfiguration.getMaxRandomisationSeconds()); + } } @Test public void setDataUpdateLicenseKey_PropertyPropagatedToDataFileConfiguration() throws Exception { String key = "licence_key"; - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setDataUpdateLicenseKey(key) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertTrue(dataFileConfiguration.getDataUpdateLicenseKeys().contains(key)); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertTrue(dataFileConfiguration.getDataUpdateLicenseKeys().contains(key)); + } } @Test public void setConcurrency_PropertyPropagatedSwigConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setConcurrency(4) - .build(); - ConfigIpiSwig swig = getConfigIpiSwig(pipeline); - assertEquals(4, swig.getConcurrency()); + .build()) { + ConfigIpiSwig swig = getConfigIpiSwig(pipeline); + assertEquals(4, swig.getConcurrency()); + } } @Test public void setDataUpdateUrl_PropertyPropagatedToDataFileConfiguration() throws Exception { String url = "https://update.me.com"; - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setDataUpdateUrl(url) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(url, dataFileConfiguration.getDataUpdateUrl()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(url, dataFileConfiguration.getDataUpdateUrl()); + } } @Test public void setDataUpdateVerifyMd5_PropertyPropagatedToDataFileConfiguration() throws Exception { - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setDataUpdateVerifyMd5(true) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertTrue(dataFileConfiguration.getVerifyMd5()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertTrue(dataFileConfiguration.getVerifyMd5()); + } } @Test public void setDataUpdateUrlFormatter_PropertyPropagatedToDataFileConfiguration() throws Exception { DataUpdateUrlFormatter urlFormatter = mock(DataUpdateUrlFormatter.class); - Pipeline pipeline = builder + try (Pipeline pipeline = builder .setDataUpdateUrlFormatter(urlFormatter) - .build(); - DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); - assertEquals(urlFormatter, dataFileConfiguration.getUrlFormatter()); + .build()) { + DataFileConfiguration dataFileConfiguration = getDataFileConfiguration(pipeline); + assertEquals(urlFormatter, dataFileConfiguration.getUrlFormatter()); + } } } \ No newline at end of file diff --git a/ip-intelligence/pom.xml b/ip-intelligence/pom.xml index 4327c14b2..1355b5932 100644 --- a/ip-intelligence/pom.xml +++ b/ip-intelligence/pom.xml @@ -1,4 +1,4 @@ - 4.0.0 @@ -9,7 +9,11 @@ pom 51Degrees :: Pipeline :: IP Intelligence - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java + + 51Degrees + https://51degrees.com/ip-intelligence?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pipeline.ip-intelligence&utm_term=organization + IP Intelligence engines for the 51Degrees Pipeline API @@ -175,6 +179,8 @@ ${maven.multiModuleProjectDirectory}/ci/pmd-ruleset.xml + + ${maven.multiModuleProjectDirectory}/ci/pmd-resource-ruleset.xml true diff --git a/location/README.md b/location/README.md index b48074eaa..031f81f12 100644 --- a/location/README.md +++ b/location/README.md @@ -10,16 +10,16 @@ This repository contains the geo-location engines for the Java implementation of ## Dependencies -The [tested versions](https://51degrees.com/documentation/_info__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=dependencies) page shows +The [tested versions](https://51degrees.com/documentation/_product_summaries__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=dependencies) page shows the JDK versions that we currently test against. The software may run fine against other versions, but additional caution should be applied. ### Data -You will require a [resource key](https://51degrees.com/documentation/_info__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=data) +You will require a [resource key](https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=data) to use the Cloud API. You can create resource keys using our [configurator](https://configure.51degrees.com/?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=data), see our -[documentation](https://51degrees.com/documentation/_concepts__configurator.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=data) on how to use this. +[documentation](https://51degrees.com/documentation/_services__configurator.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=data) on how to use this. ## Installation diff --git a/location/geo-location.shell.examples/src/main/java/fiftyone/geolocation/examples/CombiningServices.java b/location/geo-location.shell.examples/src/main/java/fiftyone/geolocation/examples/CombiningServices.java index 748499f05..c3088a99f 100644 --- a/location/geo-location.shell.examples/src/main/java/fiftyone/geolocation/examples/CombiningServices.java +++ b/location/geo-location.shell.examples/src/main/java/fiftyone/geolocation/examples/CombiningServices.java @@ -74,12 +74,14 @@ * .build(Enums.GeoLocationProvider.FiftyOneDegrees); * ``` * - * 2. Build a new Pipeline containing the engines which were just created. + * 2. Build a new Pipeline containing the engines which were just created, + * and ask it to close them when it is closed. * ``` * Pipeline pipeline = new PipelineBuilder(loggerFactory) * .addFlowElement(cloudRequestEngine) * .addFlowElement(deviceDetectionEngine) * .addFlowElement(locationEngine) + * .setAutoCloseElements(true) * .build(); * ``` * @@ -92,9 +94,10 @@ * .build(); * ``` * 3. Create a new FlowData instance ready to be populated with evidence for - * the Pipeline. + * the Pipeline, in a try-with-resources block so that it is closed when it + * is finished with. * ``` - * FlowData data = pipeline.createFlowData(); + * try (FlowData data = pipeline.createFlowData()) { * ``` * * 4. Process a longitude, latitude and User-Agent to retrieve the values @@ -148,14 +151,16 @@ public static void main(String[] args) throws Exception { new GeoLocationCloudEngineBuilder(loggerFactory) .build(Enums.GeoLocationProvider.FiftyOneDegrees); - Pipeline pipeline = + // The engines are not closed anywhere else, so the pipeline is + // asked to close them when it is closed. + try (Pipeline pipeline = new PipelineBuilder(loggerFactory) .addFlowElement(cloudRequestEngine) .addFlowElement(deviceDetectionEngine) .addFlowElement(locationEngine) + .setAutoCloseElements(true) .build(); - - try (FlowData flowData = pipeline.createFlowData()) { + FlowData flowData = pipeline.createFlowData()) { flowData .addEvidence(EVIDENCE_HEADER_USERAGENT_KEY, mobileUserAgent) .addEvidence(EVIDENCE_GEO_LAT_PARAM_KEY, "51.458048") @@ -173,7 +178,6 @@ public static void main(String[] args) throws Exception { System.out.println("Country: " + country.toString()); System.out.println("IsMobile: " + isMobile.toString()); } - pipeline.close(); } } diff --git a/location/geo-location.shell.examples/src/main/java/fiftyone/geolocation/examples/cloud/GettingStarted.java b/location/geo-location.shell.examples/src/main/java/fiftyone/geolocation/examples/cloud/GettingStarted.java index 553517c6a..8c86dfe01 100644 --- a/location/geo-location.shell.examples/src/main/java/fiftyone/geolocation/examples/cloud/GettingStarted.java +++ b/location/geo-location.shell.examples/src/main/java/fiftyone/geolocation/examples/cloud/GettingStarted.java @@ -57,9 +57,10 @@ * ``` * * 2. Create a new FlowData instance ready to be populated with evidence for - * the Pipeline. + * the Pipeline, in a try-with-resources block so that it is closed when it + * is finished with. * ``` - * FlowData data = pipeline.createFlowData(); + * try (FlowData data = pipeline.createFlowData()) { * ``` * * 3. Process a longitude and latitude to retrieve the values associated with @@ -92,14 +93,13 @@ public static void main(String[] args) throws Exception { "property as it is used by this example."); } else { - Pipeline pipeline = new GeoLocationPipelineBuilder(loggerFactory) + try (Pipeline pipeline = new GeoLocationPipelineBuilder(loggerFactory) .useCloud(resourceKey, Enums.GeoLocationProvider.FiftyOneDegrees) .setEndPoint("https://cloud.51degrees.com/api/v4") .useLazyLoading(1000) .setAutoCloseElements(true) .build(); - - try (FlowData flowData = pipeline.createFlowData()) { + FlowData flowData = pipeline.createFlowData()) { flowData.addEvidence(EVIDENCE_GEO_LAT_PARAM_KEY, "51.458048") .addEvidence(EVIDENCE_GEO_LON_PARAM_KEY, "-0.9822207999999999") .process(); @@ -115,8 +115,7 @@ public static void main(String[] args) throws Exception { System.out.println("Country: " + country.toString()); System.out.println(flowData.get(GeoData.class).getRegion()); - } - pipeline.close(); + } } } diff --git a/location/pom.xml b/location/pom.xml index e3e332f3f..930a853f5 100644 --- a/location/pom.xml +++ b/location/pom.xml @@ -20,7 +20,7 @@ ~ such notice(s) shall fulfill the requirements of that article. --> - 4.0.0 @@ -31,12 +31,12 @@ pom 51Degrees :: Pipeline :: Geo Location - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java Geo location engines for the 51Degrees Pipeline API 51Degrees - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pom.xml&utm_term=url-2 + https://51degrees.com/geolocation?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pipeline.geo-location&utm_term=organization @@ -78,6 +78,7 @@ 3.0.0 3.0.1 0.7.0 + 3.26.0 true false @@ -300,6 +301,40 @@ + + + org.apache.maven.plugins + maven-pmd-plugin + ${maven-pmd-plugin.version} + + + + ${maven.multiModuleProjectDirectory}/ci/pmd-resource-ruleset.xml + + true + true + true + false + + **/interop/swig/*.java + + + + + check-resources-closed + process-test-classes + + check + + + + diff --git a/pipeline/README.md b/pipeline/README.md index a860e40c0..8370cc09b 100644 --- a/pipeline/README.md +++ b/pipeline/README.md @@ -16,7 +16,7 @@ Reference documentation for the Java implementation can be found on the ## Pre-requisites -The [tested versions](https://51degrees.com/documentation/_info__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=pre-requisites) page shows +The [tested versions](https://51degrees.com/documentation/_product_summaries__tested_versions.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=pre-requisites) page shows the JDK versions that we currently test against. The software may run fine against other versions, but additional caution should be applied. @@ -123,7 +123,7 @@ There are several examples available in the `pipeline.developer-examples` folder how to make use of the Pipeline API in isolation. These are described in the table below. If you want examples that demonstrate how to use 51Degrees products such as device detection, then these are available in the corresponding [repository](https://github.com/51Degrees/51degrees-java/tree/main/device-detection) -and on our [website](https://51degrees.com/documentation/_examples__device_detection__index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=examples). +and on our [website](https://51degrees.com/documentation/_device_detection__examples__index.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=examples). | Example | Description | |----------------------------------------------------|-------------| diff --git a/pipeline/pipeline.caching/pom.xml b/pipeline/pipeline.caching/pom.xml index fba348a5b..2286dd55d 100644 --- a/pipeline/pipeline.caching/pom.xml +++ b/pipeline/pipeline.caching/pom.xml @@ -33,7 +33,7 @@ pipeline.caching 51Degrees :: Pipeline :: Caching - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pipeline.caching-pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java diff --git a/pipeline/pipeline.cloudrequestengine/pom.xml b/pipeline/pipeline.cloudrequestengine/pom.xml index 38d1e2115..2871e320d 100644 --- a/pipeline/pipeline.cloudrequestengine/pom.xml +++ b/pipeline/pipeline.cloudrequestengine/pom.xml @@ -34,7 +34,7 @@ pipeline.cloudrequestengine 51Degrees :: Pipeline :: Cloud Request Engine Cloud request engine for the 51Degrees Pipeline API - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pipeline.cloudrequestengine-pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java diff --git a/pipeline/pipeline.cloudrequestengine/src/main/java/fiftyone/pipeline/cloudrequestengine/Constants.java b/pipeline/pipeline.cloudrequestengine/src/main/java/fiftyone/pipeline/cloudrequestengine/Constants.java index 2e982f7e1..593c618d6 100644 --- a/pipeline/pipeline.cloudrequestengine/src/main/java/fiftyone/pipeline/cloudrequestengine/Constants.java +++ b/pipeline/pipeline.cloudrequestengine/src/main/java/fiftyone/pipeline/cloudrequestengine/Constants.java @@ -46,7 +46,7 @@ public class Messages { + "because your resource key does not include access to any " + "properties under '%s'. For more details on resource keys, " + "see our explainer: " - + "https://51degrees.com/documentation/_info__resource_keys.html?utm_source=code&utm_medium=comment&utm_campaign=51degrees-java&utm_content=pipeline.cloudrequestengine-src-main-java-fiftyone-pipeline-cloudrequestengine-constants.java&utm_term=exceptionfailedtoloadproperties"; + + "https://51degrees.com/documentation/_services__cloud__resource_keys.html?utm_source=code&utm_medium=comment&utm_campaign=51degrees-java&utm_content=pipeline.cloudrequestengine-src-main-java-fiftyone-pipeline-cloudrequestengine-constants.java&utm_term=exceptionfailedtoloadproperties"; public static final String ProcessCloudEngineNotImplemented = "This method should be overridden in the class derived from " diff --git a/pipeline/pipeline.cloudrequestengine/src/test/java/fiftyone/pipeline/cloudrequestengine/flowelements/CloudAspectEngineBaseTests.java b/pipeline/pipeline.cloudrequestengine/src/test/java/fiftyone/pipeline/cloudrequestengine/flowelements/CloudAspectEngineBaseTests.java index 54b2c8f96..514a9369e 100644 --- a/pipeline/pipeline.cloudrequestengine/src/test/java/fiftyone/pipeline/cloudrequestengine/flowelements/CloudAspectEngineBaseTests.java +++ b/pipeline/pipeline.cloudrequestengine/src/test/java/fiftyone/pipeline/cloudrequestengine/flowelements/CloudAspectEngineBaseTests.java @@ -181,9 +181,15 @@ public void init() { } @AfterEach - public void cleanUp() { - loggerFactory.assertMaxWarnings(maxWarnings); - loggerFactory.assertMaxErrors(maxErrors); + public void cleanUp() throws Exception { + try { + loggerFactory.assertMaxWarnings(maxWarnings); + loggerFactory.assertMaxErrors(maxErrors); + } finally { + if (pipeline != null) { + pipeline.close(); + } + } } private boolean propertiesContainName( @@ -370,9 +376,10 @@ public Object answer(InvocationOnMock invocationOnMock) throws Throwable { .addFlowElement(mockRequestEngine) .addFlowElement(mockTestInstance) .build(); - FlowData flowData = pipeline.createFlowData(); - flowData.addEvidence("query.user-agent", "iPhone"); - flowData.process(); + try (FlowData flowData = pipeline.createFlowData()) { + flowData.addEvidence("query.user-agent", "iPhone"); + flowData.process(); + } } catch (Exception e) { fail("An exception has occurred here."); @@ -422,9 +429,10 @@ public Object answer(InvocationOnMock invocationOnMock) throws Throwable { .addFlowElement(mockRequestEngine) .addFlowElement(mockTestInstance) .build(); - FlowData flowData = pipeline.createFlowData(); - flowData.addEvidence("query.user-agent", "iPhone"); - flowData.process(); + try (FlowData flowData = pipeline.createFlowData()) { + flowData.addEvidence("query.user-agent", "iPhone"); + flowData.process(); + } } catch (Exception e) { fail("An exception has occurred here."); @@ -480,9 +488,10 @@ public Object answer(InvocationOnMock invocationOnMock) throws Throwable { .addFlowElement(mockTestInstance) .addFlowElement(mockRequestEngine) .build(); - FlowData flowData = pipeline.createFlowData(); - flowData.addEvidence("query.user-agent", "iPhone"); - flowData.process(); + try (FlowData flowData = pipeline.createFlowData()) { + flowData.addEvidence("query.user-agent", "iPhone"); + flowData.process(); + } fail("Expected exception was not thrown"); } catch (Exception ex) { diff --git a/pipeline/pipeline.cloudrequestengine/src/test/java/fiftyone/pipeline/cloudrequestengine/flowelements/CloudRequestEngineTests.java b/pipeline/pipeline.cloudrequestengine/src/test/java/fiftyone/pipeline/cloudrequestengine/flowelements/CloudRequestEngineTests.java index c82645790..01b82a180 100644 --- a/pipeline/pipeline.cloudrequestengine/src/test/java/fiftyone/pipeline/cloudrequestengine/flowelements/CloudRequestEngineTests.java +++ b/pipeline/pipeline.cloudrequestengine/src/test/java/fiftyone/pipeline/cloudrequestengine/flowelements/CloudRequestEngineTests.java @@ -75,29 +75,30 @@ public void Process() throws Exception { .build(); - Pipeline pipeline = new PipelineBuilder(loggerFactory) + try (Pipeline pipeline = new PipelineBuilder(loggerFactory) .addFlowElement(engine).build(); - FlowData data = pipeline.createFlowData(); - data.addEvidence("query.User-Agent", userAgent); - data.process(); + FlowData data = pipeline.createFlowData()) { + data.addEvidence("query.User-Agent", userAgent); + data.process(); - String result = data.getFromElement(engine).getJsonResponse(); - assertEquals("{'device':{'value':'1'}}", result); + String result = data.getFromElement(engine).getJsonResponse(); + assertEquals("{'device':{'value':'1'}}", result); - JSONObject jsonObj = new JSONObject(result); - assertEquals(1, jsonObj.getJSONObject("device").getInt("value")); + JSONObject jsonObj = new JSONObject(result); + assertEquals(1, jsonObj.getJSONObject("device").getInt("value")); - verify(httpClient, times(1)) // we expected a single external POST request - .postData( - argThat(c -> { - return c.getURL().equals(expectedUrl); // to this uri - }), - ArgumentMatchers.anyMap(), - argThat(bytes -> { - String string = new String(bytes); - return string.contains("resource=" + resourceKey) && // content contains resource key - string.contains("User-Agent=" + userAgent); // content contains user agent - })); + verify(httpClient, times(1)) // we expected a single external POST request + .postData( + argThat(c -> { + return c.getURL().equals(expectedUrl); // to this uri + }), + ArgumentMatchers.anyMap(), + argThat(bytes -> { + String string = new String(bytes); + return string.contains("resource=" + resourceKey) && // content contains resource key + string.contains("User-Agent=" + userAgent); // content contains user agent + })); + } } /** @@ -113,19 +114,20 @@ public void Process_SuppressProcessExceptions_Exceptions_Added_To_Errors() throw .setResourceKey(resourceKey) .build(); - Pipeline pipeline = new PipelineBuilder(loggerFactory) + try (Pipeline pipeline = new PipelineBuilder(loggerFactory) .setSuppressProcessException(true) .addFlowElement(engine) .build(); - FlowData data = pipeline.createFlowData(); + FlowData data = pipeline.createFlowData()) { - data.addEvidence("query.User-Agent", userAgent); - data.process(); + data.addEvidence("query.User-Agent", userAgent); + data.process(); - assertFalse(data.getErrors().isEmpty()); - ArrayList errors = (ArrayList) data.getErrors(); - Throwable throwable = errors.get(0).getThrowable(); - assertInstanceOf(IOException.class, throwable); + assertFalse(data.getErrors().isEmpty()); + ArrayList errors = (ArrayList) data.getErrors(); + Throwable throwable = errors.get(0).getThrowable(); + assertInstanceOf(IOException.class, throwable); + } } /** @@ -330,11 +332,10 @@ public void evidencePrecidence(String name, boolean shouldWarn, String evidence1 .setResourceKey("resourcekey") .build(); - Pipeline pipeline = new PipelineBuilder(loggerFactory) + try (Pipeline pipeline = new PipelineBuilder(loggerFactory) .addFlowElement(engine) .build(); - - try (FlowData flowData = pipeline.createFlowData()) { + FlowData flowData = pipeline.createFlowData()) { flowData.addEvidence(evidence1Parts[0], evidence1Parts[1]); flowData.addEvidence(evidence2Parts[0], evidence2Parts[1]); @@ -450,11 +451,10 @@ public void getFormData(String name, Map evidence, String expect .setResourceKey("resourcekey") .build(); - Pipeline pipeline = new PipelineBuilder(loggerFactory) + try (Pipeline pipeline = new PipelineBuilder(loggerFactory) .addFlowElement(engine) .build(); - - try (FlowData data = pipeline.createFlowData()) { + FlowData data = pipeline.createFlowData()) { for (Map.Entry entry : evidence.entrySet()) { data.addEvidence(entry.getKey(), entry.getValue()); } @@ -479,11 +479,10 @@ public void getFormData_KeepsDotsInEvidenceName() throws Exception { .setResourceKey("resourcekey") .build(); - Pipeline pipeline = new PipelineBuilder(loggerFactory) + try (Pipeline pipeline = new PipelineBuilder(loggerFactory) .addFlowElement(engine) .build(); - - try (FlowData data = pipeline.createFlowData()) { + FlowData data = pipeline.createFlowData()) { data.addEvidence("query.id.usage", "non-marketing"); data.addEvidence("header.id.email", "someone@example.com"); diff --git a/pipeline/pipeline.common/pom.xml b/pipeline/pipeline.common/pom.xml index 68da0f135..cf2089abf 100644 --- a/pipeline/pipeline.common/pom.xml +++ b/pipeline/pipeline.common/pom.xml @@ -33,7 +33,7 @@ pipeline.common 51Degrees :: Pipeline :: Common - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pipeline.common-pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java diff --git a/pipeline/pipeline.core/pom.xml b/pipeline/pipeline.core/pom.xml index 3da59fe4c..3e30f3f64 100644 --- a/pipeline/pipeline.core/pom.xml +++ b/pipeline/pipeline.core/pom.xml @@ -38,7 +38,7 @@ 51Degrees :: Pipeline :: Core Core library for the 51Degrees Pipeline API - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pipeline.core-pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java diff --git a/pipeline/pipeline.core/src/main/java/fiftyone/pipeline/core/flowelements/PipelineBuilder.java b/pipeline/pipeline.core/src/main/java/fiftyone/pipeline/core/flowelements/PipelineBuilder.java index 20c846576..fbebda96e 100644 --- a/pipeline/pipeline.core/src/main/java/fiftyone/pipeline/core/flowelements/PipelineBuilder.java +++ b/pipeline/pipeline.core/src/main/java/fiftyone/pipeline/core/flowelements/PipelineBuilder.java @@ -122,9 +122,12 @@ public Pipeline buildFromConfiguration(PipelineOptions options) "The following builder parameters could not be processed: " + stringJoin(builderParams, ",")); } - } catch (PipelineConfigurationException ex) { - logger.debug("Problem with pipeline configuration, " + - "failed to create pipeline.", ex); + } catch (Exception | Error ex) { + if (ex instanceof PipelineConfigurationException) { + logger.debug("Problem with pipeline configuration, " + + "failed to create pipeline.", ex); + } + closeAfterFailure(flowElements, ex); throw ex; } // As the elements are all created within the builder, the user @@ -134,7 +137,32 @@ public Pipeline buildFromConfiguration(PipelineOptions options) // Build and return the pipeline using the list of flow elements // that have been created from the configuration options. - return build(); + try { + return build(); + } catch (Exception | Error ex) { + closeAfterFailure(flowElements, ex); + throw ex; + } + } + + /** + * Closes the elements built so far when building from configuration + * fails, as no pipeline exists to close them. A failure to close one is + * added to the original failure, which is the one reported. + * @param elements the elements built so far, emptied once closed + * @param failure the reason building failed + */ + private static void closeAfterFailure( + List elements, + Throwable failure) { + for (FlowElement element : elements) { + try { + element.close(); + } catch (Exception e) { + failure.addSuppressed(e); + } + } + elements.clear(); } /** @@ -377,17 +405,24 @@ private void addParallelElementsToList( // Iterate through the sub elements, creating them and // adding them to the list. int subCounter = 0; - for (ElementOptions subElement : elementOptions.subElements) { - if (subElement.subElements != null && - subElement.subElements.size() > 0) { - throw new PipelineConfigurationException( - "ElementOptions " + elementIndex + " contains nested " + - "sub elements. This is not supported."); - } else { - addElementToList(parallelElements, subElement, - "element " + subCounter + " in element " + elementIndex); + try { + for (ElementOptions subElement : elementOptions.subElements) { + if (subElement.subElements != null && + subElement.subElements.size() > 0) { + throw new PipelineConfigurationException( + "ElementOptions " + elementIndex + " contains nested " + + "sub elements. This is not supported."); + } else { + addElementToList(parallelElements, subElement, + "element " + subCounter + " in element " + elementIndex); + } + subCounter++; } - subCounter++; + } catch (RuntimeException | Error ex) { + // These elements are not in the builder's list yet, so they are + // closed here. + closeAfterFailure(parallelElements, ex); + throw ex; } // Now we've created all the elements, create the // ParallelElements instance and add it to the pipeline's diff --git a/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/FlowDataTests.java b/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/FlowDataTests.java index 3e04cf231..613033fa2 100644 --- a/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/FlowDataTests.java +++ b/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/FlowDataTests.java @@ -318,14 +318,15 @@ public void FlowData_GetNotPresent_Element() { } @Test(expected = IllegalStateException.class) - public void FlowData_PipelineDisposed() { + public void FlowData_PipelineDisposed() throws Exception { when(pipeline.isClosed()).thenReturn(true); - FlowData flowData = new FlowDataDefault( + try (FlowData flowData = new FlowDataDefault( mock(Logger.class), pipeline, - new EvidenceDefault(mock(Logger.class))); + new EvidenceDefault(mock(Logger.class)))) { - flowData.process(); + flowData.process(); + } } @Test diff --git a/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/ParallelElementsTests.java b/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/ParallelElementsTests.java index 8ddaa0435..617089b24 100644 --- a/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/ParallelElementsTests.java +++ b/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/ParallelElementsTests.java @@ -126,35 +126,36 @@ public TestElementData create(FlowData flowData) { Arrays.asList(element1, element2, element3)); - FlowData data = new FlowDataDefault( + try (FlowData data = new FlowDataDefault( mock(Logger.class), pipeline, - new EvidenceDefault(mock(Logger.class))); - data.process(); - - // Act - parallelElements.process(data); - - List startTimes = new ArrayList<>(); - startTimes.add((Date) data.get("element1").get("start")); - startTimes.add((Date) data.get("element2").get("start")); - startTimes.add((Date) data.get("element3").get("start")); - List endTimes = new ArrayList<>(); - endTimes.add((Date) data.get("element1").get("end")); - endTimes.add((Date) data.get("element2").get("end")); - endTimes.add((Date) data.get("element3").get("end")); - - // Assert - assertTrue("Expected no errors", - data.getErrors() == null || data.getErrors().size() == 0); - for (Date start : startTimes) { - for (Date end : endTimes) { - assertTrue( - "The elements were not processed sequentially. Times were: " + - startTimes.get(0) + " => " + endTimes.get(0) + "\n" + - startTimes.get(1) + " => " + endTimes.get(1) + "\n" + - startTimes.get(2) + " => " + endTimes.get(2), - start.before(end)); + new EvidenceDefault(mock(Logger.class)))) { + data.process(); + + // Act + parallelElements.process(data); + + List startTimes = new ArrayList<>(); + startTimes.add((Date) data.get("element1").get("start")); + startTimes.add((Date) data.get("element2").get("start")); + startTimes.add((Date) data.get("element3").get("start")); + List endTimes = new ArrayList<>(); + endTimes.add((Date) data.get("element1").get("end")); + endTimes.add((Date) data.get("element2").get("end")); + endTimes.add((Date) data.get("element3").get("end")); + + // Assert + assertTrue("Expected no errors", + data.getErrors() == null || data.getErrors().size() == 0); + for (Date start : startTimes) { + for (Date end : endTimes) { + assertTrue( + "The elements were not processed sequentially. Times were: " + + startTimes.get(0) + " => " + endTimes.get(0) + "\n" + + startTimes.get(1) + " => " + endTimes.get(1) + "\n" + + startTimes.get(2) + " => " + endTimes.get(2), + start.before(end)); + } } } } diff --git a/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/PipelineBuilderTests.java b/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/PipelineBuilderTests.java index c28bf4438..f8e1c968b 100644 --- a/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/PipelineBuilderTests.java +++ b/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/PipelineBuilderTests.java @@ -42,6 +42,7 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; import static org.mockito.ArgumentMatchers.anyString; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; @@ -233,6 +234,35 @@ public void PipelineBuilder_BuildFromConfiguration_MandatoryParameterWrongType() builder.buildFromConfiguration(opts); } + /** + * When an element fails to build, the elements built before it are + * closed, and the failure reported is the original one. + */ + @Test + public void PipelineBuilder_BuildFromConfiguration_SecondElementFails_FirstClosed() throws Exception { + RecordingElementBuilder.BUILT.clear(); + PipelineOptions opts = new PipelineOptions(); + ElementOptions first = new ElementOptions(); + first.builderName = "RecordingElementBuilder"; + opts.elements.add(first); + // The mandatory 'multiple' parameter is not set, so this one fails. + ElementOptions second = new ElementOptions(); + second.builderName = "MultiplyByElementBuilder"; + opts.elements.add(second); + + maxErrors = 1; + + try { + builder.buildFromConfiguration(opts); + fail("Expected the second element to fail to build"); + } catch (PipelineConfigurationException e) { + assertEquals(0, e.getSuppressed().length); + } + assertEquals(1, RecordingElementBuilder.BUILT.size()); + assertTrue("The first element should have been closed", + RecordingElementBuilder.BUILT.get(0).isClosed()); + } + @Test(expected = PipelineConfigurationException.class) public void PipelineBuilder_BuildFromConfiguration_NoBuilder() throws Exception { // Create the configuration object. @@ -416,26 +446,27 @@ public void PipelineBuilder_BuildFromConfiguration_ParallelElements() throws Exc options.elements.add(elOpts); // Pass the configuration to the builder to create the pipeline. - Pipeline pipeline = builder.buildFromConfiguration(options); - // Get the elements - ListSplitterElement splitterElement = pipeline.getElement(ListSplitterElement.class); - MultiplyByElement multiplyByElement = pipeline.getElement(MultiplyByElement.class); - - // Create, populate and process flow data. - try (FlowData flowData = pipeline.createFlowData()) { - flowData - .addEvidence(splitterElement.evidenceKeys.get(0), "1,2,abc") - .addEvidence(multiplyByElement.evidenceKeys.get(0), 25) - .process(); + try (Pipeline pipeline = builder.buildFromConfiguration(options)) { + // Get the elements + ListSplitterElement splitterElement = pipeline.getElement(ListSplitterElement.class); + MultiplyByElement multiplyByElement = pipeline.getElement(MultiplyByElement.class); + + // Create, populate and process flow data. + try (FlowData flowData = pipeline.createFlowData()) { + flowData + .addEvidence(splitterElement.evidenceKeys.get(0), "1,2,abc") + .addEvidence(multiplyByElement.evidenceKeys.get(0), 25) + .process(); - // Get the results and verify them. - ListSplitterElementData splitterData = flowData.getFromElement(splitterElement); - TestElementData multiplyByData = flowData.getFromElement(multiplyByElement); + // Get the results and verify them. + ListSplitterElementData splitterData = flowData.getFromElement(splitterElement); + TestElementData multiplyByData = flowData.getFromElement(multiplyByElement); - assertEquals("1", splitterData.getResult().get(0)); - assertEquals("2", splitterData.getResult().get(1)); - assertEquals("abc", splitterData.getResult().get(2)); - assertEquals(75, multiplyByData.getResult()); + assertEquals("1", splitterData.getResult().get(0)); + assertEquals("2", splitterData.getResult().get(1)); + assertEquals("abc", splitterData.getResult().get(2)); + assertEquals(75, multiplyByData.getResult()); + } } } @@ -443,58 +474,60 @@ private void VerifyListSplitterElementPipeline( PipelineOptions options, SplitOption splitOn) throws Exception { // Pass the configuration to the builder to create the pipeline. - Pipeline pipeline = builder.buildFromConfiguration(options); + try (Pipeline pipeline = builder.buildFromConfiguration(options)) { - ListSplitterElement element = pipeline.getElement(ListSplitterElement.class); - // Check we've got the expected number of evidence keys. - assertEquals(1, element.evidenceKeys.size()); + ListSplitterElement element = pipeline.getElement(ListSplitterElement.class); + // Check we've got the expected number of evidence keys. + assertEquals(1, element.evidenceKeys.size()); - // Create, populate and process flow data. - try (FlowData flowData = pipeline.createFlowData()) { - flowData.addEvidence(element.evidenceKeys.get(0), "123,456|789,0") - .process(); + // Create, populate and process flow data. + try (FlowData flowData = pipeline.createFlowData()) { + flowData.addEvidence(element.evidenceKeys.get(0), "123,456|789,0") + .process(); - // Get the result and verify it. - ListSplitterElementData elementData = flowData.getFromElement(element); - switch (splitOn) { - case Comma: - assertEquals("123", elementData.getResult().get(0)); - assertEquals("456|789", elementData.getResult().get(1)); - assertEquals("0", elementData.getResult().get(2)); - break; - case Pipe: - assertEquals("123,456", elementData.getResult().get(0)); - assertEquals("789,0", elementData.getResult().get(1)); - break; - case CommaMaxLengthThree: - assertEquals("123", elementData.getResult().get(0)); - assertEquals("456", elementData.getResult().get(1)); - assertEquals("|78", elementData.getResult().get(2)); - assertEquals("9", elementData.getResult().get(3)); - assertEquals("0", elementData.getResult().get(4)); - break; - default: - break; + // Get the result and verify it. + ListSplitterElementData elementData = flowData.getFromElement(element); + switch (splitOn) { + case Comma: + assertEquals("123", elementData.getResult().get(0)); + assertEquals("456|789", elementData.getResult().get(1)); + assertEquals("0", elementData.getResult().get(2)); + break; + case Pipe: + assertEquals("123,456", elementData.getResult().get(0)); + assertEquals("789,0", elementData.getResult().get(1)); + break; + case CommaMaxLengthThree: + assertEquals("123", elementData.getResult().get(0)); + assertEquals("456", elementData.getResult().get(1)); + assertEquals("|78", elementData.getResult().get(2)); + assertEquals("9", elementData.getResult().get(3)); + assertEquals("0", elementData.getResult().get(4)); + break; + default: + break; + } } } } private void VerifyMultiplyByElementPipeline(PipelineOptions options) throws Exception { // Pass the configuration to the builder to create the pipeline. - Pipeline pipeline = builder.buildFromConfiguration(options); - MultiplyByElement element = pipeline.getElement(MultiplyByElement.class); + try (Pipeline pipeline = builder.buildFromConfiguration(options)) { + MultiplyByElement element = pipeline.getElement(MultiplyByElement.class); - // Check we've got the expected number of evidence keys. - assertEquals(1, element.evidenceKeys.size()); + // Check we've got the expected number of evidence keys. + assertEquals(1, element.evidenceKeys.size()); - // Create, populate and process flow data. - try (FlowData flowData = pipeline.createFlowData()) { - flowData.addEvidence(element.evidenceKeys.get(0), 5) - .process(); + // Create, populate and process flow data. + try (FlowData flowData = pipeline.createFlowData()) { + flowData.addEvidence(element.evidenceKeys.get(0), 5) + .process(); - // Get the result and verify it. - TestElementData elementData = flowData.getFromElement(element); - assertEquals(40, elementData.getResult()); + // Get the result and verify it. + TestElementData elementData = flowData.getFromElement(element); + assertEquals(40, elementData.getResult()); + } } } diff --git a/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/PipelineTests.java b/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/PipelineTests.java index c688da1ee..e7d0f1d7b 100644 --- a/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/PipelineTests.java +++ b/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/flowelements/PipelineTests.java @@ -102,30 +102,31 @@ public TestElementData create(FlowData flowData) { new FlowElement[]{element1, element2}); // Don't create the flow data via the pipeline as we just want // to test Process. - FlowData data = new FlowDataDefault( + try (FlowData data = new FlowDataDefault( mock(Logger.class), pipeline, - new EvidenceDefault(mock(Logger.class))); - - // Act - data.process(); - - // Assert - assertTrue("Expected no errors", - data.getErrors() == null || data.getErrors().size() == 0); - // Check that the resulting data has the expected values - assertTrue("data from element 1 is missing in the result", - data.getDataKeys().contains("element1")); - assertTrue("data from element 2 is missing in the result", data.getDataKeys().contains("element2")); - assertEquals("done", data.get("element1").get("key").toString()); - assertEquals("done", data.get("element2").get("key").toString()); - // Check that element 1 was called before element 2. - verify(element2, times(1)).process(argThat(new ArgumentMatcher() { - @Override - public boolean matches(FlowData flowData) { - return flowData.getDataKeys().contains("element1"); - } - })); + new EvidenceDefault(mock(Logger.class)))) { + + // Act + data.process(); + + // Assert + assertTrue("Expected no errors", + data.getErrors() == null || data.getErrors().size() == 0); + // Check that the resulting data has the expected values + assertTrue("data from element 1 is missing in the result", + data.getDataKeys().contains("element1")); + assertTrue("data from element 2 is missing in the result", data.getDataKeys().contains("element2")); + assertEquals("done", data.get("element1").get("key").toString()); + assertEquals("done", data.get("element2").get("key").toString()); + // Check that element 1 was called before element 2. + verify(element2, times(1)).process(argThat(new ArgumentMatcher() { + @Override + public boolean matches(FlowData flowData) { + return flowData.getDataKeys().contains("element1"); + } + })); + } } @Test diff --git a/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/integration/PipelineIntegrationTests.java b/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/integration/PipelineIntegrationTests.java index c891b7181..1abdcd94a 100644 --- a/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/integration/PipelineIntegrationTests.java +++ b/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/integration/PipelineIntegrationTests.java @@ -54,11 +54,10 @@ public static void setup() { @Test public void PipelineIntegration_SingleElement() throws Exception { MultiplyByFiveElement fiveElement = new MultiplyByFiveElement(mock(Logger.class)); - Pipeline pipeline = new PipelineBuilder(loggerFactory) + try (Pipeline pipeline = new PipelineBuilder(loggerFactory) .addFlowElement(fiveElement) .build(); - - try (FlowData flowData = pipeline.createFlowData()) { + FlowData flowData = pipeline.createFlowData()) { flowData.addEvidence(fiveElement.evidenceKeys.get(0), 2); flowData.process(); @@ -71,12 +70,11 @@ public void PipelineIntegration_SingleElement() throws Exception { public void PipelineIntegration_MultipleElements() throws Exception { MultiplyByFiveElement fiveElement = new MultiplyByFiveElement(mock(Logger.class)); MultiplyByTenElement tenElement = new MultiplyByTenElement(mock(Logger.class)); - Pipeline pipeline = new PipelineBuilder(loggerFactory) + try (Pipeline pipeline = new PipelineBuilder(loggerFactory) .addFlowElement(fiveElement) .addFlowElement(tenElement) .build(); - - try (FlowData flowData = pipeline.createFlowData()) { + FlowData flowData = pipeline.createFlowData()) { flowData.addEvidence(fiveElement.evidenceKeys.get(0), 2); flowData.process(); @@ -91,11 +89,10 @@ public void PipelineIntegration_MultipleElements() throws Exception { public void PipelineIntegration_MultipleElementsParallel() throws Exception { MultiplyByFiveElement fiveElement = new MultiplyByFiveElement(mock(Logger.class)); MultiplyByTenElement tenElement = new MultiplyByTenElement(mock(Logger.class)); - Pipeline pipeline = new PipelineBuilder(loggerFactory) + try (Pipeline pipeline = new PipelineBuilder(loggerFactory) .addFlowElementsParallel(new FlowElement[]{fiveElement, tenElement}) .build(); - - try (FlowData flowData = pipeline.createFlowData()) { + FlowData flowData = pipeline.createFlowData()) { flowData.addEvidence(fiveElement.evidenceKeys.get(0), 2); flowData.process(); @@ -113,19 +110,20 @@ public void PipelineIntegration_StopFlag() throws Exception { FlowElement testElement = mock(FlowElement.class); when(testElement.getElementDataKey()).thenReturn("test"); when(testElement.getProperties()).thenReturn(Collections.emptyList()); - Pipeline pipeline = new PipelineBuilder(loggerFactory) + try (Pipeline pipeline = new PipelineBuilder(loggerFactory) .addFlowElement(stopElement) .addFlowElement(testElement) - .build(); + .build()) { - // Create and process flow data - try (FlowData flowData = pipeline.createFlowData()) { - flowData.process(); + // Create and process flow data + try (FlowData flowData = pipeline.createFlowData()) { + flowData.process(); - // Check that the stop flag is set - assertTrue(flowData.isStopped()); - // Check that the second element was never processed - verify(testElement, never()).process(any(FlowData.class)); + // Check that the stop flag is set + assertTrue(flowData.isStopped()); + // Check that the second element was never processed + verify(testElement, never()).process(any(FlowData.class)); + } } } } diff --git a/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/testclasses/flowelements/RecordingElementBuilder.java b/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/testclasses/flowelements/RecordingElementBuilder.java new file mode 100644 index 000000000..6ab3d4221 --- /dev/null +++ b/pipeline/pipeline.core/src/test/java/fiftyone/pipeline/core/testclasses/flowelements/RecordingElementBuilder.java @@ -0,0 +1,47 @@ +/* ********************************************************************* + * This Original Work is copyright of 51 Degrees Mobile Experts Limited. + * Copyright 2026 51 Degrees Mobile Experts Limited, Davidson House, + * Forbury Square, Reading, Berkshire, United Kingdom RG1 3EU. + * + * This Original Work is licensed under the European Union Public Licence + * (EUPL) v.1.2 and is subject to its terms as set out below. + * + * If a copy of the EUPL was not distributed with this file, You can obtain + * one at https://opensource.org/licenses/EUPL-1.2. + * + * The 'Compatible Licences' set out in the Appendix to the EUPL (as may be + * amended by the European Commission) shall be deemed incompatible for + * the purposes of the Work and the provisions of the compatibility + * clause in Article 5 of the EUPL shall not apply. + * + * If using the Work as, or as part of, a network application, by + * including the attribution notice(s) required under Article 5 of the EUPL + * in the end user terms of the application under an appropriate heading, + * such notice(s) shall fulfill the requirements of that article. + * ********************************************************************* */ + +package fiftyone.pipeline.core.testclasses.flowelements; + +import fiftyone.pipeline.annotations.ElementBuilder; +import fiftyone.pipeline.core.flowelements.FlowElement; + +import java.util.ArrayList; +import java.util.List; + +/** + * Builds list splitter elements and keeps each one, so that a test can check + * what happened to the elements a pipeline builder made. + */ +@ElementBuilder +public class RecordingElementBuilder { + + @SuppressWarnings("rawtypes") + public static final List BUILT = new ArrayList<>(); + + @SuppressWarnings("rawtypes") + public FlowElement build() { + FlowElement element = new ListSplitterElement(",", 100000); + BUILT.add(element); + return element; + } +} diff --git a/pipeline/pipeline.developer-examples/pipeline.developer-examples.cloud-engine/src/main/java/pipeline/developerexamples/cloudengine/Main.java b/pipeline/pipeline.developer-examples/pipeline.developer-examples.cloud-engine/src/main/java/pipeline/developerexamples/cloudengine/Main.java index 819bcee46..5b7c1b325 100644 --- a/pipeline/pipeline.developer-examples/pipeline.developer-examples.cloud-engine/src/main/java/pipeline/developerexamples/cloudengine/Main.java +++ b/pipeline/pipeline.developer-examples/pipeline.developer-examples.cloud-engine/src/main/java/pipeline/developerexamples/cloudengine/Main.java @@ -105,33 +105,36 @@ public void run() throws Exception { loggerFactory) .build(); - Pipeline pipeline = new PipelineBuilder(loggerFactory) + // The pipeline closes the engines when it is closed. + try (Pipeline pipeline = new PipelineBuilder(loggerFactory) .addFlowElement(cloudRequestEngine) .addFlowElement(ageElement) - .build(); + .setAutoCloseElements(true) + .build()) { - String dob = "18/12/1992"; + String dob = "18/12/1992"; - try (FlowData flowData = pipeline.createFlowData()) { - flowData - .addEvidence("cookie.date-of-birth", dob) - .process(); - - String starSign = - flowData.getFromElement(ageElement).getStarSign(); - if (starSign == null) { - // The service answered but offers no star sign - // product, which is what the 51Degrees cloud says, - // because star signs are only served by the example - // service. The connection, the property and evidence - // key negotiation and the request itself all worked. - System.out.println("The cloud service at " + endpoint() - + " does not offer the star sign product, so no " - + "star sign is available for a date of birth of " - + dob + "."); - } else { - System.out.println("With a date of birth of " + dob - + ", your star sign is " + starSign + "."); + try (FlowData flowData = pipeline.createFlowData()) { + flowData + .addEvidence("cookie.date-of-birth", dob) + .process(); + + String starSign = + flowData.getFromElement(ageElement).getStarSign(); + if (starSign == null) { + // The service answered but offers no star sign + // product, which is what the 51Degrees cloud says, + // because star signs are only served by the example + // service. The connection, the property and evidence + // key negotiation and the request itself all worked. + System.out.println("The cloud service at " + endpoint() + + " does not offer the star sign product, so no " + + "star sign is available for a date of birth of " + + dob + "."); + } else { + System.out.println("With a date of birth of " + dob + + ", your star sign is " + starSign + "."); + } } } //! [usage] diff --git a/pipeline/pipeline.developer-examples/pipeline.developer-examples.flowelement/src/main/java/pipeline/developerexamples/flowelement/Main.java b/pipeline/pipeline.developer-examples/pipeline.developer-examples.flowelement/src/main/java/pipeline/developerexamples/flowelement/Main.java index 8406ac0aa..76d7e4473 100644 --- a/pipeline/pipeline.developer-examples/pipeline.developer-examples.flowelement/src/main/java/pipeline/developerexamples/flowelement/Main.java +++ b/pipeline/pipeline.developer-examples/pipeline.developer-examples.flowelement/src/main/java/pipeline/developerexamples/flowelement/Main.java @@ -45,22 +45,25 @@ public void run() throws Exception { new SimpleFlowElementBuilder(loggerFactory) .build(); - Pipeline pipeline = new PipelineBuilder(loggerFactory) + // The pipeline closes the element when it is closed. + try (Pipeline pipeline = new PipelineBuilder(loggerFactory) .addFlowElement(starSignElement) - .build(); - Calendar dob = Calendar.getInstance(); - dob.set(1992, Calendar.DECEMBER, 18); + .setAutoCloseElements(true) + .build()) { + Calendar dob = Calendar.getInstance(); + dob.set(1992, Calendar.DECEMBER, 18); - try (FlowData flowData = pipeline.createFlowData()) { - flowData - .addEvidence("date-of-birth", dob.getTime()) - .process(); - - System.out.println("With a date of birth of " + - new SimpleDateFormat("yyyy/MM/dd").format(dob.getTime()) + - ", your star sign is " + - flowData.getFromElement(starSignElement).getStarSign() + - "."); + try (FlowData flowData = pipeline.createFlowData()) { + flowData + .addEvidence("date-of-birth", dob.getTime()) + .process(); + + System.out.println("With a date of birth of " + + new SimpleDateFormat("yyyy/MM/dd").format(dob.getTime()) + + ", your star sign is " + + flowData.getFromElement(starSignElement).getStarSign() + + "."); + } } //! [usage] } diff --git a/pipeline/pipeline.developer-examples/pipeline.developer-examples.onpremise-engine/src/main/java/pipeline/developerexamples/onpremiseengine/Main.java b/pipeline/pipeline.developer-examples/pipeline.developer-examples.onpremise-engine/src/main/java/pipeline/developerexamples/onpremiseengine/Main.java index 594cdbd5b..8d803d244 100644 --- a/pipeline/pipeline.developer-examples/pipeline.developer-examples.onpremise-engine/src/main/java/pipeline/developerexamples/onpremiseengine/Main.java +++ b/pipeline/pipeline.developer-examples/pipeline.developer-examples.onpremise-engine/src/main/java/pipeline/developerexamples/onpremiseengine/Main.java @@ -47,22 +47,26 @@ public void run() throws Exception { Main.class.getClassLoader().getResource("starsigns.csv").getPath(), false); - Pipeline pipeline = new PipelineBuilder(loggerFactory) + // The pipeline closes the engine, and with it the data file, + // when it is closed. + try (Pipeline pipeline = new PipelineBuilder(loggerFactory) .addFlowElement(starSignEngine) - .build(); - Calendar dob = Calendar.getInstance(); - dob.set(1992, Calendar.DECEMBER, 18); + .setAutoCloseElements(true) + .build()) { + Calendar dob = Calendar.getInstance(); + dob.set(1992, Calendar.DECEMBER, 18); - try (FlowData flowData = pipeline.createFlowData()) { - flowData - .addEvidence("date-of-birth", dob.getTime()) - .process(); - - System.out.println("With a date of birth of " + - new SimpleDateFormat("yyyy/MM/dd").format(dob.getTime()) + - ", your star sign is " + - flowData.getFromElement(starSignEngine).getStarSign() + - "."); + try (FlowData flowData = pipeline.createFlowData()) { + flowData + .addEvidence("date-of-birth", dob.getTime()) + .process(); + + System.out.println("With a date of birth of " + + new SimpleDateFormat("yyyy/MM/dd").format(dob.getTime()) + + ", your star sign is " + + flowData.getFromElement(starSignEngine).getStarSign() + + "."); + } } //! [usage] } diff --git a/pipeline/pipeline.developer-examples/pipeline.developer-examples.usage-sharing/src/main/java/pipeline/developerexamples/usagesharing/Main.java b/pipeline/pipeline.developer-examples/pipeline.developer-examples.usage-sharing/src/main/java/pipeline/developerexamples/usagesharing/Main.java index 407759b4d..f6da01434 100644 --- a/pipeline/pipeline.developer-examples/pipeline.developer-examples.usage-sharing/src/main/java/pipeline/developerexamples/usagesharing/Main.java +++ b/pipeline/pipeline.developer-examples/pipeline.developer-examples.usage-sharing/src/main/java/pipeline/developerexamples/usagesharing/Main.java @@ -88,26 +88,30 @@ public void run() throws Exception { new File(Objects.requireNonNull(getClass().getClassLoader(). getResource("51d.xml")).getFile())); - // Build a new Pipeline from the configuration. - Pipeline pipeline = new PipelineBuilder() + // Build a new Pipeline from the configuration. Closing the + // pipeline closes the ShareUsageElement it created, which sends + // any usage data still queued. + try (Pipeline pipeline = new PipelineBuilder() .addService(http) - .buildFromConfiguration(options); - - // Alternatively, the commented code below shows how to - // configure the ShareUsageElement in code, rather than - // using a configuration file. - //ShareUsageElement usageElement = new ShareUsageBuilder(loggerFactory, http) - // .setSharePercentage(0.01) - // .setMinimumEntriesPerMessage(2500) - // .build(); - //Pipeline pipeline = new PipelineBuilder() - // .addFlowElement(usageElement) - // .build(); + .buildFromConfiguration(options)) { - System.out.println( - "Pipeline created with share usage element. Evidence processed " + - "with this pipeline will now be periodically shared with " + - "51Degrees using the specified configuration."); + // Alternatively, the commented code below shows how to + // configure the ShareUsageElement in code, rather than + // using a configuration file. + //ShareUsageElement usageElement = new ShareUsageBuilder(loggerFactory, http) + // .setSharePercentage(0.01) + // .setMinimumEntriesPerMessage(2500) + // .build(); + //Pipeline pipeline = new PipelineBuilder() + // .addFlowElement(usageElement) + // .setAutoCloseElements(true) + // .build(); + + System.out.println( + "Pipeline created with share usage element. Evidence processed " + + "with this pipeline will now be periodically shared with " + + "51Degrees using the specified configuration."); + } } } diff --git a/pipeline/pipeline.did/pom.xml b/pipeline/pipeline.did/pom.xml index 21a2074c1..c5ba43ddc 100644 --- a/pipeline/pipeline.did/pom.xml +++ b/pipeline/pipeline.did/pom.xml @@ -39,7 +39,11 @@ value returned by the 51Degrees Cloud service. Parses the OWID envelope and exposes the Flags, License Id and match key plus the identifier type. Compare match keys, never envelopes. - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pipeline.did-pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java + + 51Degrees + https://51degrees.com/developers/51did?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pipeline.did&utm_term=organization + - 4.0.0 @@ -31,12 +31,12 @@ pom 51Degrees :: Pipeline - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pom.xml&utm_term=url + https://github.com/51Degrees/51degrees-java The 51Degrees Pipeline API provides architecture for consuming real-time digital data services. 51Degrees - https://51degrees.com?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pom.xml&utm_term=url-2 + https://51degrees.com/build-your-own?utm_source=maven&utm_medium=package&utm_campaign=51degrees-java&utm_content=pipeline&utm_term=organization @@ -95,6 +95,7 @@ 2.6 3.7 0.7.0 + 3.26.0 true false @@ -351,6 +352,40 @@ + + + org.apache.maven.plugins + maven-pmd-plugin + ${maven-pmd-plugin.version} + + + + ${maven.multiModuleProjectDirectory}/ci/pmd-resource-ruleset.xml + + true + true + true + false + + **/interop/swig/*.java + + + + + check-resources-closed + process-test-classes + + check + + + +