diff --git a/.gitmodules b/.gitmodules index f0d05919..af600379 100644 --- a/.gitmodules +++ b/.gitmodules @@ -4,9 +4,6 @@ [submodule "pipeline/owid-python"] path = pipeline/owid-python url = https://github.com/SWAN-community/owid-python.git -[submodule "ip-intelligence/ip-intelligence-data"] - path = ip-intelligence/ip-intelligence-data - url = ../ip-intelligence-data [submodule "cxx"] path = cxx url = ../51degrees-cxx diff --git a/AGENTS.md b/AGENTS.md index 8dececdf..11855b13 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,10 +39,20 @@ so a pipeline change is tested with every product in the same run. ## Submodules -The C/C++ engines and the data files are submodules, listed in the root -`.gitmodules` at their paths under each product folder. The IP Intelligence -on-premise package must compile its own copy of the common C library with -large data file support; the Device Detection one without it. +The C/C++ engines are the `cxx` submodule, listed in the root `.gitmodules`. +The data files come through cxx's own `data` submodule, at +`cxx/data/device-detection` and `cxx/data/ip-intelligence`, where each free +file is a `.link` that `cxx/data/get-data-files.ps1` or +`cxx/data/get-data-files.sh` downloads. Each product's `ci/fetch-assets.ps1` +gets the files its tests need, and the examples' `example_utils.py` runs the +script for a file that is missing. The IP Intelligence on-premise package must +compile its own copy of the common C library with large data file support; +the Device Detection one without it. + +A package's `ExtraPaths` in `ci/packages.json` cannot be a folder inside +`cxx`, because git reports any change there as a change to `cxx` itself. A +cxx update therefore releases no package, even when it brings new translations +in `cxx/data/ip-intelligence`. ## History diff --git a/README.md b/README.md index 3110be9c..2303b4fe 100644 --- a/README.md +++ b/README.md @@ -47,6 +47,29 @@ Clone with the submodules: git clone --recurse-submodules https://github.com/51Degrees/51degrees-python.git ``` +The data files the examples and tests use are not stored in a repository. The +[data](https://github.com/51Degrees/data) repository, which is the `cxx/data` +submodule, holds a `.link` file for each free one, where the file goes, and +two scripts that download them. From the root of this repository, run + +``` +sh cxx/data/get-data-files.sh +``` + +or, in PowerShell, + +``` +./cxx/data/get-data-files.ps1 +``` + +An example whose file is missing runs the script for that file itself. + +Git leaves the folder of a removed submodule on disk. A clone made before the +data files moved to `cxx/data` still has an `ip-intelligence/ip-intelligence-data` +folder, and keeps `cxx/device-detection-data` and `cxx/ip-intelligence-data` +once cxx removes them. Delete those folders, so that nothing finds an old copy +of a data file in them. + Each product's README says what else it needs, such as a C/C++ compiler for the on-premise engines and the data files. diff --git a/ci/packages.json b/ci/packages.json index 873df967..deb06ce9 100644 --- a/ci/packages.json +++ b/ci/packages.json @@ -140,7 +140,6 @@ "Name": "fiftyone_ipintelligence_onpremise", "Product": "ip-intelligence", "Path": "ip-intelligence/fiftyone_ipintelligence_onpremise", - "ExtraPaths": ["ip-intelligence/ip-intelligence-data"], "DependsOn": [ "fiftyone_ipintelligence_shared", "fiftyone_pipeline_engines_fiftyone" @@ -153,7 +152,6 @@ "Name": "fiftyone_ipintelligence_translation", "Product": "ip-intelligence", "Path": "ip-intelligence/fiftyone_ipintelligence_translation", - "ExtraPaths": ["ip-intelligence/ip-intelligence-data"], "DependsOn": [ "fiftyone_pipeline_core", "fiftyone_pipeline_translation" diff --git a/cxx b/cxx index 1bec7b57..e866f6d3 160000 --- a/cxx +++ b/cxx @@ -1 +1 @@ -Subproject commit 1bec7b570575e2e180b52435599bef9b03e35ad8 +Subproject commit e866f6d3cea936587790f87967eff7863778b30e diff --git a/device-detection/ci/fetch-assets.ps1 b/device-detection/ci/fetch-assets.ps1 index c551f8f9..45cb2875 100644 --- a/device-detection/ci/fetch-assets.ps1 +++ b/device-detection/ci/fetch-assets.ps1 @@ -11,7 +11,8 @@ if ($env:GITHUB_JOB -eq "PreBuild") { exit 0 } -$deviceDetectionData = "$PSScriptRoot/../../cxx/device-detection-data" +$data = "$PSScriptRoot/../../cxx/data" +$deviceDetectionData = "$data/device-detection" $assets = '51Degrees-LiteV4.1.hash', '20000 Evidence Records.yml', '20000 User Agents.csv' if ($DeviceDetection) { @@ -29,3 +30,12 @@ foreach ($asset in $assets) { default {New-Item -ItemType SymbolicLink -Force -Target "$PWD/assets/$_" -Path "$deviceDetectionData/$_"} } } + +# Downloads any other file the data repository links to in the folder. The +# script keeps a file that is already there, so the assets linked in above +# are not downloaded again. +$global:LASTEXITCODE = 0 +& "$data/get-data-files.ps1" device-detection +if ($LASTEXITCODE -ne 0) { + throw "Not every data file in $deviceDetectionData was downloaded" +} diff --git a/device-detection/fiftyone_devicedetection/examples/gettingstarted.py b/device-detection/fiftyone_devicedetection/examples/gettingstarted.py index 994d736f..cf3a7b7f 100644 --- a/device-detection/fiftyone_devicedetection/examples/gettingstarted.py +++ b/device-detection/fiftyone_devicedetection/examples/gettingstarted.py @@ -24,7 +24,7 @@ # First create the device detection pipeline with the desired settings. -data_file = "../../cxx/device-detection-data/51Degrees-LiteV4.1.hash" +data_file = "../../cxx/data/device-detection/51Degrees-LiteV4.1.hash" resource_key = "!!YOUR_RESOURCE_KEY!!" # Uncommment the line below 'On-Premise' and comment the line below 'Cloud' to diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/example_utils.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/example_utils.py index d36c35ea..05260f95 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/example_utils.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/example_utils.py @@ -23,6 +23,8 @@ from datetime import datetime, timedelta from threading import Event, Thread import os +import shlex +import subprocess from pathlib import Path @@ -51,6 +53,15 @@ class ExampleUtils: # file named in its configuration. DATA_FILE_ENV_VAR = "51DEGREES_DD_PATH" + # The data repository's device detection folder, from the root of a + # checkout of this repository. Each file the data repository links to + # belongs in this folder, beside its .link file. + DATA_FOLDER = os.path.join("cxx", "data", "device-detection") + + # Sent with each download, so the downloads these examples make can be + # counted. + DOWNLOAD_SOURCE = "examples-51degrees-python" + @staticmethod def get_resource_key_from_config(config): key = "" @@ -183,12 +194,29 @@ def __get_env_variable(name): else: return "" - # Find the specified filename within the specified directory. - # If no directory is specified, the working directory is used. - # If the file cannot be found, the algorithm will move to the - # parent directory and repeat the process. + # Find the specified file. A path to a file that exists is returned as + # it is. Next, the data repository's folder is checked in the working + # directory and in each directory above it, and a file the data + # repository links to is downloaded there when it is missing. Otherwise + # the working directory is searched for a file with the same name, then + # each parent directory in turn. None is returned when nothing is found. @staticmethod def find_file(file_name): + path = Path(file_name).absolute() + if path.is_file(): + return str(path.resolve()) + + # The data repository's copy is preferred to a file with the same + # name elsewhere in the tree, which may be out of date or a Git LFS + # pointer. + current = Path(os.getcwd()).absolute() + for root in [current, *current.parents]: + candidate = root / ExampleUtils.DATA_FOLDER / path.name + if candidate.is_file(): + return str(candidate) + if candidate.with_name(candidate.name + ".link").is_file(): + return ExampleUtils.__download(str(candidate)) + stop_event = Event() result = [None] # Start the file system search as a separate task. @@ -235,6 +263,39 @@ def __find_file(file_name, stop_event, result, search_dir=None): if search_dir.parent != search_dir and not stop_event.is_set(): ExampleUtils.__find_file(file_name, stop_event, result, search_dir.parent) + # Download a file the data repository links to, by running the data + # repository's own script for that file, and return its path, or None + # when it could not be downloaded. The data repository is the folder + # above the one that holds the file's .link. + @staticmethod + def __download(path): + name = os.path.basename(path) + data = os.path.dirname(os.path.dirname(path)) + if os.name == "nt": + command = [ + "powershell", "-NoProfile", "-ExecutionPolicy", "Bypass", + "-File", os.path.join(data, "get-data-files.ps1"), name, + "-Source", ExampleUtils.DOWNLOAD_SOURCE] + text = subprocess.list2cmdline(command) + else: + command = [ + "sh", os.path.join(data, "get-data-files.sh"), + "-s", ExampleUtils.DOWNLOAD_SOURCE, name] + text = shlex.join(command) + + # Flushed first, because the script writes to the same output. + print(f"{name} is missing, so it is being downloaded by running\n" + f"{text}", flush=True) + try: + subprocess.run(command, check=False) + except OSError: + pass + if os.path.isfile(path): + return path + print(f"{name} could not be downloaded. Run this command to try " + f"again.\n{text}") + return None + @staticmethod def get_data_file_date(engine): date = engine.engine.getPublishedTime() @@ -272,9 +333,10 @@ def check_data_file(pipeline, logger): f"that is more than '{ExampleUtils.DATA_FILE_AGE_WARNING}' days " + "old. A more recent data file may be needed to " + "correctly detect the latest devices, browsers, " + - "etc. The latest lite data file is available from " + - "the device-detection-data repository on GitHub " + - "https://github.com/51Degrees/device-detection-data. " + + "etc. Get the latest lite data file by running " + + "cxx/data/get-data-files.sh -u 51Degrees-LiteV4.1.hash, " + + "or cxx/data/get-data-files.ps1 51Degrees-LiteV4.1.hash " + + "-Update on Windows. " + "Find out about the Enterprise data file, which " + "includes automatic updates, on our pricing page: " + "https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_devicedetection_examples-src-fiftyone_devicedetection_examples-example_utils.py&utm_term=data-file-age-warning") diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/datafileupdate_console.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/datafileupdate_console.py index bf9ff7a3..d5fd2228 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/datafileupdate_console.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/datafileupdate_console.py @@ -53,9 +53,8 @@ # # ## Lite Data File # Lite data files (free-to-use, limited capabilities, no license key required) are created roughly -# once a month and cannot be updated using auto-update, they may be downloaded from -# [Github](https://github.com/51Degrees/device-detection-data) and are included with -# source distributions of this software. +# once a month and cannot be updated using auto-update, they may be downloaded with the +# scripts in the [data](https://github.com/51Degrees/data) repository on GitHub. # # # Update on Start-Up # You can configure the pipeline builder to download an Enterprise data file on start-up. diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_console.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_console.py index 9c695f43..f49c5bc0 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_console.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_console.py @@ -152,9 +152,9 @@ def main(argv): else: logger.log("error", "Failed to find a device detection " + - "data file. Make sure the device-detection-data " + - "submodule has been updated by running " + - "`git submodule update --recursive`.") + "data file. Make sure the data submodule is there, " + + "by running `git submodule update --init --recursive`, " + + "and run this example again to download the file.") if __name__ == "__main__": main(sys.argv[1:]) diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/app.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/app.py index c2ec0e2b..dee9b99b 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/app.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/app.py @@ -246,8 +246,9 @@ def build_config(): if foundDataFile == False: raise Exception("Failed to find a device detection data file matching " + f"'{dataFile}'. If using the lite file, then make sure the " + - "device-detection-data submodule has been updated by running " + - "`git submodule update --recursive`. Otherwise, ensure that the filename " + + "data submodule is there, by running " + + "`git submodule update --init --recursive`, and run this " + + "example again to download the file. Otherwise, ensure that the filename " + "is correct in config.json, or set the environment variable " + f"'{ExampleUtils.DATA_FILE_ENV_VAR}' to the path of the data file.") diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/templates/index.html b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/templates/index.html index 8106f4cc..d26ab8da 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/templates/index.html +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/gettingstarted_web/templates/index.html @@ -30,8 +30,8 @@
WARNING: This example is using a data file that is more than {{ utils.DATA_FILE_AGE_WARNING }} days old. A more recent data file may be needed to correctly detect the latest devices, - browsers, etc. The latest lite data file is available from the - device-detection-data + browsers, etc. Get the latest lite data file with the scripts in the + data repository on GitHub. Find out about the Enterprise data file, which includes automatic daily updates, on our pricing page.
diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/match_metrics.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/match_metrics.py index f9f254f8..5daeb284 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/match_metrics.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/match_metrics.py @@ -189,8 +189,9 @@ def main(argv): else: logger.log("error", "Failed to find a device detection data file. Make sure the " + - "device-detection-data submodule has been updated by running " + - "`git submodule update --recursive`.") + "data submodule is there, by running " + + "`git submodule update --init --recursive`, and run this " + + "example again to download the file.") if __name__ == "__main__": main(sys.argv[1:]) \ No newline at end of file diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/metadata_console.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/metadata_console.py index 1e5e0929..ed974ae6 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/metadata_console.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/metadata_console.py @@ -129,8 +129,9 @@ def main(argv): else: logger.log("error", "Failed to find a device detection data file. Make sure the " + - "device-detection-data submodule has been updated by running " + - "`git submodule update --recursive`.") + "data submodule is there, by running " + + "`git submodule update --init --recursive`, and run this " + + "example again to download the file.") if __name__ == "__main__": main(sys.argv[1:]) \ No newline at end of file diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/offlineprocessing.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/offlineprocessing.py index 2487f3e9..80ca344f 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/offlineprocessing.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/offlineprocessing.py @@ -211,8 +211,9 @@ def main(argv): else: logger.log("error", "Failed to find a device detection data file. Make sure the " + - "device-detection-data submodule has been updated by running " + - "`git submodule update --recursive`.") + "data submodule is there, by running " + + "`git submodule update --init --recursive`, and run this " + + "example again to download the file.") if __name__ == "__main__": main(sys.argv[1:]) diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/performance.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/performance.py index 665cdabf..271ce134 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/performance.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/performance.py @@ -145,7 +145,7 @@ def run(data_file, skip = False): if __name__ == "__main__": ap = argparse.ArgumentParser(description='Run detection benchmark.') ap.add_argument('-d', '--data_file', default='', help='Path to data file') - ap.add_argument('-u', '--user_agents_file', default='../../cxx/device-detection-data/20000 User Agents.csv', help='Path to user agents evidence file') + ap.add_argument('-u', '--user_agents_file', default='../../cxx/data/device-detection/20000 User Agents.csv', help='Path to user agents evidence file') ap.add_argument('-j', '--json_output', default='', help='Output results in JSON format') args = ap.parse_args() if args.data_file == "": diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/shareusage.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/shareusage.py index 5d808bf4..6cbbbfd7 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/shareusage.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/shareusage.py @@ -184,8 +184,9 @@ def main(argv): ShareUsageExample().run(input) else: print("ERROR: Failed to find the evidence file. Make sure the " - "device-detection-data submodule has been updated by running " - "`git submodule update --recursive`.") + "data submodule is there, by running " + "`git submodule update --init --recursive`, and run this " + "example again to download the file.") if __name__ == "__main__": diff --git a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/useragentclienthints_web/app.py b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/useragentclienthints_web/app.py index 6d10a1a7..570ab57e 100644 --- a/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/useragentclienthints_web/app.py +++ b/device-detection/fiftyone_devicedetection_examples/src/fiftyone_devicedetection_examples/onpremise/useragentclienthints_web/app.py @@ -148,8 +148,9 @@ def build_config(): if foundDataFile == False: raise Exception("Failed to find a device detection data file matching " + f"'{dataFile}'. If using the lite file, then make sure the " + - "device-detection-data submodule has been updated by running " + - "`git submodule update --recursive`. Otherwise, ensure that the filename " + + "data submodule is there, by running " + + "`git submodule update --init --recursive`, and run this " + + "example again to download the file. Otherwise, ensure that the filename " + "is correct in config.json.") return config diff --git a/device-detection/fiftyone_devicedetection_onpremise/tests/test_devicedetection.py b/device-detection/fiftyone_devicedetection_onpremise/tests/test_devicedetection.py index 3cd885ce..135cc69d 100644 --- a/device-detection/fiftyone_devicedetection_onpremise/tests/test_devicedetection.py +++ b/device-detection/fiftyone_devicedetection_onpremise/tests/test_devicedetection.py @@ -36,7 +36,7 @@ from fiftyone_pipeline_core.pipelinebuilder import PipelineBuilder from fiftyone_pipeline_engines.lru_cache import LRUEngineCache -data_file = "../../cxx/device-detection-data/51Degrees-LiteV4.1.hash" +data_file = "../../cxx/data/device-detection/51Degrees-LiteV4.1.hash" mobile_ua = "Mozilla/5.0 (iPhone; CPU iPhone OS 11_2 like Mac OS X) AppleWebKit/604.4.7 (KHTML, like Gecko) Mobile/15C114" @@ -46,7 +46,7 @@ # User agents file -with open('../../cxx/device-detection-data/20000 User Agents.csv', newline='') as file: +with open('../../cxx/data/device-detection/20000 User Agents.csv', newline='') as file: reader = csv.reader(file) user_agents = list(reader) no_user_agents = len(user_agents) diff --git a/device-detection/fiftyone_devicedetection_onpremise/tests/test_properties.py b/device-detection/fiftyone_devicedetection_onpremise/tests/test_properties.py index def20c1c..0d9b6435 100644 --- a/device-detection/fiftyone_devicedetection_onpremise/tests/test_properties.py +++ b/device-detection/fiftyone_devicedetection_onpremise/tests/test_properties.py @@ -28,7 +28,7 @@ from fiftyone_devicedetection_shared.utils import * from fiftyone_devicedetection_onpremise import constants -data_file = "../../cxx/device-detection-data/51Degrees-LiteV4.1.hash" +data_file = "../../cxx/data/device-detection/51Degrees-LiteV4.1.hash" mobile_ua = ("Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) " "AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile" "/11D167 Safari/9537.53") diff --git a/device-detection/fiftyone_devicedetection_onpremise/tests/test_swig.py b/device-detection/fiftyone_devicedetection_onpremise/tests/test_swig.py index 04b4ee50..0b4bea5a 100644 --- a/device-detection/fiftyone_devicedetection_onpremise/tests/test_swig.py +++ b/device-detection/fiftyone_devicedetection_onpremise/tests/test_swig.py @@ -28,7 +28,7 @@ from fiftyone_devicedetection_onpremise.swig_data import SwigData from fiftyone_devicedetection_shared.utils import * -data_file = "../../cxx/device-detection-data/51Degrees-LiteV4.1.hash" +data_file = "../../cxx/data/device-detection/51Degrees-LiteV4.1.hash" # Create a simple pipeline to access the engine with and process it with flow data deviceDetectionOnPremiseEngine = DeviceDetectionOnPremise( diff --git a/device-detection/readme.md b/device-detection/readme.md index 379b1ed1..379be178 100644 --- a/device-detection/readme.md +++ b/device-detection/readme.md @@ -43,22 +43,25 @@ to use the Cloud API. You can create resource keys using our #### On-Premise In order to perform device detection on-premise, you will need to use a 51Degrees data file. -This repository includes a free, 'lite' file in the 'device-detection-data' sub-module that has a -significantly reduced set of properties. To obtain a file with a more complete set of device -properties see the [51Degrees website](https://51degrees.com/pricing?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=on-premise). If you want to use the lite -file, you will need to install [GitLFS](https://git-lfs.github.com/): +A free, 'lite' file is available, with a significantly reduced set of properties. To obtain a +file with a more complete set of device properties see the [51Degrees website](https://51degrees.com/pricing?utm_source=github&utm_medium=readme&utm_campaign=51degrees-python&utm_content=readme.md&utm_term=on-premise). + +The lite file is not stored in a repository. The [data](https://github.com/51Degrees/data) +repository, which is the `cxx/data` submodule, links to it from +`cxx/data/device-detection`. Download it from the root of this repository by running: ``` -sudo apt-get install git-lfs -git lfs install +sh cxx/data/get-data-files.sh 51Degrees-LiteV4.1.hash ``` -Then, navigate to 'src/fiftyone_devicedetection_onpremise/cxx/device-detection-data' and execute: +or, in PowerShell: ``` -git lfs pull +./cxx/data/get-data-files.ps1 51Degrees-LiteV4.1.hash ``` +The examples run this for themselves when the file is missing. + ## Folders * `fiftyone_devicedetection` - references both cloud and on-premise packages, contains generic Device-Detection Pipeline Builder. diff --git a/ip-intelligence/ci/fetch-assets.ps1 b/ip-intelligence/ci/fetch-assets.ps1 index 00e2bd91..a26332bc 100644 --- a/ip-intelligence/ci/fetch-assets.ps1 +++ b/ip-intelligence/ci/fetch-assets.ps1 @@ -29,6 +29,14 @@ foreach ($asset in $assets) { } } +# The ASN file is not an asset, so the data repository's script downloads it +# beside its link in cxx/data/ip-intelligence, where the tests look for it. +$global:LASTEXITCODE = 0 +& "$PSScriptRoot/../../cxx/data/get-data-files.ps1" 51Degrees-IPIV4AsnIpiV41.ipi +if ($LASTEXITCODE -ne 0) { + throw "The ASN data file was not downloaded" +} + # If the Enterprise file was downloaded, prefer it for tests by exposing # it through the environment variable used by the test suites. if ($IpIntelligenceUrl) { diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/readme.md b/ip-intelligence/fiftyone_ipintelligence_examples/readme.md index af0db8c7..d9542f6a 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/readme.md +++ b/ip-intelligence/fiftyone_ipintelligence_examples/readme.md @@ -34,11 +34,12 @@ python -m fiftyone_ipintelligence_examples.cloud.gettingstarted_web ## On-premise examples Require a `.ipi` data file in the format the engine reads (see -[Data files](../readme.md#data-files)). The -`ip-intelligence-data` submodule includes a freely available network-only -file (`51Degrees-IPIV4AsnIpiV41.ipi`) that the examples fall back to; it -only contains the ASN/network properties, so location and registration -properties are reported as "not present in the data file". +[Data files](../readme.md#data-files)). The data repository, at +`cxx/data`, links to a freely available network-only file +(`51Degrees-IPIV4AsnIpiV41.ipi`) that the examples fall back to, and an +example downloads it when it is missing. It only contains the ASN/network +properties, so location and registration properties are reported as "not +present in the data file". For the full set of properties (location, registration, etc.), a more complete data file is required - to obtain one for testing please @@ -50,7 +51,7 @@ the first command line argument. - `onpremise/metadata_console.py` - list the properties available from the data file. - `onpremise/offlineprocessing.py` - batch process the evidence file - from the ip-intelligence-data repository. + from the data repository. - `onpremise/performance_console.py` - benchmark the number of IP lookups performed per second. - `onpremise/compare_console.py` - compare the location returned for each diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/example_utils.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/example_utils.py index 56e3d783..81eb5829 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/example_utils.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/example_utils.py @@ -22,6 +22,8 @@ from datetime import datetime, timedelta import os +import shlex +import subprocess from pathlib import Path @@ -43,6 +45,15 @@ class ExampleUtils: # to use when running examples. DATA_FILE_ENV_VAR = "IPINTELLIGENCEDATAFILE" + # The data repository's IP intelligence folder, from the root of a + # checkout of this repository. Each file the data repository links to + # belongs in this folder, beside its .link file. + DATA_FOLDER = os.path.join("cxx", "data", "ip-intelligence") + + # Sent with each download, so the downloads these examples make can be + # counted. + DOWNLOAD_SOURCE = "examples-51degrees-python" + @staticmethod def get_human_readable(ip_data, property): try: @@ -112,8 +123,9 @@ def find_file(file_name): """! Find the specified file. First checks the data file environment variable, then searches up the directory tree from the current - working directory looking in 'assets' and 'ip-intelligence-data' - folders. + working directory looking in 'assets' and the data repository's + folder. A file the data repository links to is downloaded there + when it is missing. """ env = os.environ.get(ExampleUtils.DATA_FILE_ENV_VAR) @@ -123,15 +135,54 @@ def find_file(file_name): current = Path(os.getcwd()).resolve() roots = [current, *current.parents] for root in roots[:8]: - for sub_dir in [ - ".", - "assets", - "ip-intelligence-data", - os.path.join("cxx", "ip-intelligence-data")]: + for sub_dir in [".", "assets", ExampleUtils.DATA_FOLDER]: candidate = os.path.join(str(root), sub_dir, file_name) if os.path.isfile(candidate): return candidate + for root in roots[:8]: + candidate = os.path.join( + str(root), ExampleUtils.DATA_FOLDER, file_name) + if os.path.isfile(candidate + ".link"): + return ExampleUtils.__download(candidate) + + return None + + @staticmethod + def __download(path): + + """! + Download a file the data repository links to, by running the data + repository's own script for that file, and return its path, or None + when it could not be downloaded. The data repository is the folder + above the one that holds the file's .link. + """ + + name = os.path.basename(path) + data = os.path.dirname(os.path.dirname(path)) + if os.name == "nt": + command = [ + "powershell", "-NoProfile", "-ExecutionPolicy", "Bypass", + "-File", os.path.join(data, "get-data-files.ps1"), name, + "-Source", ExampleUtils.DOWNLOAD_SOURCE] + text = subprocess.list2cmdline(command) + else: + command = [ + "sh", os.path.join(data, "get-data-files.sh"), + "-s", ExampleUtils.DOWNLOAD_SOURCE, name] + text = shlex.join(command) + + # Flushed first, because the script writes to the same output. + print(f"{name} is missing, so it is being downloaded by running\n" + f"{text}", flush=True) + try: + subprocess.run(command, check=False) + except OSError: + pass + if os.path.isfile(path): + return path + print(f"{name} could not be downloaded. Run this command to try " + f"again.\n{text}") return None @staticmethod @@ -167,9 +218,10 @@ def check_data_file(pipeline, logger): f"This data file is more than " f"{ExampleUtils.DATA_FILE_AGE_WARNING} days old. A more " f"recent data file may be needed to correctly resolve " - f"the latest IP addresses. The latest data file is " - f"available from the ip-intelligence-data repository on " - f"GitHub https://github.com/51Degrees/ip-intelligence-data.") + f"the latest IP addresses. Get the latest free data files " + f"by running cxx/data/get-data-files.sh -u ip-intelligence, " + f"or cxx/data/get-data-files.ps1 ip-intelligence -Update " + f"on Windows.") if product == "Lite": logger.log("warning", diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/compare_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/compare_console.py index d0479c2e..062e24da 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/compare_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/compare_console.py @@ -30,8 +30,8 @@ # an IP-to-location service compare to real world information, especially # when evaluating different solutions. # -# The input CSV (geoip_comparison_evidence.csv in the ip-intelligence-data -# repository) provides these fields: +# The input CSV (geoip_comparison_evidence.csv in the data repository's +# ip-intelligence folder) provides these fields: # DateTimeUtc, Latitude, Longitude, Ip, AddressFamily, Continent, Country # # For each record the engine is used to obtain the latitude and longitude @@ -56,7 +56,7 @@ from fiftyone_pipeline_core.logger import Logger from fiftyone_ipintelligence_shared.example_constants import ENTERPRISE_DATAFILE_NAME -# Name of the truth CSV file in the ip-intelligence-data repository. +# Name of the truth CSV file in the data repository. GEOIP_COMPARISON_EVIDENCE_FILE_NAME = "geoip_comparison_evidence.csv" # Mean radius of the earth in kilometres, used for the distance calculation. @@ -184,8 +184,8 @@ def main(argv): else: logger.log("error", "Failed to find a data file or truth file. Make sure the " - "ip-intelligence-data submodule has been updated by running " - "`git submodule update --recursive`. To obtain an enterprise " + "data submodule is there, by running " + "`git submodule update --init --recursive`. To obtain an enterprise " "data file for testing, please contact us: " "https://51degrees.com/contact-us?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-onpremise-compare_console.py&utm_term=data-file-required") diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/countries_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/countries_console.py index 35266657..6d26ce73 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/countries_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/countries_console.py @@ -150,8 +150,9 @@ def main(argv): else: logger.log("error", "Failed to find an IP intelligence data file. Make sure the " - "ip-intelligence-data submodule has been updated by running " - "`git submodule update --recursive`, or supply the path to a " + "data submodule is there, by running " + "`git submodule update --init --recursive`, and run this " + "example again to download the file, or supply the path to a " "data file as the first command line argument.") diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_console.py index 0f211919..f5ad8cdc 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_console.py @@ -32,10 +32,10 @@ # 2. How to pass input data (evidence) to the Pipeline # 3. How to retrieve the results # -# This example uses a local IP intelligence data file (.ipi). The -# ip-intelligence-data submodule provides a freely available data file; -# a more complete data file can be supplied as the first command line -# argument. +# This example uses a local IP intelligence data file (.ipi). The data +# repository links to a freely available data file, which the example +# downloads when it is missing. A more complete data file can be supplied +# as the first command line argument. # # Required PyPi Dependencies: # - fiftyone_ipintelligence @@ -170,8 +170,9 @@ def main(argv): else: logger.log("error", "Failed to find an IP intelligence data file. Make sure the " - "ip-intelligence-data submodule has been updated by running " - "`git submodule update --recursive`, or supply the path to a " + "data submodule is there, by running " + "`git submodule update --init --recursive`, and run this " + "example again to download the file, or supply the path to a " "data file as the first command line argument.") if __name__ == "__main__": diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_web/app.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_web/app.py index 49bc8b72..debc4a1b 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_web/app.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/gettingstarted_web/app.py @@ -133,9 +133,11 @@ def main(data_file = None): data_file = ExampleUtils.find_file(LITE_DATAFILE_NAME) if not data_file: raise Exception( - "Failed to find an IP intelligence data file. Download the " - "Lite file from https://51ddatafiles.blob.core.windows.net/" - "enterpriseipi/51Degrees-LiteIpiV41.ipi.gz") + "Failed to find an IP intelligence data file. Make sure the " + "data submodule is there, by running " + "`git submodule update --init --recursive`, and run this " + "example again to download the file, or supply the path to a " + "data file as the first command line argument.") build_pipeline(data_file) app.run(debug=False) diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/metadata_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/metadata_console.py index e10df504..ec2be8a1 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/metadata_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/metadata_console.py @@ -66,9 +66,8 @@ def main(argv): MetadataConsole().run(data_file, logger, print) else: logger.log("error", - "Failed to find an IP intelligence data file. Make sure the " - "ip-intelligence-data submodule has been updated by running " - "`git submodule update --recursive`. To obtain an enterprise " + "Failed to find an IP intelligence data file. Supply its path " + "as the first command line argument. To obtain an enterprise " "data file for testing, please contact us: " "https://51degrees.com/contact-us?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-onpremise-metadata_console.py&utm_term=data-file-required") diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/mixed_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/mixed_console.py index 11805592..07839ab1 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/mixed_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/mixed_console.py @@ -48,8 +48,8 @@ from fiftyone_ipintelligence_examples.example_utils import ExampleUtils from fiftyone_ipintelligence_shared.example_constants import ASN_DATAFILE_NAME -# Default name of the device detection data file (the lite file included in -# the device-detection-data repository). +# Default name of the device detection data file (the lite file the data +# repository links to). HASH_DATAFILE_NAME = "51Degrees-LiteV4.1.hash" # Combined evidence: a User-Agent for device detection and an IP address for @@ -199,14 +199,19 @@ def main(argv): else: if not device_file: logger.log("error", - "Failed to find a device detection data file. Make sure the " - "device-detection-data submodule has been updated, or supply " - "the path as the first command line argument.") + "Failed to find a device detection data file. Supply its " + "path as the first command line argument. Get the free Lite " + "file by running " + "cxx/data/get-data-files.sh 51Degrees-LiteV4.1.hash, or " + "cxx/data/get-data-files.ps1 51Degrees-LiteV4.1.hash on " + "Windows, which saves it in cxx/data/device-detection.") if not ip_file: logger.log("error", "Failed to find an IP intelligence data file. Make sure the " - "ip-intelligence-data submodule has been updated, or supply " - "the path as the second command line argument.") + "data submodule is there, by running " + "`git submodule update --init --recursive`, and run this " + "example again to download the file, or supply the path as " + "the second command line argument.") if __name__ == "__main__": main(sys.argv[1:]) diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/offlineprocessing.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/offlineprocessing.py index a32c11cb..9c7109cc 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/offlineprocessing.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/offlineprocessing.py @@ -23,8 +23,8 @@ ## @example onpremise/offlineprocessing.py # Provides an example of processing a YAML file containing evidence for # IP intelligence. There are 20,000 IP addresses in the supplied file of -# evidence records ('evidence.yml' in the ip-intelligence-data -# repository). +# evidence records ('evidence.yml' in the data repository's +# ip-intelligence folder). # # We create a pipeline using the on-premise engine, deserialize the # YAML evidence records, pass each entry to the pipeline and write the @@ -120,8 +120,8 @@ def main(argv): else: logger.log("error", "Failed to find a data file or evidence file. Make sure the " - "ip-intelligence-data submodule has been updated by running " - "`git submodule update --recursive`.") + "data submodule is there, by running " + "`git submodule update --init --recursive`.") if __name__ == "__main__": main(sys.argv[1:]) diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/performance_console.py b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/performance_console.py index 7ca71715..310bd727 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/performance_console.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/src/fiftyone_ipintelligence_examples/onpremise/performance_console.py @@ -25,7 +25,7 @@ # # This example measures the performance of the on-premise IP intelligence # engine. It processes the IP addresses from the evidence file supplied in -# the ip-intelligence-data repository and reports the number of IP lookups +# the data repository and reports the number of IP lookups # performed per second. # # Requesting fewer properties reduces processing time. If you do not specify @@ -225,8 +225,8 @@ def main(argv): else: logger.log("error", "Failed to find a data file or evidence file. Make sure the " - "ip-intelligence-data submodule has been updated by running " - "`git submodule update --recursive`. To obtain an enterprise " + "data submodule is there, by running " + "`git submodule update --init --recursive`. To obtain an enterprise " "data file for testing, please contact us: " "https://51degrees.com/contact-us?utm_source=code&utm_medium=example&utm_campaign=51degrees-python&utm_content=fiftyone_ipintelligence_examples-src-fiftyone_ipintelligence_examples-onpremise-performance_console.py&utm_term=data-file-required") diff --git a/ip-intelligence/fiftyone_ipintelligence_examples/tests/test_examples.py b/ip-intelligence/fiftyone_ipintelligence_examples/tests/test_examples.py index b0e359a3..a2d17c30 100644 --- a/ip-intelligence/fiftyone_ipintelligence_examples/tests/test_examples.py +++ b/ip-intelligence/fiftyone_ipintelligence_examples/tests/test_examples.py @@ -29,10 +29,10 @@ from fiftyone_ipintelligence_shared.example_constants import ASN_DATAFILE_NAME from fiftyone_ipintelligence_shared.example_constants import EVIDENCE_FILE_NAME -# The tests run against the freely available data file committed to the -# ip-intelligence-data submodule, so they only rely on properties that the -# free data file provides (location properties such as Latitude, Longitude -# and Areas are not present in it). +# The tests run against the freely available ASN data file the data +# repository links to, so they only rely on properties that the free data +# file provides (location properties such as Latitude, Longitude and Areas +# are not present in it). data_file = ExampleUtils.find_file(ASN_DATAFILE_NAME) evidence_file = ExampleUtils.find_file(EVIDENCE_FILE_NAME) truth_file = ExampleUtils.find_file("geoip_comparison_evidence.csv") @@ -61,8 +61,8 @@ class ExampleTests(unittest.TestCase): @unittest.skipUnless( bool(data_file), "No IP intelligence data file available") def test_onpremise_datafile_loads(self): - # Verify the freely available data file committed to the - # ip-intelligence-data submodule loads and processes correctly. + # Verify the freely available ASN data file loads and processes + # correctly. from fiftyone_ipintelligence.ipintelligence_pipelinebuilder import IpIntelligencePipelineBuilder pipeline = IpIntelligencePipelineBuilder( data_file_path=data_file, diff --git a/ip-intelligence/fiftyone_ipintelligence_onpremise/readme.md b/ip-intelligence/fiftyone_ipintelligence_onpremise/readme.md index b7eb9ac3..65b11d8e 100644 --- a/ip-intelligence/fiftyone_ipintelligence_onpremise/readme.md +++ b/ip-intelligence/fiftyone_ipintelligence_onpremise/readme.md @@ -28,12 +28,17 @@ python setup.py build_clib build_ext ## Data files -The on-premise engine requires a `.ipi` data file. A Lite data file can be -downloaded from -[51Degrees](https://51ddatafiles.blob.core.windows.net/enterpriseipi/51Degrees-LiteIpiV41.ipi.gz), -see also the [ip-intelligence-data](https://github.com/51Degrees/ip-intelligence-data) -repository which is included as a nested submodule and contains evidence -files for testing. +The on-premise engine requires a `.ipi` data file. The free Lite data file, +`51Degrees-LiteV41.ipi`, is linked from the +[data](https://github.com/51Degrees/data) repository, which is included as +the nested submodule `cxx/data` and contains evidence files for testing. From +the root of a checkout of +[51degrees-python](https://github.com/51Degrees/51degrees-python), download it +by running `sh cxx/data/get-data-files.sh 51Degrees-LiteV41.ipi`, or +`./cxx/data/get-data-files.ps1 51Degrees-LiteV41.ipi` in PowerShell. Without a +checkout, download it from +https://distributor.51degrees.com/api/v2/free/ipi-lite-v41, which sends it +gzip compressed. ## Usage @@ -41,7 +46,7 @@ files for testing. from fiftyone_ipintelligence_onpremise.ipintelligence_onpremise_pipelinebuilder import IpIntelligenceOnPremisePipelineBuilder pipeline = IpIntelligenceOnPremisePipelineBuilder( - data_file_path="51Degrees-LiteIpiV41.ipi", + data_file_path="51Degrees-LiteV41.ipi", licence_keys="", usage_sharing=False).build() diff --git a/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/constants.py b/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/constants.py index 6f9123f3..2c5180f8 100644 --- a/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/constants.py +++ b/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/constants.py @@ -42,11 +42,12 @@ def get_data_file_path(): os.path.dirname(__file__), "..", "..", "assets", "51Degrees-LiteIpiV41.ipi") - # The ASN data file committed to the repository's ip-intelligence-data - # submodule. Unlike the Lite and Enterprise files it needs no download - # or secret, so it is always present wherever the engine can be built. + # The free ASN data file, which ci/fetch-assets.ps1 downloads with the + # data repository's script, beside its link in cxx/data/ip-intelligence. + # Unlike the Enterprise file it needs no secret. @staticmethod - def get_committed_data_file_path(): + def get_asn_data_file_path(): return os.path.join( os.path.dirname(__file__), - "..", "..", "ip-intelligence-data", ASN_DATAFILE_NAME) + "..", "..", "..", "cxx", "data", "ip-intelligence", + ASN_DATAFILE_NAME) diff --git a/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/test_ipintelligence.py b/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/test_ipintelligence.py index 13f5d042..df9f835e 100644 --- a/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/test_ipintelligence.py +++ b/ip-intelligence/fiftyone_ipintelligence_onpremise/tests/test_ipintelligence.py @@ -171,42 +171,41 @@ def test_missing_licence_key(self): self.assertIn("licence key is required", str(context.exception)) -class CommittedDataFileTests(unittest.TestCase): +class AsnDataFileTests(unittest.TestCase): """! - The native engine accepts exactly one data file format version, so the - cxx (51degrees-cxx) pin and the ip-intelligence-data pin must agree. - They once did not: the engine moved to format 4.6 while the data pin - still held the 4.5 ASN file. These tests open the ASN file committed to - the data submodule, which needs no download or secret, so pins that - disagree fail here on any machine. - - They do not prove the engine reads the data files 51Degrees publishes. - That is IpIntelligenceOnPremiseTests above, against the Lite or - Enterprise file CI downloads; it skips when neither is present. + The native engine accepts exactly one data file format version. These + tests open the free ASN data file. The Distributor always serves the + latest one published, and it needs no secret, so an engine that cannot + read the format 51Degrees publishes fails here on any machine. + + IpIntelligenceOnPremiseTests above runs against the Lite or Enterprise + file CI downloads, and skips when neither is present. test_onpremise_datafile_loads in the examples package also opens the ASN file, but it skips silently when the file is not found and it runs after the examples build, so it is not relied on here. """ - def test_committed_data_file_present(self): + def test_asn_data_file_present(self): """! - The committed data file exists. If it does not, the - ip-intelligence-data submodule was not checked out, and the test + The ASN data file has been downloaded. If it has not, the test below would prove nothing. """ self.assertTrue( - os.path.isfile(TestConstants.get_committed_data_file_path()), - "Run 'git submodule update --init --recursive'.") + os.path.isfile(TestConstants.get_asn_data_file_path()), + "Download it from the repository root by running " + "'sh cxx/data/get-data-files.sh 51Degrees-IPIV4AsnIpiV41.ipi', " + "or './cxx/data/get-data-files.ps1 51Degrees-IPIV4AsnIpiV41.ipi' " + "in PowerShell.") - def test_committed_data_file_opens(self): + def test_asn_data_file_opens(self): """! - The engine opens the committed data file, so it reads that file's - format version, and returns its properties for an IPv4 address. + The engine opens the ASN data file, so it reads that file's format + version, and returns its properties for an IPv4 address. """ engine = IpIntelligenceOnPremise( - data_file_path=TestConstants.get_committed_data_file_path(), + data_file_path=TestConstants.get_asn_data_file_path(), licence_keys="") pipeline = PipelineBuilder().add(engine).build() diff --git a/ip-intelligence/fiftyone_ipintelligence_shared/src/fiftyone_ipintelligence_shared/example_constants.py b/ip-intelligence/fiftyone_ipintelligence_shared/src/fiftyone_ipintelligence_shared/example_constants.py index e11dd421..35945298 100644 --- a/ip-intelligence/fiftyone_ipintelligence_shared/src/fiftyone_ipintelligence_shared/example_constants.py +++ b/ip-intelligence/fiftyone_ipintelligence_shared/src/fiftyone_ipintelligence_shared/example_constants.py @@ -38,12 +38,13 @@ # Note: the .NET examples' Constants.cs still uses the legacy name # "51Degrees-EnterpriseIpiV41.ipi", but the published enterprise/ASN data # files use the "IPIV4" infix (cf. 51Degrees-IPIV4AsnIpiV41.ipi in the -# ip-intelligence-data repo), so the current name is used here. +# data repository), so the current name is used here. ENTERPRISE_DATAFILE_NAME = '51Degrees-IPIV4EnterpriseIpiV41.ipi' -LITE_DATAFILE_NAME = '51Degrees-LiteIpiV41.ipi' -# The ASN data file is the freely available data file committed -# to the ip-intelligence-data submodule. It only contains the network -# (ASN) properties, but is useful for testing the on-premise engine against -# the current data format without an enterprise data file. +# The free Lite data file, named as the data repository links to it. +LITE_DATAFILE_NAME = '51Degrees-LiteV41.ipi' +# The ASN data file is a freely available data file the data repository +# links to. It only contains the network (ASN) properties, but is useful for +# testing the on-premise engine against the current data format without an +# enterprise data file. ASN_DATAFILE_NAME = '51Degrees-IPIV4AsnIpiV41.ipi' EVIDENCE_FILE_NAME = 'evidence.yml' diff --git a/ip-intelligence/fiftyone_ipintelligence_translation/.gitignore b/ip-intelligence/fiftyone_ipintelligence_translation/.gitignore index 36ed5ddc..c39739ff 100644 --- a/ip-intelligence/fiftyone_ipintelligence_translation/.gitignore +++ b/ip-intelligence/fiftyone_ipintelligence_translation/.gitignore @@ -1,3 +1,3 @@ -# Translation YAML staged from the ip-intelligence-data submodule at build -# time (see setup.py stage_translations). Not committed; generated on build. +# Translation YAML staged from cxx/data/ip-intelligence at build time (see +# setup.py stage_translations). Not committed; generated on build. src/fiftyone_ipintelligence_translation/translations/ diff --git a/ip-intelligence/fiftyone_ipintelligence_translation/setup.py b/ip-intelligence/fiftyone_ipintelligence_translation/setup.py index a29e4097..3f06c4a4 100644 --- a/ip-intelligence/fiftyone_ipintelligence_translation/setup.py +++ b/ip-intelligence/fiftyone_ipintelligence_translation/setup.py @@ -40,14 +40,17 @@ def read(file_name): def stage_translations(): - """Copy the translation YAML files shipped in the ip-intelligence-data - submodule into the package so they are included in the built distribution. - This mirrors how the .NET project embeds the same files as resources. If the - submodule is not present the package falls back to reading it at run time, - so any failure here is ignored.""" + """Copy the translation YAML files from the data repository, the + cxx/data submodule, into the package so they are included in the built + distribution. This mirrors how the .NET project embeds the same files as + resources. A build from a source distribution has no cxx folder, but its + package already holds the files, so a missing folder is skipped. Any + failure here is ignored, and a package run from a source checkout reads + the folder at run time instead.""" here = os.path.dirname(os.path.abspath(__file__)) - source = os.path.join(here, "..", "ip-intelligence-data", "Translations") + source = os.path.join( + here, "..", "..", "cxx", "data", "ip-intelligence", "Translations") dest = os.path.join( here, "src", "fiftyone_ipintelligence_translation", "translations") if not os.path.isdir(source): diff --git a/ip-intelligence/fiftyone_ipintelligence_translation/src/fiftyone_ipintelligence_translation/resources.py b/ip-intelligence/fiftyone_ipintelligence_translation/src/fiftyone_ipintelligence_translation/resources.py index eb9aff55..4c0ffc4c 100644 --- a/ip-intelligence/fiftyone_ipintelligence_translation/src/fiftyone_ipintelligence_translation/resources.py +++ b/ip-intelligence/fiftyone_ipintelligence_translation/src/fiftyone_ipintelligence_translation/resources.py @@ -30,14 +30,15 @@ def _find_submodule_translations(): - """Walk up from this package looking for the ip-intelligence-data - submodule's Translations directory, used when the package is run from a - source checkout rather than an installed distribution.""" + """Walk up from this package looking for the data repository's + Translations directory, cxx/data/ip-intelligence/Translations, used when + the package is run from a source checkout rather than an installed + distribution.""" current = _PACKAGE_DIR for _ in range(8): candidate = os.path.join( - current, "ip-intelligence-data", "Translations") + current, "cxx", "data", "ip-intelligence", "Translations") if os.path.isdir(candidate): return candidate parent = os.path.dirname(current) @@ -50,9 +51,9 @@ def _find_submodule_translations(): def _translation_files(): """Return a dict of translation file name -> absolute path. The packaged - translations directory is preferred; if it is empty the ip-intelligence-data - submodule is used as a fallback (countrycodes at the root, countries under - OSM).""" + translations directory is preferred; if it is empty the data repository's + Translations directory is used as a fallback (countrycodes at the root, + countries under OSM).""" files = {} diff --git a/ip-intelligence/fiftyone_ipintelligence_translation/tests/test_translation.py b/ip-intelligence/fiftyone_ipintelligence_translation/tests/test_translation.py index 4decac97..199ca3b6 100644 --- a/ip-intelligence/fiftyone_ipintelligence_translation/tests/test_translation.py +++ b/ip-intelligence/fiftyone_ipintelligence_translation/tests/test_translation.py @@ -20,6 +20,7 @@ # such notice(s) shall fulfill the requirements of that article. # ********************************************************************* +import os import unittest import yaml @@ -34,6 +35,7 @@ from fiftyone_ipintelligence_translation.countries_translation_engine_builder import ( CountriesTranslationEngineBuilder) from fiftyone_ipintelligence_translation.comparer import make_comparer +from fiftyone_ipintelligence_translation import resources from fiftyone_ipintelligence_translation.resources import ( get_all_countries, get_country_resources) @@ -448,5 +450,18 @@ def test_population_translated_names(self): self.assertNotEqual(geo_codes[0], pop_codes[0]) +class PackagedTranslationsTests(unittest.TestCase): + + def test_translations_are_packaged(self): + # The tests run against the built package. A build that cannot find + # the data repository's Translations folder still succeeds, without + # the YAML files, and the other tests can still pass by reading the + # folder from the checkout, so this checks the package itself. + folder = resources._PACKAGED_TRANSLATIONS + names = os.listdir(folder) if os.path.isdir(folder) else [] + self.assertIn("countrycodes.en_GB.yml", names) + self.assertIn("countries.de_DE.yml", names) + + if __name__ == "__main__": unittest.main() diff --git a/ip-intelligence/ip-intelligence-data b/ip-intelligence/ip-intelligence-data deleted file mode 160000 index b8b3d7d4..00000000 --- a/ip-intelligence/ip-intelligence-data +++ /dev/null @@ -1 +0,0 @@ -Subproject commit b8b3d7d4e7747a0412281b9b48e0144b2ef13fcc diff --git a/ip-intelligence/readme.md b/ip-intelligence/readme.md index 68e79e2e..18d82f7e 100644 --- a/ip-intelligence/readme.md +++ b/ip-intelligence/readme.md @@ -28,10 +28,9 @@ page shows the Python versions that we currently test against. The on-premise engine wraps the native [51degrees-cxx](https://github.com/51Degrees/51degrees-cxx/blob/main/docs/ipi/README.md) -library, and the -[ip-intelligence-data](https://github.com/51Degrees/ip-intelligence-data) -repository (data and evidence files used for testing) is also included, both -as git submodules. When cloning, use: +library, which is included as the `cxx` git submodule. It has the +[data](https://github.com/51Degrees/data) repository (data and evidence files +used for testing) as its own submodule, at `cxx/data`. When cloning, use: ``` git clone --recurse-submodules https://github.com/51Degrees/51degrees-python.git @@ -45,13 +44,25 @@ git submodule update --init --recursive ### Data files -The on-premise engine requires a `.ipi` data file. The -[ip-intelligence-data](https://github.com/51Degrees/ip-intelligence-data) -submodule provides a freely available network-only data file +The on-premise engine requires a `.ipi` data file. The data repository's +`cxx/data/ip-intelligence` folder holds the evidence files used for testing, +and links to a freely available network-only data file (`51Degrees-IPIV4AsnIpiV41.ipi`), which the examples tests and -`CommittedDataFileTests` use, along with the evidence files used for -testing. The on-premise unit tests default to -`assets/51Degrees-LiteIpiV41.ipi`. +`AsnDataFileTests` use. The linked files are not stored in a repository. +Download them from the root of this repository by running: + +``` +sh cxx/data/get-data-files.sh ip-intelligence +``` + +or, in PowerShell: + +``` +./cxx/data/get-data-files.ps1 ip-intelligence +``` + +An example whose file is missing runs the script for that file itself. The +on-premise unit tests default to `assets/51Degrees-LiteIpiV41.ipi`. For the full set of properties (location, registration, etc.) an enterprise data file is required. To obtain one for testing, please @@ -59,12 +70,12 @@ data file is required. To obtain one for testing, please line argument to the examples, or place it within the project space. Note: the engine reads exactly one data file format version, set by -`FIFTYONE_DEGREES_IPI_TARGET_VERSION_MAJOR` and `_MINOR` in the -`ip-intelligence-cxx` submodule's `src/ipi.c`, and refuses any other as -"an unsupported version". Move the `ip-intelligence-cxx` and -`ip-intelligence-data` pins together. `CommittedDataFileTests` in +`FIFTYONE_DEGREES_IPI_TARGET_VERSION_MAJOR` and `_MINOR` in the `cxx` +submodule's `src/ipi/ipi.c`, and refuses any other as "an unsupported +version". The ASN data file is downloaded from the Distributor, which always +serves the latest one published, so `AsnDataFileTests` in [test_ipintelligence.py](fiftyone_ipintelligence_onpremise/tests/test_ipintelligence.py) -fails when the two disagree. +fails when the engine cannot read the format 51Degrees publishes. ## Folders @@ -77,8 +88,6 @@ fails when the two disagree. wraps ip-intelligence-cxx via SWIG. - `fiftyone_ipintelligence_shared` - code shared between the engines. - `fiftyone_ipintelligence_examples` - usage examples for both engines. -- `ip-intelligence-data` - submodule containing the test data file and - evidence files used by the examples and tests. ## Installation