From e12ce2cd49e73b3afb00a3403378031225b2b47f Mon Sep 17 00:00:00 2001 From: Caglar Pir Date: Tue, 27 Oct 2026 14:02:27 +0100 Subject: [PATCH 1/4] Update the README for the release Link to the command reference, and keep the README short: the long paragraphs on the process report and the upload history now point to it. Fix what was wrong: - upload has no --file_types: a folder of zip files is uploaded with the description file of the zipped images. - %g.gpx matches foo.GPX on macOS too, whose file system is case-insensitive by default. - With both options, the --geotag_source sources also geotag the videos that the --video_geotag_source sources leave. - native, camm, gopro and blackvue take no pattern. - The cutoff example set --offset_angle, and ended with a backslash. Add the Novatek dashcams that are read natively, ExifTool, GPX and NMEA for other videos, and the GPX track points that are read (#935). --- README.md | 90 +++++++++++++++++++++++++------------------------------ 1 file changed, 40 insertions(+), 50 deletions(-) diff --git a/README.md b/README.md index 5b0b7915..4b2e4c6d 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,8 @@ mapillary_tools process_and_upload MY_CAPTURE_DIR mapillary_tools --help ``` +See the [command reference](docs/command_reference.md) for what each command does, its options, and common pitfalls. + - [Supported File Formats](#supported-file-formats) @@ -45,6 +47,7 @@ mapillary_tools --help - [Process and Upload](#process-and-upload) - [Process](#process) - [Upload](#upload) + - [Check Upload History](#check-upload-history) - [Advanced Usage](#advanced-usage) - [Local Video Processing](#local-video-processing) - [Install FFmpeg](#install-ffmpeg) @@ -78,7 +81,7 @@ mapillary_tools can upload both images and videos. ## Image Formats -mapillary_tools supports JPG/JEPG images (.jpg, .jpeg), with the following EXIF tags minimally required: +mapillary_tools supports JPG/JPEG images (.jpg, .jpeg), with the following EXIF tags minimally required: - GPS Longitude - GPS Latitude @@ -100,6 +103,10 @@ mapillary_tools supports videos (.mp4, .360) that contain any of the following t - [BlackVue](https://blackvue.com/) videos - [DR900S-1CH](https://shop.blackvue.com/product/dr900x-1ch-plus/) - [DR900X Plus](https://shop.blackvue.com/product/dr900x-2ch-plus/) +- Novatek-based dashcams, such as Viofo, Vantrue and Anker Roav dashcams + +Other videos can be geotagged from the GPS that [ExifTool](https://exiftool.org/) reads, if it is installed, +or from a GPX or NMEA track recorded along with them (see [Geotagging with GPX](#geotagging-with-gpx)). # Installation @@ -209,7 +216,7 @@ mapillary_tools process MY_CAPTURE_DIR MY_VIDEO_DIR/*.mp4 Duplicate check with custom distance and angle: ```sh -# Mark images that are 3 meters closer to its previous one as duplicates. +# Mark images within 3 meters of the previous one as duplicates. # Duplicates won't be uploaded mapillary_tools process MY_CAPTURE_DIR \ --duplicate_distance 3 \ @@ -223,36 +230,22 @@ Split sequences with the custom cutoff distance or custom capture time gap: # OR their capture times are 120 seconds apart, # then split the sequence from there mapillary_tools process MY_CAPTURE_DIR \ - --offset_angle 90 \ --cutoff_distance 100 \ - --cutoff_time 120 \ + --cutoff_time 120 ``` -To receive a machine-readable inventory of recognized inputs and unsupported -files without changing the image description format, specify an optional -process report: +In a directory, files that are not supported images or videos are left out. A file named on the command line that +is not one fails, with a `MapillaryUnsupportedFileError` entry in the image description file, unless it is a +sidecar or companion file such as `.gpx`, `.xml` or `.thm`: those are skipped with a warning. + +To write a JSON report of the files found and the unsupported files skipped, specify `--process_report_path` +(see the [command reference](docs/command_reference.md#process) for its format): ```sh mapillary_tools process MY_CAPTURE_DIR \ --process_report_path /tmp/mapillary_process_report.json ``` -The version 1 report contains `discovered_file_count`, `processed_file_count`, -`skipped_file_count`, and `skipped_files`. Recognized files that produce a -description-level processing error are included in `processed_file_count`. -Unsupported entries include their filename, category, stable reason code, and -normalized extension. Hidden/system files and common GPS or metadata sidecars -are ignored during directory discovery, but an explicitly supplied unsupported -file is always reported. Such a file also fails processing, with a -`MapillaryUnsupportedFileError` entry in the image description file, whether or -not the report is requested, unless it is a known sidecar or companion file -(`.thm`, `.lrv`, `.lrf`, `.aae`, `.gpx`, `.nmea`, `.kml`, `.kmz`, `.fit`, -`.xml`, `.json`, `.xmp`, `.srt`, `.txt`, `.csv`, `.log`, `.md5`, macOS `._*` -files and `.DS_Store`, and Windows `Thumbs.db` and `desktop.ini`): those are -skipped with a warning, so that all the files of a camera folder can be named. -The report is written only to the requested local path and contains absolute -source filenames, so treat it as local capture metadata. - ## Upload After processing you should get the [image description file](#image-description). Pass it to the `upload` command to upload them: @@ -288,6 +281,8 @@ A name is 2 to 32 letters, numbers, hyphens and underscores, starting with a let Saving a name that is already saved with another ID replaces the ID. To list or delete saved names, see [Authenticate](#authenticate). +Files that were uploaded from this computer before are skipped. To upload them again, specify `--reupload`. + ## Check Upload History To check whether processed image sequences or videos were previously uploaded @@ -299,15 +294,9 @@ mapillary_tools check_upload_history MY_CAPTURE_DIR \ --desc_path /tmp/mapillary_image_description.json ``` -The command prints a JSON array with one entry per upload candidate. Each entry -contains the file type, sequence UUID when applicable, sequence checksum, -member filenames, the subset found in history as -`already_uploaded_filenames`, and an `already_uploaded` boolean. -`already_uploaded` is true only when every member filename is represented in -local history. For images, this includes files that were members of a larger -previously uploaded sequence. It reads only the local upload history: it does -not require authentication, make network requests, upload data, or modify -history. +It prints a JSON array with one entry per image sequence or video, which tells whether it was already uploaded +(see the [command reference](docs/command_reference.md#check_upload_history)). It reads only the local upload +history: it needs no login, makes no network requests, and changes nothing. # Advanced Usage @@ -365,6 +354,9 @@ gap of more than an hour between two of its points (such as between two trips in and are not geotagged from it. The same applies to videos. To allow longer gaps, set the environment variable `MAPILLARY_TOOLS_MAX_GPX_GAP_SECONDS` (3600 by default). +Only the track points (``) of a GPX file that have a time are read, not its routes or waypoints. A GPX file +in which no track point has a time that can be read fails with an error that says why. + ```sh mapillary_tools process MY_IMAGE_DIR --geotag_source "gpx" --geotag_source_path MY_EXTERNAL_GPS.gpx ``` @@ -464,10 +456,10 @@ PATTERN specifies how the data source file is named, starting from the video fil Supported sources and their default pattern are: * `native`, or `video`: parse the video looking for geolocation in any format that mapillary_tools reads: GoPro, CAMM, -BlackVue and others. Pattern: `%f` -* `camm`: parse the video looking for a CAMM track. Pattern: `%f` -* `gopro`: parse the video looking for geolocation in GoPro format. Pattern: `%f` -* `blackvue`: parse the video looking for geolocation in BlackVue format. Pattern: `%f` +BlackVue and others. No pattern: it reads the video itself +* `camm`: parse the video looking for a CAMM track. No pattern +* `gopro`: parse the video looking for geolocation in GoPro format. No pattern +* `blackvue`: parse the video looking for geolocation in BlackVue format. No pattern * `gpx`: external GPX file. Pattern: `%g.gpx` * `nmea`: external NMEA file. Pattern: `%g.nmea` * `exiftool_xml`: external XML file generated by exiftool. Pattern: `%g.xml` @@ -477,11 +469,12 @@ Notes: * `exiftool_runtime` only works if exiftool is installed on the system. You can find it at https://exiftool.org/ or through your software manager. If exiftool is installed, but is not in the default execution path, it is possible to specify its location by setting the environment variable `MAPILLARY_TOOLS_EXIFTOOL_PATH`. -* Pattern are case-sensitive or not depending on the filesystem - in Windows, `%g.gpx` will match both `basename.gpx` -and `basename.GPX`, in MacOs, Linux or other Unix systems no. -* If both `--video_geotag_source` and `--geotag_source` are specified, `--video_geotag_source` will apply to video files -and `--geotag_source` to image files. If only `--video_geotag_source` is specified, image files (including the frames -that `video_process` samples) are geotagged by the default sources. +* Patterns are case-sensitive or not depending on the file system: `%g.gpx` matches both `basename.gpx` and +`basename.GPX` on Windows and, by default, on macOS, but not on Linux. +* If both `--video_geotag_source` and `--geotag_source` are specified, the `--video_geotag_source` sources are tried +first, on videos only, and the `--geotag_source` sources then geotag the images and the videos left. If only +`--video_geotag_source` is specified, image files (including the frames that `video_process` samples) are geotagged +by the default sources. ### Examples @@ -571,14 +564,14 @@ Here is a minimal example: "MAPLatitude": 58.5927694, "MAPLongitude": 16.1840944, "MAPCaptureTime": "2021_02_13_13_24_41_140", - "filename": "/MY_IMAGE_DIR/IMG_0291.jpg" + "filename": "/MY_IMAGE_DIR/IMG_0001.jpg" }, { "error": { "type": "MapillaryGeoTaggingError", "message": "Unable to extract GPS Longitude or GPS Latitude from the image" }, - "filename": "/MY_IMAGE_DIR/IMG_0292.jpg" + "filename": "/MY_IMAGE_DIR/IMG_0002.jpg" } ] ``` @@ -602,18 +595,15 @@ mapillary_tools upload MY_IMAGE_DIR --desc_path /tmp/mapillary_image_descriptio ## Zip Images -When [uploading](#upload) an image directory, internally the `upload` command will zip sequences in the temporary -directory (`TMPDIR`) and then upload these zip files. - -mapillary_tools provides `zip` command that allows users to specify where to store the zip files, usually somewhere with -faster IO or more free space. +The `zip` command writes each processed image sequence into a zip file, with the metadata in the images. +The `upload` command uploads zip files as they are, so they can be uploaded later. ```sh # Zip processed images in MY_IMAGE_DIR and write zip files in MY_ZIPFILES mapillary_tools zip MY_IMAGE_DIR MY_ZIPFILES -# Upload all the zip files (*.zip) in MY_ZIPFILES: -mapillary_tools upload --file_types zip MY_ZIPFILES +# Upload all the zip files (*.zip) in MY_ZIPFILES +mapillary_tools upload MY_ZIPFILES --desc_path MY_IMAGE_DIR/mapillary_image_description.json ``` # Development From 531e3402f9cbeeee7970e26eb7b752d1e7014952 Mon Sep 17 00:00:00 2001 From: Caglar Pir Date: Tue, 27 Oct 2026 15:58:58 +0100 Subject: [PATCH 2/4] Link the command reference by URL, and fix what the README says of the upload history PyPI shows the README as the project description, where relative links lead nowhere: link the reference by its GitHub URL, as the schema link and the logo are. upload compares each image sequence, video or zip file as a whole with the upload history, so a sequence with one image added is uploaded again; check_upload_history reports zip files too. In process, a timelapse without GPS of its own works with a GPX of the whole track, which process only warns of, as the command reference says. --- README.md | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 4b2e4c6d..eb11867c 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ mapillary_tools process_and_upload MY_CAPTURE_DIR mapillary_tools --help ``` -See the [command reference](docs/command_reference.md) for what each command does, its options, and common pitfalls. +See the [command reference](https://github.com/mapillary/mapillary_tools/blob/main/docs/command_reference.md) for what each command does, its options, and common pitfalls. @@ -239,7 +239,7 @@ is not one fails, with a `MapillaryUnsupportedFileError` entry in the image desc sidecar or companion file such as `.gpx`, `.xml` or `.thm`: those are skipped with a warning. To write a JSON report of the files found and the unsupported files skipped, specify `--process_report_path` -(see the [command reference](docs/command_reference.md#process) for its format): +(see the [command reference](https://github.com/mapillary/mapillary_tools/blob/main/docs/command_reference.md#process) for its format): ```sh mapillary_tools process MY_CAPTURE_DIR \ @@ -281,11 +281,13 @@ A name is 2 to 32 letters, numbers, hyphens and underscores, starting with a let Saving a name that is already saved with another ID replaces the ID. To list or delete saved names, see [Authenticate](#authenticate). -Files that were uploaded from this computer before are skipped. To upload them again, specify `--reupload`. +Image sequences, videos and zip files that were uploaded from this computer before are skipped. Each is compared as a +whole: a sequence with one image added or removed is uploaded again. To upload them again anyway, specify +`--reupload`. ## Check Upload History -To check whether processed image sequences or videos were previously uploaded +To check whether processed image sequences, videos or zip files were previously uploaded from this device, use the same import paths and description file that would be passed to `upload`: @@ -294,9 +296,10 @@ mapillary_tools check_upload_history MY_CAPTURE_DIR \ --desc_path /tmp/mapillary_image_description.json ``` -It prints a JSON array with one entry per image sequence or video, which tells whether it was already uploaded -(see the [command reference](docs/command_reference.md#check_upload_history)). It reads only the local upload -history: it needs no login, makes no network requests, and changes nothing. +It prints a JSON array with one entry per image sequence, video or zip file, which tells whether it was already +uploaded (see the +[command reference](https://github.com/mapillary/mapillary_tools/blob/main/docs/command_reference.md#check_upload_history)). +It reads only the local upload history: it needs no login, makes no network requests, and changes nothing. # Advanced Usage @@ -510,7 +513,9 @@ VIDEO_DIR/ ### Limitations of `--video_geotag_source` **A GPX track is synced by the GPS clock of the video, if the video has GPS of its own, and otherwise aligned with -the start of the video.** In `process`, a timelapse video is supported for a GPX only if it has GPS of its own. +the start of the video.** In `process`, a video without GPS of its own is assumed to start with the track, and a track +that lasts more than twice as long as the video gets a warning: unless the video is a timelapse of the whole track, trim +the GPX to the video, or sample the video with `video_process` and `--video_start_time`. As with images, `--interpolation_offset_time N` adds N seconds to the time of the video to find its positions along the GPX track, and `--interpolation_use_gpx_start_time` aligns the start of the video with the start of the track, even From 8c02556c16f8ec6754320ab0e4c3ea175361fdea Mon Sep 17 00:00:00 2001 From: Caglar Pir Date: Fri, 30 Oct 2026 11:42:08 +0100 Subject: [PATCH 3/4] Follow #944 and #945 in the README A GPX is assumed to start with a video without GPS, in process and in every kind of sampling, with a warning when the two differ in length, and is never placed by the creation time: only --interpolation_use_gpx_start_time still refuses a long track, in video_process. The process report lists the geotagging warnings. --- README.md | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index eb11867c..75e78352 100644 --- a/README.md +++ b/README.md @@ -238,8 +238,9 @@ In a directory, files that are not supported images or videos are left out. A fi is not one fails, with a `MapillaryUnsupportedFileError` entry in the image description file, unless it is a sidecar or companion file such as `.gpx`, `.xml` or `.thm`: those are skipped with a warning. -To write a JSON report of the files found and the unsupported files skipped, specify `--process_report_path` -(see the [command reference](https://github.com/mapillary/mapillary_tools/blob/main/docs/command_reference.md#process) for its format): +To write a JSON report of the files found, the unsupported files skipped and the warnings about the files processed, +specify `--process_report_path` (see the +[command reference](https://github.com/mapillary/mapillary_tools/blob/main/docs/command_reference.md#process) for its format): ```sh mapillary_tools process MY_CAPTURE_DIR \ @@ -376,11 +377,9 @@ mapillary_tools video_process MY_VIDEO.mp4 \ --video_start_time 2025_03_14_07_00_30_000 ``` -Without it, distance-based sampling (the default) assumes that the track starts with the video, and refuses a track -more than twice as long as the video. Interval-based sampling (`--video_sample_distance -1 --video_sample_interval N`) -reads the start time from the video creation time instead. Cameras disagree on whether the creation time marks the start -or the end of the recording (most dashcams write the end), so mapillary_tools uses the camera model or a date and time -in the file name to tell which, and assumes the start when neither says. +Without it, the track is assumed to start with the video. The creation time of a video, which comes from the camera's +clock, is never used to place a track. When the track lasts more than a minute longer or shorter than the video, +mapillary_tools warns: trim the GPX to the video, or specify `--video_start_time`. A timelapse video packs several seconds of capture into each second of video. Its GPS clock, if it has one, tells how many. If it has none, specify how many with `--video_duration_ratio`, for instance 60 for a timelapse that took a frame @@ -412,8 +411,8 @@ misses fails. To avoid that, process the images on their own with `--file_types Another option `--interpolation_use_gpx_start_time` moves your images to align with the beginning of the GPS track. This is useful when you can confirm that you start GPS recording and capturing at the same time, or with a known delay. -For a video without a GPS clock, a track that lasts more than twice as long as the video spans at its -`--video_duration_ratio` is then refused. If the GPS logger was left running after the video, pass the start of the +For a video without a GPS clock, `video_process` then refuses a track that lasts more than twice as long as the video +spans at its `--video_duration_ratio`. If the GPS logger was left running after the video, pass the start of the track as `--video_start_time` instead, which places the video the same way: the error prints it. ```sh @@ -513,9 +512,9 @@ VIDEO_DIR/ ### Limitations of `--video_geotag_source` **A GPX track is synced by the GPS clock of the video, if the video has GPS of its own, and otherwise aligned with -the start of the video.** In `process`, a video without GPS of its own is assumed to start with the track, and a track -that lasts more than twice as long as the video gets a warning: unless the video is a timelapse of the whole track, trim -the GPX to the video, or sample the video with `video_process` and `--video_start_time`. +the start of the video.** A video without GPS of its own is assumed to start with the track, with a warning when the +track lasts more than a minute longer or shorter than the video: trim the GPX to the video, or sample the video with +`video_process` and `--video_start_time`. The track of a timelapse is expected to last longer, and is uploaded whole. As with images, `--interpolation_offset_time N` adds N seconds to the time of the video to find its positions along the GPX track, and `--interpolation_use_gpx_start_time` aligns the start of the video with the start of the track, even From 5f7fc252ae6bbd40d84e78a900cdb345ac446021 Mon Sep 17 00:00:00 2001 From: Caglar Pir Date: Wed, 4 Nov 2026 14:41:28 +0100 Subject: [PATCH 4/4] Say that a GPX far longer than the video is refused again (#948) video_process refuses a track that runs on past the end of the video for longer than the video lasts, and process a track of a day or more on a video less than half as long. --- README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 75e78352..f116345c 100644 --- a/README.md +++ b/README.md @@ -379,7 +379,9 @@ mapillary_tools video_process MY_VIDEO.mp4 \ Without it, the track is assumed to start with the video. The creation time of a video, which comes from the camera's clock, is never used to place a track. When the track lasts more than a minute longer or shorter than the video, -mapillary_tools warns: trim the GPX to the video, or specify `--video_start_time`. +mapillary_tools warns: trim the GPX to the video, or specify `--video_start_time`. A track that runs on past the end +of the video for longer than the video lasts is refused, as the video could start anywhere along it: the error prints +the start of the track, to pass as `--video_start_time` if the GPS logger was started with the video. A timelapse video packs several seconds of capture into each second of video. Its GPS clock, if it has one, tells how many. If it has none, specify how many with `--video_duration_ratio`, for instance 60 for a timelapse that took a frame @@ -514,7 +516,8 @@ VIDEO_DIR/ **A GPX track is synced by the GPS clock of the video, if the video has GPS of its own, and otherwise aligned with the start of the video.** A video without GPS of its own is assumed to start with the track, with a warning when the track lasts more than a minute longer or shorter than the video: trim the GPX to the video, or sample the video with -`video_process` and `--video_start_time`. The track of a timelapse is expected to last longer, and is uploaded whole. +`video_process` and `--video_start_time`. The track of a timelapse is expected to last longer, and is uploaded whole, +but a track of a day or more is refused unless the video lasts more than half as long. As with images, `--interpolation_offset_time N` adds N seconds to the time of the video to find its positions along the GPX track, and `--interpolation_use_gpx_start_time` aligns the start of the video with the start of the track, even