NIFI-15958 Add progress logging for repository scanning

[exceptionfactory]

Summary

NIFI-15958 Adds incremental logging to the Provenance Repository for re-indexing of partitions and also adds incremental logging to the File System Repository for archive files.

This approach maintains existing behavior, but provides an indication of progress during operations that can take longer for large amounts of file-based indexing.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000
• Pull request contains commits signed with a registered key indicating Verified status

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using ./mvnw clean install -P contrib-check
  • JDK 21
  • JDK 25

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files

[joewitt]

one nit to avoid confusion on the log output

Drop the second “of”. Keep files as the only fraction. Treat events as a running count, same as the existing finish line (Finished re-indexing {} events across {} files).

Current

Re-indexed {} of {} Files for Partition [{}] including {} of {} Events

That reads as two progress bars. The second is records indexed so far over maxEventId - minEventIdToReindex, which is an ID span, not a file or record total. People will do 8,000/50,000 in their head and think they are 16% done when files are already done.

Better

Re-indexing partition [{}]: completed {} of {} event files; {} events indexed so far

With the same args: partitionName, filesCompleted, totalFilesToReindex, reindexedCount.get().

Why

• “X of Y event files” is the thing that actually advances on a timer (one file finished, including skip/fail). That is the hung-or-not heartbeat.
• “N events indexed so far” is still useful (index is moving), and it matches the completion log, which already does not say of {eventsToReindex}.
• You do not lose the ID-range context. That is already on the line two screens up: last indexed ID vs last written ID, “re-indexing up to the last {} events.” Repeating it as a denominator is what makes the new line a fake percent.

If you want one extra crumb without lying, append the IDs, not a fraction:

…; {} events indexed so far (target IDs {}–{})

minEventIdToReindex and maxEventId are real bounds. Still not “percent of events left.”

I would not log eventsToReindex in the progress line at all.

[exceptionfactory]

Thanks for the feedback @joewitt. I made a more slight adjustment, removing the eventsToIndex number to avoid the confusion around number of total events. I didn't modify the rest of the log message, but if you prefer the restructuring, I can push another change.

[joewitt]

+1. merging now. thanks