IGNITE-27833 [ducktests] Add Multi-DC Integration Test for Cross-DC Network Partition Resilience

[maksaska]

Thank you for submitting the pull request to the Apache Ignite.

In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:

The Contribution Checklist

• There is a single JIRA ticket related to the pull request.
• The web-link to the pull request is attached to the JIRA ticket.
• The JIRA ticket has the Patch Available state.
• The pull request body describes changes that have been made.
  The description explains WHAT and WHY was made instead of HOW.
• The pull request title is treated as the final commit message.
  The following pattern must be used: IGNITE-XXXX Change summary where XXXX - number of JIRA issue.
• A reviewer has been mentioned through the JIRA comments
  (see the Maintainers list)
• The pull request has been checked by the Teamcity Bot and
  the green visa attached to the JIRA ticket (see TC.Bot: Check PR)

Notes

• How to Contribute
• Coding abbreviation rules
• Coding Guidelines
• Apache Ignite Teamcity Bot

If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com #ignite channel.

[anton-vinogradov]

Reviewed the whole PR. The netem/iptables layering (tcset baseline + per-pair DROP chains, so healing is a pure chain flush) is nicely designed, and the tests read well. A few findings below, most with one-line fixes; the two control_utility.py ones and the connect_timeout default affect shared infrastructure beyond MDC.

Nits not tied to a diff line:

• modules/ducktests/tests/tox.ini envlist still contains py38, while the workflow dropped it and tcconfig==0.30.1 requires Python >= 3.9 — a local tox run will fail on the py38 env. Worth removing it there too.
• thin_client_test.py reserves @cluster(num_nodes=8) but allocates 7 (2+2 servers, 1 runner, 2 thin clients).

[anton-vinogradov]

Round 2: verified all previous threads against the new head - every one of them is properly fixed (including the tcconfig==0.29.1 route: it requires only Python >=3.7, so the py38 env is consistent again; also compiled with -Pducktests and the strict -Pcheckstyle profile locally - clean).

The final pass still found a few things, two of them merge-blocking:

• blocker: the "codestyle" commit accidentally commented out the Ignite/Zookeeper/Kafka installs in the ducktests Docker image;
• blocker: the new ClusterState.coordinator field breaks positional unpacking of cluster_state() in existing tests;
• one correction to my own round-1 comment about --user-attributes ordering (it is client-side HashMap order, not TreeMap order - details inline, sorry for the detour);
• plus a handful of test-strength/robustness notes and two doc nits.

Details inline.