[#800] Allocate the replication test listen ports per invocation

[vharseko]

Fixes #800 — item 2 of the Proposed fixes of #792, deliberately left out of #795.

Problem

AssuredReplicationServerTest allocated the four listen ports of its real replication
servers once, in @BeforeClass, and its 360 invocations then re-created those servers
on the same four numbers for ~20 minutes.

TestCaseUtils.findFreePorts() binds a probe socket, closes it and returns the number, so
between two invocations nothing holds the port: an ephemeral local port, a socket the
previous invocation is still closing, or a self-connect can take it, and the bind of the
next replication server fails. That is how #792 happened, on invocation 26 of the class.

Note what the window actually is. While a replication server listens on its port, the
kernel will not hand that number out as an ephemeral local port, so the port is only
unprotected between the teardown of one invocation and the bind of the next — tens to
hundreds of milliseconds, 360 times. Allocating a fresh number per invocation shrinks that
window to the microseconds between the probe socket being closed and the listen socket
being bound, and, more importantly, hands out a number which was never bound in this JVM
before, so nothing left over from the previous invocation can be sitting on it.

Changes

• AssuredReplicationServerTest allocates its four ports in initTest(), which every
  invocation runs before it creates anything — the four methods which have no initTest()
  of their own (testSafeDataLevelOnePrecommit, testSafeDataLevelHighPrecommit,
  testSafeDataLevelHighNightly, testSafeReadOneRSComplexPrecommit) delegate to the
  bodies which do. All four ports are taken at once because the multi-replication-server
  tests configure each server with the URLs of the others, so the numbers cannot change in
  the middle of a topology. The setUp() override became super.setUp() only and is
  dropped: @BeforeClass setUp() is inherited from ReplicationTestCase, as in the thirty
  other classes of the package which do not override it.
• FractionalReplicationTest (36 invocations) had the same shape — port pinned in
  @BeforeClass, a replication server created on it by every invocation — and moves the
  allocation into its existing initTest().
• AssuredReplicationPluginTest (13 invocations) is the same again, with the fake
  replication server binding the pinned port in start(). It has no per test hook, hence a
  @BeforeMethod; both the fake replication server and the domain configuration entry
  which points at it are created by the test method itself, so they see the new number.
• TestCaseUtils.findFreePorts() closed its probe sockets inside the try block and left
  an empty finally: an allocation which failed halfway left the sockets it had already
  bound open, holding their ports in listen state, and bindFreePort() hands out each
  number only once. They are closed in the finally block now.

Port budget: the class which needs it most now takes 360 × 4 = 1440 numbers, and
bindFreePort() walks down from 65531, so this JVM ends around 64091 — well above the
Linux ephemeral range (32768–60999). Worth keeping in mind if
testSafeDataLevelHigh (enabled = false) is ever switched on.

Not changed

The issue lists nine classes calling TestCaseUtils.findFreePorts(). Eight of them do not
re-bind a pinned port and are left alone:

• TopologyViewTest, ReplicationServerFailoverTest, GroupIdHandshakeTest,
  ReplicationServerLoadBalancingTest, ReplicationDomainTest and
  ReplicationServerDynamicConfTest already allocate per test method — none of them is
  data-provider driven, so a test method is an invocation.
• GenerationIdTest re-allocates its three ports in postTest()
  (GenerationIdTest.java:1064); four test methods, seven replication servers.
• ReplicationServerTest takes its port in configure() from @BeforeClass but creates
  the replication server once for the class, and takes a fresh port when it restarts it
  (stopChangelog()). ChangelogBackendTestCase, which is not in the list, is the same:
  one replication server per class, removed in @AfterClass.

Relation to #795

#795 is not merged yet, so on master a replication server whose listen port is taken still
only logs ERR_COULD_NOT_BIND_CHANGELOG and comes up without a listener. Once it is in,
its retry makes a momentarily occupied port self-healing and its probe reports what was
observed on the port, which is what will identify the holder if this ever happens again.
This change is independent of it and reduces the number of chances to find out.

Tests

mvn -o -pl opendj-server-legacy verify -P precommit -Dit.test=AssuredReplicationServerTest
Tests run: 360, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1242 s

mvn -o -pl opendj-server-legacy verify -P precommit \
    -Dit.test='FractionalReplicationTest,AssuredReplicationPluginTest'
Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 91 s   -- FractionalReplicationTest
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 193 s  -- AssuredReplicationPluginTest

The whole replication package is running locally as well, since TestCaseUtils is shared
by every test; the result will be added as a comment.

[vharseko]

The whole replication package, locally, with -P precommit:

mvn -o -pl opendj-server-legacy verify -P precommit \
    -Dit.test='**/replication/**/*Test.java,**/replication/**/*TestCase.java'

Tests run: 3488, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS -- 47:30 min