test: move system-tests that are too big for Namespace to system_test_large

[basvandijk]

To prepare for moving the "CI Main" workflow fully to Namespace we need to move out any system-test that's too big for Namespace's 32 CPU RBE workers.

The following tests are moved to the "Release System Tests" job of the "Release Testing" workflow which runs on pushes to rc--* branches which are created from master every night:

• //rs/tests/consensus/subnet_recovery:sr_app_failover_nodes_test
• //rs/tests/consensus/subnet_recovery:sr_app_failover_nodes_enable_chain_keys_test
• //rs/tests/consensus/subnet_recovery:tecdsa_signature_same_subnet_test
• //rs/tests/message_routing/xnet:subnet_delete_test
• //rs/tests/nested/nns_recovery:nr_broken_dfinity_node
• //rs/tests/nested/nns_recovery:nr_local
• //rs/tests/nested/nns_recovery:nr_no_elect_fix_like_np
• //rs/tests/nested/nns_recovery:nr_all_broken_seq_np_actions
• //rs/tests/nns:nns_delegation_mainnet_nns_version_test

//rs/tests/networking:nns_delegation_mainnet_nns_version_test is now supported on the local backend.

[bjoernek]

LGTM from governance.

Two things before you merge:

1. Once it's system_test_large there's no pre-merge escape hatch left: system_test_large is in EXCLUDED_TAGS in ci/scripts/targets.py, applied after everything else, so neither CI_ALL_BAZEL_TARGETS nor a PULL_REQUEST_BAZEL_TARGETS entry will run it on a PR any more. Is that intended? If someone is doing risky work on subnet creation, running it by hand is the only option.

2. Who watches "Release System Tests" failures? I don't see a notification step in release-testing.yml.

[basvandijk]

@bjoernek regarding 1: there would indeed be no way to automatically run //rs/tests/nns:create_subnet_test from a PR (except for manually triggering the "Release Testing" workflow on your PR which would be expensive). This is intended. Would that be an issue?

Who watches "Release System Tests" failures? I don't see a notification step in release-testing.yml.

Infra and DRE do. We have the #release-management-alerts channel for that.