Skip to content

security: close the remaining gaps after the Grafana hardening - #512

Open
OisinKyne wants to merge 2 commits into
mainfrom
oisin/hardening-follow-up
Open

OisinKyne wants to merge 2 commits into
mainfrom
oisin/hardening-follow-up

Conversation

@OisinKyne

@OisinKyne OisinKyne commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Follow-up to the Grafana hardening. It closes the remaining ways the default setup, or a setup copied from the samples, could expose more than the P2P ports.

  • Example ports bind to 127.0.0.1. The commented-out ports in docker-compose.override.yml.sample published the EL RPC and Engine API (8545/8551), the beacon API and charon's 3600/3620 on all interfaces once uncommented. Docker-published ports bypass ufw, so they're now 127.0.0.1: with a note saying why.
  • check_exposure.py fails CI on more:
    • public ports in *.sample files;
    • ${VAR-default} in ports, which an empty VAR= in .env turns into all interfaces;
    • Grafana accepting passwords: [auth] disable_login_form and [auth.basic] enabled in grafana.ini, any GF_* variable or env_file on Grafana (both override the ini file);
    • Grafana joining a network other than monitoring, or mounting repository files writable (short or long syntax);
    • the Docker socket mounted anywhere but docker-socket-proxy;
    • privileged, or host network, PID or IPC namespaces, outside compose-debug.yml.
  • security-check.sh looks for more signs of tampering in the old data/grafana: extra organizations, dashboards that aren't provisioned, alert rules, contact points, and users, tokens, organizations, alert rules or dashboards that were created and later deleted (from sqlite_sequence). On Grafana 12 and later, dashboards and folders live in the resource table, so it also lists those not annotated grafana.app/managedBy: classic-file-provisioning, and deleted ones from resource_history. Datasource deletions aren't counted, because provisioning deletes and recreates the datasources on every start; unexpected datasource names still are. It lists changed files under grafana/, which older releases mounted writable, and warns against git stash apply bringing them back. Added files are listed for review without failing the check, since the README suggests adding custom dashboards there. It prints commands to archive data/grafana with a hash before deleting it. The README describes the extended check and tells operators to rotate secrets stored in Grafana, which encrypts them with a default key.
  • security-check.sh reports every published port that isn't P2P and is bound to a public address, from docker compose ps. Exposed ports are counted separately from signs of tampering, so a port finding alone doesn't print the evidence-archiving advice. The exit status fails on either.
  • Read-only mounts. Grafana's provisioning files and dashboards are mounted read-only, so Grafana can't change its own provisioning on the host.
  • README re-enable-logins snippet turns anonymous access off, sets GF_SECURITY_SECRET_KEY, and says to run reset-admin-password before restarting, since an existing Grafana volume still has admin / admin. It no longer enables basic auth, which the login form doesn't need.
  • Client version variables. .env.sample documents CL_GRANDINE_VERSION, CL_LIGHTHOUSE_VERSION, CL_TEKU_VERSION, CL_LODESTAR_VERSION, NETHERMIND_VERSION and RETH_VERSION, but the compose files read other names, so a pinned client version was silently ignored. They now read the documented name and fall back to the old one. Release note: a version left uncommented in .env, which was ignored until now, takes effect after upgrading.
  • Nethermind JSON-RPC modules are limited to Eth,Subscribe,TxPool,Web3,Proof,Net,Health,Rpc. Nethermind's defaults also enable Trace and Parity, which nothing in the stack uses. The Engine API port has its own module list and is unaffected.

Not included, for a tested release: a per-node Engine API JWT instead of the committed jwt/jwt.hex; running Prometheus and Loki as non-root; a narrower Docker socket proxy.

- Bind the example ports in docker-compose.override.yml.sample to
  127.0.0.1, since Docker-published ports bypass ufw.
- Extend check_exposure.py to example ports, GF_* variables, env_file,
  networks and writable mounts on Grafana, Docker socket mounts and
  privileged or host-namespace services.
- Report every non-P2P port published on a public address in
  security-check.sh, separately from signs of tampering.
- Have the README's re-enable-logins snippet turn anonymous access off,
  set a secret key, and reset the admin password first.
- Read the client version variables that .env.sample documents.
- Limit Nethermind's JSON-RPC modules to the ones the stack needs.
…dashboards

Provisioning deletes and recreates datasources on every start, so stop
checking deleted datasource IDs; the datasource names are still checked.
Grafana 12 and later store dashboards and folders in the resource table,
so also report ones not managed by file provisioning, and deleted ones
from resource_history. List added files under grafana/ for review
instead of counting them, since operators may add their own dashboards,
and no longer suggest git clean.
@sonarqubecloud

Copy link
Copy Markdown

OisinKyne added a commit that referenced this pull request Oct 28, 2026
## Summary

Backport of #512 to `main-v1.11`, plus the charon bump to v1.11.3, which
redacts credentials and API keys in logged beacon node and execution
URLs.

- Cherry-picks both #512 commits. The client version variable fix
conflicted on the default versions; this line keeps its own defaults
(Grandine 2.0.6, Lighthouse v8.3.1, Nethermind 2.2.0, Reth v2.8.0) and
takes the documented variable names.
- `charon` v1.11.2 → v1.11.3 in the compose files and the `scripts/edit`
scripts. The edit e2e test keeps v1.11.2 until the image exists.

**Merge after charon v1.11.3 is published**, otherwise `docker compose
pull` fails.

**Release note:** a client version left uncommented in `.env`, which was
ignored until now, takes effect after upgrading.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant