Skip to content

DEVNOTES.md

Latest commit

 

History

History
291 lines (211 loc) · 11.2 KB

DEVNOTES.md

File metadata and controls

291 lines (211 loc) · 11.2 KB
Raw

Local Development

Implementation plans and migration design notes live in docs/plans. Check that directory before starting larger feature, migration, or refactor work.

Check out repository:

git clone git@github.com:broadinstitute/consent.git

Build, test

cd consent
mvn clean test package

Tests require docker or another OCI runtime to spin up postgres and http servers that run against localhost. Ensure that your test environment supports that. Command line docker, podman, and Docker Desktop have all worked to run these tests. Podman was observed to perform better when running tests, however docker is the tool of choice for other developers and the CI/CD environment.

Docker

Consent docker images are stored in GCR: gcr.io/broad-dsp-gcr-public/consent

# build the docker image
docker build . -t consent

This image can then be run with the proper configuration files provided.

Render Configs

Specific to internal Broad systems:

APP_NAME=consent ENV=local OUTPUT_DIR=config ../firecloud-develop/configure.rb

Otherwise, use src/test/resources/consent-config.yml as a template to create your own environment-specific configuration.

Spin up application:

Specific to internal Broad systems:

mvn clean compile
docker-compose -p consent -f config/docker-compose.yaml up

Or, if not using docker:

java -jar /path/to/consent.jar server /path/to/config/file

Visit local swagger page: https://local.dsde-dev.broadinstitute.org:27443

Debugging

Port 7777 is open in the configured docker compose. Set up a remote debug configuration pointing to local.dsde-dev.broadinstitute.org and the defaults should be correct.

Developing with a local Elastic Search instance:

elastic:
   image: docker.elastic.co/elasticsearch/elasticsearch:9.4.4
   ports:
     - "9200:9200"
   container_name: elastic
   volumes:
     - elastic:/usr/share/elasticsearch/data
   deploy:
     resources:
       limits:
         memory: 4gb
   environment:
     - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
     # X-Pack Security is OFF by default so the default `docker compose up` behaves exactly as
     # before. Epics A-C and E (application-layer fallback) need no security. To work on Epic D
     # (native DLS/FLS), start the stack with security on:
     #
     #   ES_SECURITY_ENABLED=true docker-compose -p consent -f config/docker-compose.yaml up
     #
     # DLS/FLS is a Platinum feature, so also activate the 30-day trial license once per cluster:
     #
     #   curl -u elastic:devpassword -XPOST 'localhost:9200/_license/start_trial?acknowledge=true'
     #
     # See DEVNOTES.md ("Developing with a local Elastic Search instance") for the full workflow.
     - xpack.security.enabled=${ES_SECURITY_ENABLED:-false}
     # Bootstraps the `elastic` superuser password when security is on; ignored when it is off.
     # Must match authUser/authPassword in consent.yaml.
     - ELASTIC_PASSWORD=${ELASTIC_PASSWORD:-devpassword}
     # Correct in both modes: transport SSL is only required for multi-node clusters.
     - xpack.security.transport.ssl.enabled=false
     # Keep the HTTP layer on plain http so consent's `protocol: http` client keeps working.
     - xpack.security.http.ssl.enabled=false

I also suggest changing the default bucket location so uploaded ontology files do not interfere with other dev environments.

Running local Elastic Search with security enabled

By default the local cluster runs with xpack.security.enabled=false: requests are unauthenticated, which is what most work needs. Only work on native Elasticsearch document- and field-level security (DLS/FLS) requires security to be on. Application-layer authorization work does not — it never touches Elasticsearch security.

Security is env-var gated in the compose file, so the default stays off for everyone else:

mvn clean compile
ES_SECURITY_ENABLED=true docker-compose -p consent -f config/docker-compose.yaml up

That bootstraps the elastic superuser with the password from ELASTIC_PASSWORD (default devpassword). Override it with ES_SECURITY_ENABLED=true ELASTIC_PASSWORD=<your-password> docker-compose ... up, and keep elasticSearch.authUser/authPassword in config/consent.yaml in sync.

The HTTP layer stays on plain http in both modes, so consent's elasticSearch.protocol: http needs no change. xpack.security.transport.ssl.enabled=false is correct even with security enabled — transport SSL is only required for multi-node clusters.

Enabling DLS/FLS locally (trial license required)

Enabling security is necessary but not sufficient for DLS/FLS. The Docker image self-generates a basic license, and DLS/FLS is a Platinum/Enterprise feature. With a basic license, creating a role or API key that carries a DLS query or an FLS field_security grant fails closed with HTTP 403 current license is non-compliant for [field and document level security]. Note that POST /_security/api_key accepts such a role descriptor at creation time — the rejection happens later, on the search request.

Activate the 30-day trial license once the secured cluster is up:

curl -u elastic:devpassword -XPOST 'localhost:9200/_license/start_trial?acknowledge=true'
curl -u elastic:devpassword 'localhost:9200/_license'   # => "type": "trial"

Caveats:

  • A trial can be started once per cluster (GET /_license/trial_status reports eligibility). After 30 days the license reverts to basic and DLS/FLS stops working. To get another trial, wipe the cluster's data volume: docker-compose -p consent -f config/docker-compose.yaml down then docker volume rm consent_elastic (this deletes local indices — they must be re-indexed).
  • Authentication, role-based access control, and API keys all work fine on the basic license. Only the DLS/FLS grants require trial/Platinum.

Upgrading the local Elastic Search version

Everything described above — the two security modes, the license gating, and that an unenforceable DLS/FLS grant fails closed instead of returning unrestricted data — is asserted by tests, so an upgrade does not need to be re-verified by hand:

# 1. bump ElasticSearchTestCluster.IMAGE (the only version pin in the test tree)
./mvnw test -Dgroups=elasticsearch
# 2. then match it here: config/docker-compose.yaml, pom.xml (elasticsearch-rest-client), this file

These four classes carry the JUnit tag elasticsearch and are not run by CI — see "How they run in CI" below. Running them is a deliberate step when changing the Elasticsearch version or the Epic D security work, not something a build does for you.

See "Qualifying a new Elasticsearch version" in the integration test README.

How To...

How to add a new email

DUOS uses SendGrid for sending emails and FreeMarker for expanding email templates before passing them to SendGrid. To add a new email, follow these steps: 1. 2. Create the HTML template and add it to src/main/resources/freemarker. 2. Add a new entry to EmailType that references that file. 3. Add a new MailMessage subclass that references this EmailType and implements methods to provide data for the freemarker template and database operation 4. Add a method to EmailService that creates this message and passes it to sendMessage().

How to scan with OWASP

Consent code includes a maven plugin for the OWASP dependency check tool. Three environment variables are needed to use this. You will need the dotNet 8 SDK to generate a report.

Install the dotNet 8 SDK with:

$ brew install --cask dotnet-sdk@8

Register at the National Vulnerability Database for an API key here: https://nvd.nist.gov/developers/request-an-api-key

Assign the key value they issue you as environment variable NVD_API_KEY

e.g.

$ export NVD_API_KEY=<your api key here>

Register at the OSS Index for an API key: https://ossindex.sonatype.org/

Assign the username you used (email address) to the environment variable OSS_INDEX_USERNAME

e.g.

$ export OSS_INDEX_USERNAME=<your username>

Assign the API Token (visible on the User Settings page) to the environment variable OSS_INDEX_PASSWORD

e.g.

$ export OSS_INDEX_PASSWORD=<your API token>

Run the dependency checker: $ mvn org.owasp:dependency-check-maven:check

Integration Testing

Integration tests live in src/test/java/**/integration/ and are run as part of the standard mvn test lifecycle — no special profile, external server, or manual Postgres setup is required.

How they work

Each test class extends ContainerTests, which uses a JUnit 5 DropwizardAppExtension to boot the full application in-process against the config at src/test/resources/consent-ci.yaml. A WireMock server on port 9999 stands in for all external services (Sam, ECM, GCS, etc.).

Database seeding is performed programmatically in ContainerTests.seedDatabase() via typed DAO calls (@BeforeAll). The seed data is fully synthetic and idempotent. To add new baseline rows, extend the relevant seed* helper method inside ContainerTests.

Database

ContainerTests starts its own Testcontainers PostgreSQLContainer in a static initializer and passes the container's coordinates directly to DropwizardAppExtension via ConfigOverride. The hardcoded coordinates in consent-ci.yaml are never reached at runtime. No local Postgres is needed in any environment.

How they run in CI

The GitHub Actions workflow at .github/workflows/coverage.yaml runs mvn clean test on every push/PR to develop, which exercises unit and integration tests together via Testcontainers — no additional CI configuration is needed.

The exception is the Elasticsearch container tests. They are tagged elasticsearch, and the ci profile in pom.xml — activated by the CI=true that GitHub Actions sets in every job — feeds that tag to surefire's excludedGroups, so they do not run in CI. They start up to three Elasticsearch containers and exist to qualify a local Elasticsearch version rather than to gate the build; see "Upgrading the local Elastic Search version" above.

To run them in CI anyway (a workflow edit, or a workflow_dispatch job), pass an empty value on the command line, which overrides the profile:

./mvnw clean jacoco:prepare-agent test jacoco:report -DexcludedTestGroups=

Running integration tests locally

Integration tests only:

mvn clean test -Dtest="org.broadinstitute.consent.integration.**"

All tests (unit + integration together, as CI does):

mvn clean test

Note that locally this also runs the Elasticsearch container tests, which CI skips. To match CI exactly, add -DexcludedTestGroups=elasticsearch.

From the IDE: run or debug any test class in the integration package directly — DAOTestHelper activates automatically and provides the database.