feat(elasticsearch): reenable

artifacts/docs/README.md

@@ -68,6 +68,7 @@ You can check the database connector's setup logs using `portainer`.
 ### elasticsearch
 
 You cannot access the search engine via a web interface.
+See `docs/elasticsearch.md` for how the helpdesk trusts this service's certificate and when that certificate has to be replaced.
 
 ### geoip
 

docs/elasticsearch.md

@@ -0,0 +1,55 @@
+# Elasticsearch certificate
+
+The helpdesk talks to the search engine over HTTPS, and the certificate securing that connection is one the search engine issued to itself.
+Nothing in this repository creates it and no certificate authority outside the stack vouches for it, so the helpdesk only accepts it because the authority's certificate was imported into the helpdesk by hand.
+This document records where that certificate comes from, how it got into the helpdesk, and when it has to be replaced.
+
+## Where the certificate comes from
+
+On the first start of a node, the search engine runs its own security auto-configuration.
+It generates a certificate authority and a server certificate, writes both into `config/certs/` inside the container, and enables HTTPS.
+That directory is the `elasticsearch-configuration` volume, so the files survive redeployments.
+
+Auto-configuration only ever runs once.
+It skips itself as soon as `xpack.security.enabled` is present in `config/elasticsearch.yml`, which is exactly what its own first run writes there.
+Everything below therefore describes a one-time setup.
+
+The name the helpdesk connects to has to appear in the server certificate, otherwise verification fails on a hostname mismatch even when the authority is trusted.
+Auto-configuration copies the value of `network.publish_host` into the certificate for this reason, which is why `src/development/elasticsearch/compose.yaml` sets it to `elasticsearch`.
+
+## How the helpdesk trusts it
+
+Read the authority's certificate out of the running container:
+
+```sh
+docker exec "$(docker ps -q -f name=vibetype_elasticsearch)" \
+  cat /usr/share/elasticsearch/config/certs/http_ca.crt
+```
+
+Then add it in the helpdesk under **Settings > Security > SSL Certificates**, either by uploading the file or by pasting its contents including the `-----BEGIN CERTIFICATE-----` delimiters.
+The console equivalent is `rails r 'SSLCertificate.create!(certificate: STDIN.read)'`.
+
+The helpdesk verifies the connection by default, controlled by its `es_ssl_verify` setting.
+Turning that setting off instead of importing the certificate is possible but is not how this stack is set up.
+
+## When it has to be replaced
+
+Auto-configuration issues the authority for three years and the server certificate for two, and it renews neither.
+The certificates currently in production were generated when the search engine first started, around February 2026:
+
+| Certificate | Expires |
+| --- | --- |
+| Server certificate | around February 2028 |
+| Certificate authority | around February 2029 |
+
+Once the server certificate lapses, the helpdesk stops reaching the search engine and search results silently go stale.
+Check the authority's dates with the command below, and subtract a year for the server certificate.
+The image ships no `openssl`, so the certificate is piped out to the one on the node:
+
+```sh
+docker exec "$(docker ps -q -f name=vibetype_elasticsearch)" \
+  cat /usr/share/elasticsearch/config/certs/http_ca.crt | openssl x509 -noout -dates
+```
+
+Replacing them means removing the `elasticsearch-configuration` volume so auto-configuration runs again, then repeating the import above with the newly generated authority.
+The old certificate in the helpdesk should be deleted at that point, since it no longer matches anything.

src/development/elasticsearch/compose.yaml

@@ -8,23 +8,26 @@ secrets:
 services:
   elasticsearch:
     # You cannot access the search engine via a web interface.
+    # See `docs/elasticsearch.md` for how the helpdesk trusts this service's certificate and when that certificate has to be replaced.
     deploy:
       labels:
         - dargstack.profiles=zammad
-      replicas: 0 # TODO: reenable once server setup is more fault tolerant
     environment:
       bootstrap.memory_lock: "true"
       discovery.type: single-node
       ELASTIC_PASSWORD_FILE: /run/secrets/elasticsearch-password
       ES_JAVA_OPTS: -Xms1g -Xmx1g
+      # The helpdesk sends bulk requests that exceed the 100mb default while it rebuilds its search index.
+      http.max_content_length: 400mb
       KEYSTORE_PASSWORD_FILE: /run/secrets/elasticsearch-keystore-password
+      # The automatic security configuration puts this name into the certificate it generates, which is what lets the helpdesk verify the connection.
       network.publish_host: elasticsearch
-    # healthcheck:
-    #   test: ["CMD-SHELL", "NETRC=$(mktemp) && printf 'machine localhost\\nlogin elastic\\npassword %s\\n' \"$(cat /run/secrets/elasticsearch-password)\" > \"$NETRC\" && curl -fsS --netrc-file \"$NETRC\" --cacert /usr/share/elasticsearch/config/certs/http_ca.crt 'https://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s' && rm -f \"$NETRC\" || { rm -f \"$NETRC\"; exit 1; }"]
-    #   interval: 30s
-    #   timeout: 10s
-    #   retries: 3
-    #   start_period: 60s
+    healthcheck:
+      test: ["CMD-SHELL", "NETRC=$(mktemp) && printf 'machine localhost\\nlogin elastic\\npassword %s\\n' \"$(cat /run/secrets/elasticsearch-password)\" > \"$NETRC\" && curl -fsS --netrc-file \"$NETRC\" --cacert /usr/share/elasticsearch/config/certs/http_ca.crt 'https://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s' && rm -f \"$NETRC\" || { rm -f \"$NETRC\"; exit 1; }"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
     image: elasticsearch:9.5.3
     secrets:
       - source: elasticsearch-keystore-password

src/development/zammad/compose.yaml

@@ -5,7 +5,7 @@ x-shared:
       labels:
         - dargstack.profiles=zammad
     environment: &zammad-environment
-      ELASTICSEARCH_ENABLED: "false"
+      ELASTICSEARCH_ENABLED: "true"
       ELASTICSEARCH_HOST: elasticsearch
       ELASTICSEARCH_SCHEMA: https
       ELASTICSEARCH_USER: elastic