Logging

Databucket writes its log to the standard output. Locally it is plain text; on a server it can write ECS JSON (Elastic Common Schema, one JSON object per line), which Elasticsearch and Kibana understand without any further mapping.

What is logged

• Access log - one line per /api request: method, path, status, duration and client address, e.g. POST /api/public/signin 401 100ms from 10.0.0.7. Requests to bucket data (/api/bucket/**: rows, reservations, history), which make most of the traffic, have their own logger.
• Operations - one line per significant change, written after the change is committed, e.g. data.reserve: bucket=users count=4 owner=tester. Each has an action name in event.action:
  • auth.* - sign-in (auth.signin, auth.signin-failed, auth.sso-login, auth.sso-signin-failed, ...), sign-up, password change and reset, change of the project,
  • data.* - data.create, data.modify, data.delete, data.reserve, data.history-clear,
  • project.* - create, modify, delete, clear, dataset import and export,
  • user.*, and create / modify / delete of the project settings: bucket.*, class.*, columns.*, enum.*, filter.*, group.*, svg.*, tag.*, task.*, team.*, view.*.
• Errors and the rest of the application log.

Every line written while serving a request carries the request id. The same id is returned to the client in the X-Request-Id header, and server errors of the API name it in their message, so a reported error can be found in the log.

Fields

In the ECS JSON log the context of a line is in separate fields:

Field	Meaning
http.request.id	id of the request (also the X-Request-Id response header)
user.name	the signed-in user
databucket.project.id	the project of the signed-in user
event.action	the operation, e.g. data.reserve
http.request.method, url.path	method and path of the request (access log)
http.response.status_code	response status (access log)
event.duration	duration of the request in nanoseconds, as ECS defines it (access log)
client.ip	client address; behind a proxy or ingress taken from the X-Forwarded-For header
databucket.filter.format	the filter format of a data request: logic, rules, rules-raw, conditions
log.level, log.logger, message	level, logger and text of the line
service.name, service.version, service.environment, service.node.name	databucket, its version, the environment, the host (on Kubernetes the pod name)

In the text log the request id, user, project and filter format are shown after the logger name: req=... user=... project=... filter=....

Configuration

Set with environment variables (or the same keys in the configuration file):

Variable	Meaning
LOGGING_STRUCTURED_FORMAT_CONSOLE=ecs	ECS JSON instead of text
LOGGING_STRUCTURED_ECS_SERVICE_ENVIRONMENT=prd	the environment in service.environment
LOGGING_LEVEL_<LOGGER>=<level>	the level of a logger, e.g. LOGGING_LEVEL_PL_DATABUCKET_SERVER_ACCESS_DATA=warn

Loggers:

Logger	Default	What
pl.databucket.server.access	info	access log of the API
pl.databucket.server.access.data	info	access log of bucket data, most of the traffic - turn it to warn when it is too much
pl.databucket.server.operation	info	operations; one logger per area: operation.auth, operation.data, operation.bucket, ...
pl.databucket.server.service.data	info	at debug: the SQL built for bucket data with its parameters, which include data values - don't leave it on in production

Kubernetes and Elastic

1. Set LOGGING_STRUCTURED_FORMAT_CONSOLE=ecs (and LOGGING_STRUCTURED_ECS_SERVICE_ENVIRONMENT) in the deployment.
2. Let the log collector of the cluster (Filebeat or Elastic Agent, usually a DaemonSet) read the container log of Databucket and parse each line as JSON into the root of the document. With Filebeat it is the ndjson parser:

   parsers:
     - container: ~
     - ndjson:
         target: ""
         overwrite_keys: true
         expand_keys: true
         add_error_key: true

3. In Kibana, create a data view on the index or data stream the collector writes to, with @timestamp as the time field.

Trying it locally

The repository has a Docker Compose setup with Elasticsearch, Kibana and Filebeat next to Databucket (from #186):

cd docker
docker compose -f docker-compose.yaml -f docker-compose.elastic.yaml up --build

Databucket is at http://databucket.localhost, Kibana at http://localhost:5601: open Discover and the data view Databucket logs. Try e.g. event.action : auth.signin-failed or http.response.status_code >= 400, and filter by a http.request.id to see everything one request did. Elasticsearch needs about 1.5 GB of memory; the setup has no security and is meant only for a local look. docker compose ... down -v removes it together with the stored logs.