feat: Add ability to generate more realistic learner journey data

README.rst

@@ -149,6 +149,84 @@ To try out the new UI mode:
 
 
 
+Learner journeys with known results
+-----------------------------------
+The backends above pick each event independently, which is good for volume but
+means nobody knows what the reports *should* show. The ``journeys`` command
+instead simulates each enrolled learner moving through a properly nested course
+(sections > subsections > units > problems and videos) in time-ordered
+sessions, and writes the expected engagement results alongside the data, so
+report numbers can be checked against a known answer (the "oracle", computed in
+``xapi_db_load/journeys/oracle.py``):
+
+::
+
+    ❯ xapi-db-load journeys --config_file example_configs/journeys_oracle.yaml --now "2026-10-08 12:00:00"
+
+The same config, seed and ``--now`` always produce identical files. Example
+configs:
+
+- ``journeys_oracle.yaml``: ~9K events, every edge case turned on (``mailto:``
+  actors, deleted units, duplicate events, pause and resume at the same instant).
+  Use it to check correctness.
+- ``journeys_m.yaml`` / ``journeys_l.yaml``: ~15M / ~75M events for benchmarks.
+
+Output, in ``journeys.output_dir`` or ``--output_dir``:
+
+- ``courses``, ``blocks``, ``external_ids``, ``user_profiles``: event sink rows,
+  in the same CSV layouts as the ``csv`` backend. Each course is published
+  ``course_publishes`` times; deleted units only appear in the earlier publishes.
+- ``xapi``: ``xapi_events_all`` rows (``event_id``, ``emission_time``, ``event``).
+- ``expected_engagement``: one row per learner and section / subsection with
+  pages, problems or videos, with ``done``, ``total`` and the Aspects status
+  label. For videos, ``done`` is what the events can show (each play paired with
+  the next video event) and ``done_truth`` is what the learner actually watched.
+- ``expected_video_seconds``: watched seconds per learner and video, total and
+  distinct, both observable and actual.
+- ``manifest.json``: counts, settings, and the most active learner in each
+  course, for the learner dashboard.
+
+Files are gzipped CSV (the expected results have a header row). To load a
+dataset into an Aspects install, use ``load-journeys`` with a config that has
+the same ClickHouse settings as ``load-db``, plus the ``journeys`` section or
+``--dataset_dir``::
+
+    ❯ xapi-db-load load-journeys --config_file my_config.yaml --dataset_dir logs/journeys_oracle
+
+It inserts the event sink rows straight into ClickHouse (``event_sink_db_name``,
+default ``event_sink``) and sends the xAPI events through ``backend``:
+``clickhouse`` inserts them into ``db_name``.``xapi_events_all``, ``ralph`` POSTs
+them to ``lrs_url`` (without the duplicate events, which an LRS rejects), and
+``vector`` logs them to stdout for Vector to pick up, pausing
+``vector_batch_pause_seconds`` (default 0.1) after every ``vector_batch_size``
+(default 100) events. The expected results go
+into the ``expected_db_name`` database (default ``journeys``). Ralph and Vector
+insert asynchronously, so the command waits, up to ``--wait_seconds``, until
+every event has arrived. To load journeys next to data from ``load-db``, set
+``org_prefix`` and ``first_user_id`` in the ``journeys`` section so the
+organizations and users don't collide with ``load-db``'s ``Org0``... and user
+ids from 0.
+
+To load the files by hand instead, copy the directory under the ClickHouse
+``user_files`` path and insert with the ``file()`` table function, for example::
+
+    insert into xapi.xapi_events_all
+    select * from file('journeys/xapi.csv.gz', 'CSV',
+        'event_id UUID, emission_time DateTime64(6), event String');
+
+The materialized views only see one insert block at a time, so to imitate live
+traffic, set ``sort_events: true`` (this holds every event in memory) and insert
+in small blocks, e.g. ``settings max_block_size = 20000,
+min_insert_block_size_rows = 20000, min_insert_block_size_bytes = 0,
+max_insert_threads = 1, max_threads = 1``.
+
+``window_days`` must stay under 365, because Aspects drops xAPI events older than
+a year. Every ``behavior`` setting is described on the ``Behavior`` class in
+``xapi_db_load/journeys/simulate.py``. Course template sizes are ``[low, high]``
+ranges, except ``problems_per_vertical_weights`` and
+``videos_per_vertical_weights``, which weight how many problems and videos each
+unit gets: ``[0.4, 0.4, 0.2]`` means 40% none, 40% one and 20% two.
+
 Secrets and environment variable overrides
 ------------------------------------------
 Sensitive credentials should not be committed to source control. The following

docs/xapi_db_load.journeys.rst

@@ -0,0 +1,53 @@
+xapi\_db\_load.journeys package
+===============================
+
+Submodules
+----------
+
+xapi\_db\_load.journeys.generate module
+---------------------------------------
+
+.. automodule:: xapi_db_load.journeys.generate
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+xapi\_db\_load.journeys.oracle module
+-------------------------------------
+
+.. automodule:: xapi_db_load.journeys.oracle
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+xapi\_db\_load.journeys.simulate module
+---------------------------------------
+
+.. automodule:: xapi_db_load.journeys.simulate
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+xapi\_db\_load.journeys.statements module
+-----------------------------------------
+
+.. automodule:: xapi_db_load.journeys.statements
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+xapi\_db\_load.journeys.structure module
+----------------------------------------
+
+.. automodule:: xapi_db_load.journeys.structure
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
+Module contents
+---------------
+
+.. automodule:: xapi_db_load.journeys
+   :members:
+   :show-inheritance:
+   :undoc-members:

docs/xapi_db_load.rst

@@ -9,6 +9,7 @@ Subpackages
 
    xapi_db_load.backends
    xapi_db_load.fixtures
+   xapi_db_load.journeys
    xapi_db_load.tests
    xapi_db_load.ui
    xapi_db_load.xapi

docs/xapi_db_load.tests.rst

@@ -28,6 +28,14 @@ xapi\_db\_load.tests.test\_event\_generator module
    :show-inheritance:
    :undoc-members:
 
+xapi\_db\_load.tests.test\_journeys module
+------------------------------------------
+
+.. automodule:: xapi_db_load.tests.test_journeys
+   :members:
+   :show-inheritance:
+   :undoc-members:
+
 xapi\_db\_load.tests.test\_ui module
 ------------------------------------
 

example_configs/journeys_l.yaml

@@ -0,0 +1,51 @@
+# Large learner-journey dataset (~75M xAPI events, 100k learners) for benchmarks.
+# Generate: xapi-db-load journeys --config_file example_configs/journeys_l.yaml
+lms_url: http://localhost:18000
+
+journeys:
+  seed: 2027
+  output_dir: logs/journeys_l
+  now: null
+  window_days: 330
+  running_course_fraction: 0.4
+  course_length_days: 120
+  num_organizations: 5
+  num_actors: 100000
+  mbox_actor_fraction: 0.05
+  num_actor_profile_changes: 2
+  course_publishes: 5
+  deleted_unit_course_fraction: 0.1
+  sort_events: false
+  courses:
+    - {template: small, count: 300, runs: [1, 3], learners: [50, 300]}
+    - {template: medium, count: 200, runs: [1, 3], learners: [200, 800]}
+    - {template: large, count: 75, runs: [1, 2], learners: [300, 1100]}
+  course_templates:
+    small:
+      chapters: [3, 5]
+      sequentials_per_chapter: [2, 4]
+      verticals_per_sequential: [2, 5]
+      problems_per_vertical_weights: [0.5, 0.3, 0.2]
+      videos_per_vertical_weights: [0.5, 0.45, 0.05]
+      graded_sequential_fraction: 0.3
+      video_length: [60, 600]
+    medium:
+      chapters: [5, 8]
+      sequentials_per_chapter: [3, 5]
+      verticals_per_sequential: [3, 6]
+      problems_per_vertical_weights: [0.4, 0.35, 0.2, 0.05]
+      videos_per_vertical_weights: [0.45, 0.5, 0.05]
+      graded_sequential_fraction: 0.3
+      video_length: [60, 900]
+    large:
+      chapters: [8, 14]
+      sequentials_per_chapter: [3, 6]
+      verticals_per_sequential: [3, 8]
+      problems_per_vertical_weights: [0.35, 0.35, 0.2, 0.1]
+      videos_per_vertical_weights: [0.4, 0.55, 0.05]
+      graded_sequential_fraction: 0.3
+      video_length: [60, 1200]
+  behavior:
+    completer_fraction: 0.1
+    continue_prob: 0.9
+    duplicate_event_prob: 0.001

example_configs/journeys_m.yaml

@@ -0,0 +1,51 @@
+# Medium learner-journey dataset (~15M xAPI events, 25k learners) for benchmarks.
+# Generate: xapi-db-load journeys --config_file example_configs/journeys_m.yaml
+lms_url: http://localhost:18000
+
+journeys:
+  seed: 2026
+  output_dir: logs/journeys_m
+  now: null
+  window_days: 330
+  running_course_fraction: 0.4
+  course_length_days: 120
+  num_organizations: 5
+  num_actors: 25000
+  mbox_actor_fraction: 0.05
+  num_actor_profile_changes: 2
+  course_publishes: 5
+  deleted_unit_course_fraction: 0.1
+  sort_events: false
+  courses:
+    - {template: small, count: 60, runs: [1, 3], learners: [50, 300]}
+    - {template: medium, count: 40, runs: [1, 3], learners: [200, 800]}
+    - {template: large, count: 15, runs: [1, 2], learners: [300, 1100]}
+  course_templates:
+    small:
+      chapters: [3, 5]
+      sequentials_per_chapter: [2, 4]
+      verticals_per_sequential: [2, 5]
+      problems_per_vertical_weights: [0.5, 0.3, 0.2]
+      videos_per_vertical_weights: [0.5, 0.45, 0.05]
+      graded_sequential_fraction: 0.3
+      video_length: [60, 600]
+    medium:
+      chapters: [5, 8]
+      sequentials_per_chapter: [3, 5]
+      verticals_per_sequential: [3, 6]
+      problems_per_vertical_weights: [0.4, 0.35, 0.2, 0.05]
+      videos_per_vertical_weights: [0.45, 0.5, 0.05]
+      graded_sequential_fraction: 0.3
+      video_length: [60, 900]
+    large:
+      chapters: [8, 14]
+      sequentials_per_chapter: [3, 6]
+      verticals_per_sequential: [3, 8]
+      problems_per_vertical_weights: [0.35, 0.35, 0.2, 0.1]
+      videos_per_vertical_weights: [0.4, 0.55, 0.05]
+      graded_sequential_fraction: 0.3
+      video_length: [60, 1200]
+  behavior:
+    completer_fraction: 0.1
+    continue_prob: 0.9
+    duplicate_event_prob: 0.001

example_configs/journeys_oracle.yaml

@@ -0,0 +1,49 @@
+# Small, deterministic learner-journey dataset with every edge case turned on.
+# Generate: xapi-db-load journeys --config_file example_configs/journeys_oracle.yaml
+# The expected_*.csv.gz files hold the known-correct engagement results.
+lms_url: http://localhost:18000
+
+journeys:
+  seed: 1335
+  output_dir: logs/journeys_oracle
+  # Pin "now" so the dataset is identical on every run. Keep it within a day of the time the data
+  # is loaded if you want some events inside refreshable-view lookback windows.
+  now: null
+  window_days: 200
+  running_course_fraction: 0.6
+  course_length_days: 90
+  num_organizations: 2
+  num_actors: 60
+  mbox_actor_fraction: 0.15
+  num_actor_profile_changes: 3
+  course_publishes: 3
+  deleted_unit_course_fraction: 0.5
+  # Order xapi.csv by time across all learners, as live traffic arrives, so watch sessions get
+  # split across insert batches when loaded in small blocks.
+  sort_events: true
+  courses:
+    - {template: small, count: 2, runs: [1, 2], learners: [15, 30]}
+    - {template: medium, count: 2, runs: [1, 1], learners: [20, 40]}
+  course_templates:
+    small:
+      chapters: [2, 3]
+      sequentials_per_chapter: [1, 3]
+      verticals_per_sequential: [1, 4]
+      problems_per_vertical_weights: [0.5, 0.3, 0.2]
+      videos_per_vertical_weights: [0.5, 0.4, 0.1]
+      graded_sequential_fraction: 0.3
+      video_length: [30, 240]
+    medium:
+      chapters: [3, 5]
+      sequentials_per_chapter: [2, 4]
+      verticals_per_sequential: [2, 5]
+      problems_per_vertical_weights: [0.4, 0.4, 0.2]
+      videos_per_vertical_weights: [0.5, 0.4, 0.1]
+      graded_sequential_fraction: 0.3
+      video_length: [30, 300]
+  behavior:
+    completer_fraction: 0.25
+    continue_prob: 0.85
+    same_time_resume_prob: 0.2
+    duplicate_event_prob: 0.02
+    recent_activity_fraction: 0.15

xapi_db_load/journeys/__init__.py

@@ -0,0 +1,9 @@
+"""
+Learner-journey data generation.
+
+Unlike the random event mode, which picks every event independently, this mode simulates each
+enrolled learner moving through a properly nested course: navigating units in order, attempting
+the problems and watching the videos in them, in time-ordered sessions. Because the generator knows
+exactly what each learner did, it also writes the expected engagement results, so reports can be
+checked against a known answer.
+"""