Integrate Synthea with PyHealth

.gitignore

@@ -154,4 +154,8 @@ test-resources/meds_demo/
 .venv312/
 
 # Tool caches
-.ruff_cache/
+.ruff_cache/
+
+# Generated Synthea outputs and dataset caches
+.artifacts/
+examples/.artifacts/

docs/api/datasets.rst

@@ -288,6 +288,7 @@ Available Datasets
     datasets/pyhealth.datasets.ISRUCDataset
     datasets/pyhealth.datasets.MIMICExtractDataset
     datasets/pyhealth.datasets.OMOPDataset
+    datasets/pyhealth.datasets.SyntheaCSVDataset
     datasets/pyhealth.datasets.DREAMTDataset
     datasets/pyhealth.datasets.SHHSDataset
     datasets/pyhealth.datasets.SleepEDFDataset

docs/api/datasets/pyhealth.datasets.SyntheaCSVDataset.rst

@@ -0,0 +1,169 @@
+pyhealth.datasets.SyntheaCSVDataset
+===================================
+
+Loads the CSV export of `Synthea <https://github.com/synthetichealth/synthea>`_,
+a synthetic patient simulator, through the standard PyHealth dataset API.
+Synthea data is fully synthetic, so it needs no credentialing and is useful for
+prototyping pipelines before you have access to MIMIC or eICU.
+
+Getting Data
+------------
+
+``root`` is a directory of Synthea CSV files. There are two ways to get one:
+
+- **Generate it** with :class:`~pyhealth.models.Synthea` (requires Java 17+;
+  see its Setup section). This lets you choose the population size, seed,
+  location, and age range.
+- **Download a pre-generated sample** from the
+  `Synthea downloads page <https://synthea.mitre.org/downloads>`_ and unzip it.
+  No Java is needed; pick the CSV variant.
+
+.. code-block:: python
+
+    from pyhealth.datasets import SyntheaCSVDataset
+    from pyhealth.models import Synthea
+
+    csv_dir = Synthea("./synthea-output").generate(population=100, seed=42)
+    # or: csv_dir = "/path/to/unzipped/sample/csv"  # directory with patients.csv
+
+    dataset = SyntheaCSVDataset(
+        csv_dir,
+        tables=["conditions", "medications", "observations"],
+    )
+    dataset.stats()
+
+How Synthea Differs From Other PyHealth Datasets
+------------------------------------------------
+
+**What Synthea writes is very different from MIMIC; what PyHealth gives you
+back is the same.** The differences to watch for are in the raw files and in
+the medical codes, not in the dataset object.
+
+Raw output: many wide tables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A Synthea run writes about 18 CSV files. Each has many columns, every row is
+keyed by a UUID, and the files include synthetic billing and insurance data.
+For example, the first rows of ``conditions.csv`` and ``observations.csv``:
+
+.. code-block:: text
+
+    START,STOP,PATIENT,ENCOUNTER,SYSTEM,CODE,DESCRIPTION
+    2004-07-17,,ba419d35-...-eebf02485a56,ba419d35-...-4d3d0f870810,SNOMED-CT,224299000,Received higher education (finding)
+
+    DATE,PATIENT,ENCOUNTER,CATEGORY,CODE,DESCRIPTION,VALUE,UNITS,TYPE
+    2023-08-12T20:03:25Z,ba419d35-...-eebf02485a56,ba419d35-...-b4779d5cf14b,laboratory,4548-4,Hemoglobin A1c/Hemoglobin.total in Blood,6.3,%,numeric
+
+Not every file can be loaded, because PyHealth organizes events by patient:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 70
+
+   * - Files
+     - Loaded?
+   * - ``patients``, ``encounters``, ``conditions``, ``medications``,
+       ``observations``, ``procedures``, ``immunizations``, ``careplans``,
+       ``allergies``, ``devices``, ``imaging_studies``, ``supplies``,
+       ``payer_transitions``
+     - Yes, by default. Files Synthea did not emit (no rows) are skipped.
+   * - ``claims``, ``claims_transactions``
+     - Only when listed in ``tables``. They are large and mostly billing
+       detail.
+   * - ``organizations``, ``providers``, ``payers``
+     - No. They describe institutions, not patients, so they have no patient
+       column.
+
+After loading: the standard PyHealth event table
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Like every PyHealth dataset, ``SyntheaCSVDataset`` turns each row of each table
+into one **event** with three fixed columns, ``patient_id``, ``event_type`` (the
+table name), and ``timestamp``, plus one ``<table>/<column>`` column per kept
+attribute. Columns belonging to other tables are null for that event:
+
+.. code-block:: text
+
+    patient_id     event_type    timestamp            conditions/code  conditions/description  observations/code  observations/value  ...
+    ba419d35-...   conditions    2004-07-17 00:00:00  224299000        Received higher ...     null               null
+    ba419d35-...   observations  2023-08-12 20:03:25  null             null                    4548-4             6.3
+
+Through the patient API, the table prefix is dropped:
+
+.. code-block:: python
+
+    patient = dataset.get_patient(dataset.unique_patient_ids[0])
+    for event in patient.get_events(event_type="conditions"):
+        print(event.timestamp, event.code, event.description)
+
+    numeric_obs = patient.get_events(
+        event_type="observations",
+        filters=[("type", "==", "numeric")],
+    )
+
+The columns kept per table are listed in
+``pyhealth/datasets/configs/synthea_csv.yaml``. Columns that identify a person
+(name, SSN, street address, coordinates) are not loaded.
+
+Medical codes: SNOMED-CT, RxNorm, LOINC, CVX
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This difference matters most. MIMIC codes diagnoses with ICD-9/ICD-10 and drugs
+with NDC; Synthea uses different vocabularies:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 25 45
+
+   * - Table
+     - Code system
+     - Example ``code``
+   * - ``conditions``
+     - SNOMED-CT
+     - ``714628002`` (Prediabetes)
+   * - ``procedures``, ``encounters``, ``careplans``, ``devices``,
+       ``supplies``
+     - SNOMED-CT
+     - ``252160004`` (Standard pregnancy test)
+   * - ``medications``
+     - RxNorm
+     - ``849574`` (Naproxen sodium 220 MG Oral Tablet)
+   * - ``observations``
+     - LOINC
+     - ``4548-4`` (Hemoglobin A1c)
+   * - ``immunizations``
+     - CVX
+     - ``140`` (Influenza, split virus, trivalent, PF)
+   * - ``allergies``
+     - Mostly SNOMED-CT; drug allergies use RxNorm. The ``system`` column is
+       often ``Unknown``, so do not rely on it.
+     - ``84489001`` (Mold)
+
+As a result:
+
+- Built-in tasks written for MIMIC (for example
+  :class:`~pyhealth.tasks.MortalityPredictionMIMIC3`) read ICD columns and do
+  not work on Synthea. Write a task for Synthea instead (see :doc:`../tasks`).
+- ICD-based code mappings in :mod:`pyhealth.medcode` (such as ICD to CCS) do
+  not apply. RxNorm codes are supported by ``pyhealth.medcode.RxNorm``.
+- Every code comes with a human-readable ``description`` column.
+
+Other differences
+^^^^^^^^^^^^^^^^^
+
+- **Each patient has a ``patients`` event at their birth date.** Demographics
+  (``gender``, ``race``, ``deathdate``, ...) are attributes of that event.
+- **Visits are not a separate level.** Each clinical event has an
+  ``encounter`` attribute holding the UUID of its encounter; group events by it
+  to rebuild visits.
+- **Lab values are strings.** ``observations/value`` holds numbers and text
+  answers in one column; ``observations/type`` (``numeric`` or ``text``) says
+  which.
+- **Timestamps mix dates and date-times across tables** (for example
+  ``conditions`` has dates, ``observations`` has date-times). PyHealth parses
+  both; date-only events get a midnight timestamp.
+
+.. autoclass:: pyhealth.datasets.SyntheaCSVDataset
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/api/models.rst

@@ -245,6 +245,7 @@ API Reference
     models/pyhealth.models.PromptEHR
     models/pyhealth.models.MedGAN
     models/pyhealth.models.CorGAN
+    models/pyhealth.models.Synthea
     models/pyhealth.models.SDOH
     models/pyhealth.models.VisionEmbeddingModel
     models/pyhealth.models.TextEmbedding

docs/api/models/pyhealth.models.Synthea.rst

@@ -0,0 +1,154 @@
+pyhealth.models.Synthea
+===================================
+
+`Synthea <https://github.com/synthetichealth/synthea>`_ is a rule-based
+synthetic patient simulator from MITRE. Unlike the trained generators (HALO,
+MedGAN, ...), Synthea is not fitted to data; it simulates patient lifetimes from
+clinical care modules. This class runs the pinned Synthea release as a Java
+subprocess and writes CSV files to disk. It subclasses
+:class:`~pyhealth.models.BaseModel` for API consistency but has no trainable
+parameters and no forward pass; load its output with
+:class:`~pyhealth.datasets.SyntheaCSVDataset`.
+
+Quick Start
+-----------
+
+The constructor only sets up the generator: where output goes, and where to
+find Java and the Synthea JAR. Nothing runs until
+:meth:`~pyhealth.models.Synthea.generate`, which takes the population settings
+and returns the directory of CSV files that
+:class:`~pyhealth.datasets.SyntheaCSVDataset` takes as ``root``.
+
+.. code-block:: python
+
+    from pyhealth.datasets import SyntheaCSVDataset
+    from pyhealth.models import Synthea
+
+    synthea = Synthea("./synthea-output")  # setup only; nothing runs yet
+
+    csv_dir = synthea.generate(population=100, seed=42)  # runs Java
+    dataset = SyntheaCSVDataset(csv_dir, tables=["conditions", "medications"])
+    dataset.stats()
+
+    # The same instance can generate as many populations as you like.
+    larger = synthea.generate(population=1000, seed=42)
+    ohio = synthea.generate(population=100, seed=42, state="Ohio")
+
+Each combination of settings is written to its own subdirectory of
+``output_dir``, so populations never overwrite each other::
+
+    synthea-output/
+    └── 3f9c...e1/            # SHA-256 of the generation settings
+        └── csv/
+            ├── patients.csv
+            ├── conditions.csv
+            └── ...
+
+Calling ``generate()`` again with the same settings returns the existing
+directory without rerunning Synthea, which can take minutes. Pass
+``overwrite=True`` to force a fresh run. If you omit ``seed``, a random seed is
+chosen and logged, so every call produces a new population; pass a seed when
+you need reproducible output.
+
+See :doc:`../datasets/pyhealth.datasets.SyntheaCSVDataset` for what the CSV
+files contain and how they differ from other PyHealth datasets.
+
+Setup
+-----
+
+You do **not** need to download or build Synthea yourself. The only manual
+step is installing Java.
+
+**1. Install Java 17 or newer.** ``pip install pyhealth`` does not provide it.
+Any JDK distribution works, for example:
+
+- conda: ``conda install -c conda-forge openjdk``
+- macOS (Homebrew): ``brew install openjdk``
+- Ubuntu/Debian: ``sudo apt install openjdk-17-jre-headless``
+- Windows or any platform: an installer from
+  `Eclipse Temurin <https://adoptium.net>`_
+
+Check the installation with ``java -version``; the first line should report
+version 17 or higher.
+
+**2. Generate.** On the first call to :meth:`~pyhealth.models.Synthea.generate`,
+PyHealth downloads the pinned Synthea release JAR (v4.0.0, about 200 MB),
+verifies its SHA-256 checksum, and caches it under the PyHealth cache
+directory:
+
+- Linux: ``~/.cache/pyhealth/synthea/``
+- macOS: ``~/Library/Caches/pyhealth/synthea/``
+- Windows: ``%LOCALAPPDATA%\pyhealth\pyhealth\Cache\synthea\``
+
+Later runs reuse the cached JAR.
+
+**Offline machines or a custom Synthea build.** Download
+``synthea-with-dependencies.jar`` from the
+`Synthea releases page <https://github.com/synthetichealth/synthea/releases>`_
+on a machine with internet access, copy it over, and pass its path:
+
+.. code-block:: python
+
+    synthea = Synthea(
+        "./synthea-output",
+        jar_path="/path/to/synthea-with-dependencies.jar",
+        auto_download=False,
+    )
+
+**How Java is found.** The first of these that exists is used:
+
+1. the ``java_path`` argument,
+2. ``$JAVA_HOME/bin/java``,
+3. ``java`` on ``PATH``.
+
+If none is found, or it is older than Java 17,
+:meth:`~pyhealth.models.Synthea.generate` raises a ``RuntimeError`` naming the
+executable and its version. Constructing the class and calling
+:meth:`~pyhealth.models.Synthea.build_argv` do not need Java.
+
+Choosing What to Generate
+-------------------------
+
+Synthea's command-line options are arguments of
+:meth:`~pyhealth.models.Synthea.generate` (``population``, ``seed``,
+``state``, ``city``, ``gender``, ``age_range``, ``reference_date``, ...).
+Anything from Synthea's
+`synthea.properties <https://github.com/synthetichealth/synthea/wiki/Common-Configuration>`_
+file goes in ``synthea_config``. Use
+:meth:`~pyhealth.models.Synthea.get_available_config` to list the properties
+the JAR accepts.
+
+.. code-block:: python
+
+    csv_dir = synthea.generate(
+        population=500,
+        seed=7,
+        state="Massachusetts",
+        age_range="40-80",
+        synthea_config={
+            "generate.only_alive_patients": True,
+            "exporter.years_of_history": 10,
+        },
+    )
+
+CSV export is switched on automatically and other export formats (FHIR, C-CDA,
+...) are switched off. ``synthea_config`` cannot change which exporters run or
+where output is written; use ``output_dir`` for that.
+
+To check the exact command without running anything (no Java needed), call
+:meth:`~pyhealth.models.Synthea.build_argv` with the same settings::
+
+    synthea.build_argv("java", "synthea.jar", population=500, seed=7)
+
+Reference:
+    Walonoski, J., Kramer, M., Nichols, J., Quina, A., Moesel, C., Hall, D.,
+    Duffett, C., Dube, K., Gallagher, T., & McLachlan, S. (2018).
+    *Synthea: An approach, method, and software mechanism for generating
+    synthetic patients and the synthetic electronic health care record.*
+    Journal of the American Medical Informatics Association, 25(3), 230-238.
+    https://doi.org/10.1093/jamia/ocx079
+
+.. autoclass:: pyhealth.models.Synthea
+    :members:
+    :undoc-members:
+    :show-inheritance: