diff --git a/skills/README.md b/skills/README.md new file mode 100644 index 000000000..2db571ef5 --- /dev/null +++ b/skills/README.md @@ -0,0 +1,14 @@ +# dpdata agent skills + +Start with [`dpdata-workflows`](dpdata-workflows/SKILL.md) to route a request by +user intent. + +- [`dpdata-convert`](dpdata-convert/SKILL.md) converts producer outputs and + checks the labeled-data contract. +- [`dpdata-inspect`](dpdata-inspect/SKILL.md) inspects and validates systems, + including the deterministic validator in `scripts/`. +- [`dpdata-extend`](dpdata-extend/SKILL.md) groups developer references for + format plugins, drivers, and minimizers. + +The historical `dpdata-cli`, `dpdata-driver`, `dpdata-minimizer`, and +`dpdata-plugin` directories remain compatibility entry points. diff --git a/skills/dpdata-cli/README.md b/skills/dpdata-cli/README.md index 3c28e020d..0ceb31f30 100644 --- a/skills/dpdata-cli/README.md +++ b/skills/dpdata-cli/README.md @@ -1,71 +1,6 @@ -# dpdata CLI Agent Skill +# dpdata CLI skill (compatibility entry point) -An Agent Skill for using [dpdata](https://github.com/deepmodeling/dpdata) CLI to convert and manipulate atomic simulation data formats. - -## Installation - -To install this skill, provide the skill's GitHub URL to your agent (e.g., OpenClaw): - -```text -Install the dpdata-cli skill from https://github.com/deepmodeling/dpdata/tree/devel/skills/dpdata-cli -``` - -Or manually copy to your agent's skills directory: - -```bash -cp -r skills/dpdata-cli /path/to/your/agent/skills/ -``` - -## Usage - -Once installed, ask your AI agent to work with dpdata: - -```text -Convert my VASP OUTCAR to DeePMD-kit format -``` - -```text -Convert LAMMPS dump file to VASP POSCAR -``` - -## Features - -- **Format Conversion**: Convert between 50+ DFT/MD formats -- **DeePMD-kit Support**: Prepare training data for machine learning potentials -- **Auto-detection**: Automatic format detection for common file types -- **Multi-system Support**: Handle directories with multiple systems - -## Supported Formats - -Formats may be updated. See [Formats Reference (stable)](https://docs.deepmodeling.com/projects/dpdata/en/stable/formats.html) for the latest list. - -### DeePMD-kit - -`deepmd/raw`, `deepmd/comp`, `deepmd/npy`, `deepmd/npy/mixed`, `deepmd/hdf5` - -### VASP - -`vasp/poscar`, `vasp/contcar`, `vasp/outcar`, `vasp/xml` - -### LAMMPS - -`lammps/lmp`, `lammps/dump` - -### ABACUS - -`stru`, `abacus/scf`, `abacus/md`, `abacus/relax` - -### And many more... - -QE, CP2K, Gaussian, ORCA, PSI4, FHI-aims, SIESTA, PWmat, AMBER, GROMACS, ASE, pymatgen, XYZ, etc. - -## Requirements - -- [uv](https://docs.astral.sh/uv/) for running dpdata via `uvx` - -## References - -- [dpdata Documentation](https://docs.deepmodeling.com/projects/dpdata/) -- [CLI Reference](https://docs.deepmodeling.com/projects/dpdata/en/stable/cli.html) -- [Formats Reference](https://docs.deepmodeling.com/projects/dpdata/en/stable/formats.html) -- [GitHub Repository](https://github.com/deepmodeling/dpdata) +Use [`skills/dpdata-convert`](../dpdata-convert/SKILL.md) for new conversion +work. It includes the command reference, data-contract checks, and links to +producer-specific references. The `dpdata-cli` path remains available for +existing agent installations. diff --git a/skills/dpdata-cli/SKILL.md b/skills/dpdata-cli/SKILL.md index ac2a5189c..4115df455 100644 --- a/skills/dpdata-cli/SKILL.md +++ b/skills/dpdata-cli/SKILL.md @@ -1,222 +1,17 @@ --- name: dpdata-cli -description: Convert and manipulate atomic simulation data formats using dpdata CLI. Use when converting between DFT/MD output formats or preparing DeePMD-kit training data. Supports 50+ formats including deepmd, xyz or extxyz, ABACUS, VASP, LAMMPS, CP2K, etc. -compatibility: Requires uvx (uv) for running dpdata +description: Compatibility entry point for the dpdata conversion workflow. Use dpdata-convert for user-facing conversion and data-contract validation. +compatibility: Requires uvx (uv) for the command examples. metadata: - author: njzjz-bot - version: '1.0' repository: https://github.com/deepmodeling/dpdata --- -# dpdata CLI +# dpdata CLI compatibility entry point -dpdata is a tool for manipulating multiple atomic simulation data formats. This skill enables format conversion between various DFT/MD software outputs via command line. +Conversion requests now use [`dpdata-convert`](../dpdata-convert/SKILL.md), +which combines the command reference with semantic validation and producer +contracts. This path remains so existing agents that request `dpdata-cli` +continue to resolve. -## Quick Start - -Run dpdata via uvx: - -```bash -uvx dpdata [options] -``` - -## Command Line Usage - -```text -dpdata: Manipulating multiple atomic simulation data formats -usage: dpdata [-h] [--to_file TO_FILE] [--from_format FROM_FORMAT] - [--to_format TO_FORMAT] [--no-labeled] [--multi] - [--type-map TYPE_MAP [TYPE_MAP ...]] [--version] - from_file -``` - -### Arguments - -| Argument | Description | -| --------------------- | ----------------------------------------------------- | -| `from_file` | Read data from a file (positional) | -| `--to_file`, `-O` | Dump data to a file | -| `--from_format`, `-i` | Format of from_file (default: "auto") | -| `--to_format`, `-o` | Format of to_file | -| `--no-labeled`, `-n` | Labels aren't provided (default: False) | -| `--multi`, `-m` | System contains multiple directories (default: False) | -| `--type-map`, `-t` | Type map for atom types | -| `--version` | Show dpdata version and exit | - -## Common Examples - -### Convert VASP OUTCAR to deepmd format - -```bash -uvx dpdata OUTCAR -i vasp/outcar -O deepmd_data -o deepmd/raw -``` - -### Convert LAMMPS dump to VASP POSCAR - -```bash -uvx dpdata dump.lammps -i lammps/dump -O POSCAR -o vasp/poscar -``` - -### Convert with type map - -```bash -uvx dpdata OUTCAR -i vasp/outcar -O deepmd_data -o deepmd/raw -t C H O N -``` - -### Convert multiple systems - -```bash -uvx dpdata data_dir -i vasp/outcar -O output_dir -o deepmd/comp --multi -``` - -### Convert to deepmd/npy (compressed format) - -```bash -uvx dpdata OUTCAR -i vasp/outcar -O deepmd_npy -o deepmd/npy -``` - -### Convert to deepmd/hdf5 - -```bash -uvx dpdata OUTCAR -i vasp/outcar -O data.h5 -o deepmd/hdf5 -``` - -## Supported Formats - -Formats may be updated. For the complete and latest list, see: - -- [Formats Reference (stable)](https://docs.deepmodeling.com/projects/dpdata/en/stable/formats.html) - -### DeePMD-kit Formats - -| Format Name | Description | -| ---------------------------- | ---------------------------------- | -| `deepmd/raw` | DeePMD-kit raw text format | -| `deepmd/comp` / `deepmd/npy` | DeePMD-kit compressed numpy format | -| `deepmd/npy/mixed` | DeePMD-kit mixed type format | -| `deepmd/hdf5` | DeePMD-kit HDF5 format | - -### VASP Formats - -| Format Name | Description | -| ----------------------------------------------------- | -------------------- | -| `vasp/poscar` / `vasp/contcar` / `poscar` / `contcar` | VASP structure files | -| `vasp/outcar` / `outcar` | VASP OUTCAR output | -| `vasp/xml` / `xml` | VASP XML output | -| `vasp/string` | VASP string format | - -### LAMMPS Formats - -| Format Name | Description | -| ---------------------- | ---------------- | -| `lammps/lmp` / `lmp` | LAMMPS data file | -| `lammps/dump` / `dump` | LAMMPS dump file | - -### ABACUS Formats - -| Format Name | Description | -| -------------------------------------------------------- | --------------------- | -| `stru` / `abacus/stru` | ABACUS structure file | -| `abacus/lcao/scf` / `abacus/pw/scf` / `abacus/scf` | ABACUS SCF output | -| `abacus/lcao/md` / `abacus/pw/md` / `abacus/md` | ABACUS MD output | -| `abacus/lcao/relax` / `abacus/pw/relax` / `abacus/relax` | ABACUS relax output | - -### Quantum ESPRESSO Formats - -| Format Name | Description | -| ------------ | ---------------- | -| `qe/cp/traj` | QE CP trajectory | -| `qe/pw/scf` | QE PWscf output | - -### CP2K Formats - -| Format Name | Description | -| ------------------ | ---------------- | -| `cp2k/output` | CP2K output | -| `cp2k/aimd_output` | CP2K AIMD output | - -### Gaussian Formats - -| Format Name | Description | -| --------------- | ----------------------------- | -| `gaussian/log` | Gaussian log file | -| `gaussian/fchk` | Gaussian formatted checkpoint | -| `gaussian/md` | Gaussian MD output | -| `gaussian/gjf` | Gaussian input file | - -### Other Formats - -| Format Name | Description | -| ------------------------------------------------------------------- | --------------------- | -| `xyz` | XYZ format | -| `mace/xyz` / `nequip/xyz` / `gpumd/xyz` / `extxyz` / `quip/gap/xyz` | Extended XYZ variants | -| `ase/structure` | ASE structure format | -| `ase/traj` | ASE trajectory | -| `pymatgen/structure` | pymatgen structure | -| `pymatgen/molecule` | pymatgen molecule | -| `gromacs/gro` / `gro` | GROMACS gro file | -| `siesta/output` | SIESTA output | -| `siesta/aimd_output` | SIESTA AIMD output | -| `pwmat/output` / `pwmat/mlmd` / `pwmat/movement` | PWmat output | -| `pwmat/final.config` / `pwmat/atom.config` | PWmat config | -| `orca/spout` | ORCA output | -| `psi4/out` | PSI4 output | -| `dftbplus` | DFTB+ output | -| `fhi_aims/output` / `fhi_aims/md` | FHI-aims output | -| `amber/md` | AMBER MD | -| `n2p2` | n2p2 format | -| `mol_file` / `mol` | MOL file | -| `sdf_file` / `sdf` | SDF file | -| `openmx/md` | OpenMX MD | -| `sqm/out` | SQM output | -| `sqm/in` | SQM input | -| `list` | List format | -| `3dmol` | 3Dmol visualization | - -### Extended XYZ (extxyz) Format Details - -The following format names are **all equivalent** and invoke the same reader/writer: - -- `extxyz` — general-purpose extended XYZ -- `quip/gap/xyz` / `quip/gap/xyz_file` — QUIP/GAP framework datasets -- `mace/xyz` — MACE model training data -- `nequip/xyz` — NequIP model training data -- `gpumd/xyz` — GPUMD simulation data - -> **Important:** Plain `xyz` is a **different**, simpler format that stores atom species and coordinates but does not provide per-frame properties such as energies, forces, cell information, or periodic boundary conditions. Do not confuse `xyz` with `extxyz`. - -#### CLI Examples for extxyz - -```bash -# Multi-frame extxyz trajectory → deepmd/npy (--multi preserves all frames) -uvx dpdata data.xyz -i extxyz -O deepmd_data -o deepmd/npy --multi - -# Heterogeneous extxyz (mixed compositions) → deepmd/npy/mixed -uvx dpdata data.xyz -i extxyz -O deepmd_data -o deepmd/npy/mixed --multi - -# Convert extxyz to VASP POSCAR (single-frame input only) -uvx dpdata data.xyz -i extxyz -O POSCAR -o vasp/poscar - -# Convert a single-frame extxyz with explicit type map -uvx dpdata data.xyz -i extxyz -O deepmd_data -o deepmd/npy -t H C N O -``` - -#### Tips for extxyz - -1. **Use `-i extxyz` explicitly** when working with `.xyz` files that contain extended XYZ data. Using `-i xyz` retains only atom species and coordinates, omitting per-frame properties such as energies, forces, and cell information. -1. **Use `--multi` for any multi-frame extxyz file.** Without `--multi`, the CLI constructs a `LabeledSystem` that keeps only the first frame. This applies to all multi-frame trajectories, not just heterogeneous (mixed-composition) files. dpdata groups frames by chemical formula internally. -1. **All alias names behave identically** — `-i mace/xyz`, `-i nequip/xyz`, `-i gpumd/xyz`, `-i quip/gap/xyz`, `-i quip/gap/xyz_file`, and `-i extxyz` produce the same result. - -## Tips - -1. **Auto-detection**: Use `-i auto` (default) to let dpdata detect format automatically -1. **Type mapping**: Use `-t` to specify atom type order for deepmd formats -1. **Multi-system**: Use `--multi` for directories containing multiple systems -1. **Compressed output**: Use `deepmd/npy` or `deepmd/hdf5` for smaller file sizes - -## References - -- [dpdata Documentation](https://docs.deepmodeling.com/projects/dpdata/) -- [CLI Reference](https://docs.deepmodeling.com/projects/dpdata/en/stable/cli.html) -- [Formats Reference](https://docs.deepmodeling.com/projects/dpdata/en/stable/formats.html) -- [GitHub Repository](https://github.com/deepmodeling/dpdata) +For the full command and format table, see +[`cli-reference.md`](../dpdata-convert/references/cli-reference.md). diff --git a/skills/dpdata-convert/SKILL.md b/skills/dpdata-convert/SKILL.md new file mode 100644 index 000000000..460dc5f24 --- /dev/null +++ b/skills/dpdata-convert/SKILL.md @@ -0,0 +1,69 @@ +--- +name: dpdata-convert +description: Convert atomistic producer outputs with dpdata while preserving and validating the labeled-data contract. Use for format conversion, dataset preparation, and exports from VASP, CP2K, ABACUS, Quantum ESPRESSO, LAMMPS, or other supported producers. +compatibility: Requires dpdata and the dependencies of the selected format. +metadata: + repository: https://github.com/deepmodeling/dpdata +--- + +# Convert atomistic data + +Use this workflow when a user starts with a producer output and wants a +`dpdata` system, a training dataset, or another file format. The command is +only one part of the task: the resulting arrays must retain their scientific +meaning. + +## Procedure + +1. **Identify the producer contract.** Record the producer and version, input + and output paths, parser/`dpdata` version, and whether the source contains + energies, forces, and virials/stress. Load the matching reference only when + the producer is known: + [`CP2K`](references/producers/cp2k.md), + [`VASP`](references/producers/vasp.md), + [`ABACUS`](references/producers/abacus.md), or + [`Quantum ESPRESSO`](references/producers/qe.md). +1. **Choose the data shape.** Use `LabeledSystem` for data that must carry + energies, and `System` for structures without labels. Use `MultiSystems` + or `--multi` only when directories contain independent systems. Do not + silently turn partial labels into a complete dataset; state which labels + are required and what happens when one is absent. +1. **Convert without discarding the source.** Keep the raw files and write the + converted output to a new path. Include an explicit `--from_format` when + auto-detection could be ambiguous. Supply `--type-map` when the producer's + atom ordering is not self-describing. +1. **Validate the output.** Check frame alignment, atom ordering, type-map + consistency, finite values, cell/PBC assumptions, units, and virial/stress + conventions. The reusable checks and their limits are in + [`semantic-validation.md`](references/semantic-validation.md); the + deterministic validator is part of [`dpdata-inspect`](../dpdata-inspect/SKILL.md). +1. **Record provenance.** Keep the raw path, producer/version, parser version, + command, format, type map, units, and any dropped or missing labels next to + the converted data. A successful parser run is not evidence that a dataset + is suitable for training. + +## Command reference + +The CLI uses the same format names as the Python API: + +```bash +uvx dpdata INPUT -i PRODUCER_FORMAT -O OUTPUT -o TARGET_FORMAT +``` + +Examples: + +```bash +# A labeled VASP trajectory to DeePMD raw data +uvx dpdata OUTCAR -i vasp/outcar -O deepmd_data -o deepmd/raw + +# A directory containing independent systems +uvx dpdata data_dir -i vasp/outcar -O output_dir -o deepmd/comp --multi + +# Make the atom order explicit when reading a LAMMPS dump +uvx dpdata dump.lammps -i lammps/dump -O POSCAR -o vasp/poscar -t C H O +``` + +For all options and format aliases, load +[`cli-reference.md`](references/cli-reference.md). The reference is kept +separate so a conversion request does not require loading every format at +once. diff --git a/skills/dpdata-convert/references/cli-reference.md b/skills/dpdata-convert/references/cli-reference.md new file mode 100644 index 000000000..ac2a5189c --- /dev/null +++ b/skills/dpdata-convert/references/cli-reference.md @@ -0,0 +1,222 @@ +--- +name: dpdata-cli +description: Convert and manipulate atomic simulation data formats using dpdata CLI. Use when converting between DFT/MD output formats or preparing DeePMD-kit training data. Supports 50+ formats including deepmd, xyz or extxyz, ABACUS, VASP, LAMMPS, CP2K, etc. +compatibility: Requires uvx (uv) for running dpdata +metadata: + author: njzjz-bot + version: '1.0' + repository: https://github.com/deepmodeling/dpdata +--- + +# dpdata CLI + +dpdata is a tool for manipulating multiple atomic simulation data formats. This skill enables format conversion between various DFT/MD software outputs via command line. + +## Quick Start + +Run dpdata via uvx: + +```bash +uvx dpdata [options] +``` + +## Command Line Usage + +```text +dpdata: Manipulating multiple atomic simulation data formats +usage: dpdata [-h] [--to_file TO_FILE] [--from_format FROM_FORMAT] + [--to_format TO_FORMAT] [--no-labeled] [--multi] + [--type-map TYPE_MAP [TYPE_MAP ...]] [--version] + from_file +``` + +### Arguments + +| Argument | Description | +| --------------------- | ----------------------------------------------------- | +| `from_file` | Read data from a file (positional) | +| `--to_file`, `-O` | Dump data to a file | +| `--from_format`, `-i` | Format of from_file (default: "auto") | +| `--to_format`, `-o` | Format of to_file | +| `--no-labeled`, `-n` | Labels aren't provided (default: False) | +| `--multi`, `-m` | System contains multiple directories (default: False) | +| `--type-map`, `-t` | Type map for atom types | +| `--version` | Show dpdata version and exit | + +## Common Examples + +### Convert VASP OUTCAR to deepmd format + +```bash +uvx dpdata OUTCAR -i vasp/outcar -O deepmd_data -o deepmd/raw +``` + +### Convert LAMMPS dump to VASP POSCAR + +```bash +uvx dpdata dump.lammps -i lammps/dump -O POSCAR -o vasp/poscar +``` + +### Convert with type map + +```bash +uvx dpdata OUTCAR -i vasp/outcar -O deepmd_data -o deepmd/raw -t C H O N +``` + +### Convert multiple systems + +```bash +uvx dpdata data_dir -i vasp/outcar -O output_dir -o deepmd/comp --multi +``` + +### Convert to deepmd/npy (compressed format) + +```bash +uvx dpdata OUTCAR -i vasp/outcar -O deepmd_npy -o deepmd/npy +``` + +### Convert to deepmd/hdf5 + +```bash +uvx dpdata OUTCAR -i vasp/outcar -O data.h5 -o deepmd/hdf5 +``` + +## Supported Formats + +Formats may be updated. For the complete and latest list, see: + +- [Formats Reference (stable)](https://docs.deepmodeling.com/projects/dpdata/en/stable/formats.html) + +### DeePMD-kit Formats + +| Format Name | Description | +| ---------------------------- | ---------------------------------- | +| `deepmd/raw` | DeePMD-kit raw text format | +| `deepmd/comp` / `deepmd/npy` | DeePMD-kit compressed numpy format | +| `deepmd/npy/mixed` | DeePMD-kit mixed type format | +| `deepmd/hdf5` | DeePMD-kit HDF5 format | + +### VASP Formats + +| Format Name | Description | +| ----------------------------------------------------- | -------------------- | +| `vasp/poscar` / `vasp/contcar` / `poscar` / `contcar` | VASP structure files | +| `vasp/outcar` / `outcar` | VASP OUTCAR output | +| `vasp/xml` / `xml` | VASP XML output | +| `vasp/string` | VASP string format | + +### LAMMPS Formats + +| Format Name | Description | +| ---------------------- | ---------------- | +| `lammps/lmp` / `lmp` | LAMMPS data file | +| `lammps/dump` / `dump` | LAMMPS dump file | + +### ABACUS Formats + +| Format Name | Description | +| -------------------------------------------------------- | --------------------- | +| `stru` / `abacus/stru` | ABACUS structure file | +| `abacus/lcao/scf` / `abacus/pw/scf` / `abacus/scf` | ABACUS SCF output | +| `abacus/lcao/md` / `abacus/pw/md` / `abacus/md` | ABACUS MD output | +| `abacus/lcao/relax` / `abacus/pw/relax` / `abacus/relax` | ABACUS relax output | + +### Quantum ESPRESSO Formats + +| Format Name | Description | +| ------------ | ---------------- | +| `qe/cp/traj` | QE CP trajectory | +| `qe/pw/scf` | QE PWscf output | + +### CP2K Formats + +| Format Name | Description | +| ------------------ | ---------------- | +| `cp2k/output` | CP2K output | +| `cp2k/aimd_output` | CP2K AIMD output | + +### Gaussian Formats + +| Format Name | Description | +| --------------- | ----------------------------- | +| `gaussian/log` | Gaussian log file | +| `gaussian/fchk` | Gaussian formatted checkpoint | +| `gaussian/md` | Gaussian MD output | +| `gaussian/gjf` | Gaussian input file | + +### Other Formats + +| Format Name | Description | +| ------------------------------------------------------------------- | --------------------- | +| `xyz` | XYZ format | +| `mace/xyz` / `nequip/xyz` / `gpumd/xyz` / `extxyz` / `quip/gap/xyz` | Extended XYZ variants | +| `ase/structure` | ASE structure format | +| `ase/traj` | ASE trajectory | +| `pymatgen/structure` | pymatgen structure | +| `pymatgen/molecule` | pymatgen molecule | +| `gromacs/gro` / `gro` | GROMACS gro file | +| `siesta/output` | SIESTA output | +| `siesta/aimd_output` | SIESTA AIMD output | +| `pwmat/output` / `pwmat/mlmd` / `pwmat/movement` | PWmat output | +| `pwmat/final.config` / `pwmat/atom.config` | PWmat config | +| `orca/spout` | ORCA output | +| `psi4/out` | PSI4 output | +| `dftbplus` | DFTB+ output | +| `fhi_aims/output` / `fhi_aims/md` | FHI-aims output | +| `amber/md` | AMBER MD | +| `n2p2` | n2p2 format | +| `mol_file` / `mol` | MOL file | +| `sdf_file` / `sdf` | SDF file | +| `openmx/md` | OpenMX MD | +| `sqm/out` | SQM output | +| `sqm/in` | SQM input | +| `list` | List format | +| `3dmol` | 3Dmol visualization | + +### Extended XYZ (extxyz) Format Details + +The following format names are **all equivalent** and invoke the same reader/writer: + +- `extxyz` — general-purpose extended XYZ +- `quip/gap/xyz` / `quip/gap/xyz_file` — QUIP/GAP framework datasets +- `mace/xyz` — MACE model training data +- `nequip/xyz` — NequIP model training data +- `gpumd/xyz` — GPUMD simulation data + +> **Important:** Plain `xyz` is a **different**, simpler format that stores atom species and coordinates but does not provide per-frame properties such as energies, forces, cell information, or periodic boundary conditions. Do not confuse `xyz` with `extxyz`. + +#### CLI Examples for extxyz + +```bash +# Multi-frame extxyz trajectory → deepmd/npy (--multi preserves all frames) +uvx dpdata data.xyz -i extxyz -O deepmd_data -o deepmd/npy --multi + +# Heterogeneous extxyz (mixed compositions) → deepmd/npy/mixed +uvx dpdata data.xyz -i extxyz -O deepmd_data -o deepmd/npy/mixed --multi + +# Convert extxyz to VASP POSCAR (single-frame input only) +uvx dpdata data.xyz -i extxyz -O POSCAR -o vasp/poscar + +# Convert a single-frame extxyz with explicit type map +uvx dpdata data.xyz -i extxyz -O deepmd_data -o deepmd/npy -t H C N O +``` + +#### Tips for extxyz + +1. **Use `-i extxyz` explicitly** when working with `.xyz` files that contain extended XYZ data. Using `-i xyz` retains only atom species and coordinates, omitting per-frame properties such as energies, forces, and cell information. +1. **Use `--multi` for any multi-frame extxyz file.** Without `--multi`, the CLI constructs a `LabeledSystem` that keeps only the first frame. This applies to all multi-frame trajectories, not just heterogeneous (mixed-composition) files. dpdata groups frames by chemical formula internally. +1. **All alias names behave identically** — `-i mace/xyz`, `-i nequip/xyz`, `-i gpumd/xyz`, `-i quip/gap/xyz`, `-i quip/gap/xyz_file`, and `-i extxyz` produce the same result. + +## Tips + +1. **Auto-detection**: Use `-i auto` (default) to let dpdata detect format automatically +1. **Type mapping**: Use `-t` to specify atom type order for deepmd formats +1. **Multi-system**: Use `--multi` for directories containing multiple systems +1. **Compressed output**: Use `deepmd/npy` or `deepmd/hdf5` for smaller file sizes + +## References + +- [dpdata Documentation](https://docs.deepmodeling.com/projects/dpdata/) +- [CLI Reference](https://docs.deepmodeling.com/projects/dpdata/en/stable/cli.html) +- [Formats Reference](https://docs.deepmodeling.com/projects/dpdata/en/stable/formats.html) +- [GitHub Repository](https://github.com/deepmodeling/dpdata) diff --git a/skills/dpdata-convert/references/producers/abacus.md b/skills/dpdata-convert/references/producers/abacus.md new file mode 100644 index 000000000..37ec7c955 --- /dev/null +++ b/skills/dpdata-convert/references/producers/abacus.md @@ -0,0 +1,18 @@ +# ABACUS producer contract + +Use this reference when converting ABACUS SCF, MD, relax, or `STRU` output. +Record the ABACUS version, calculation mode (PW or LCAO), and the input/output +paths used by the parser. + +Before accepting the converted data, verify: + +- the `STRU` atom type labels and atom order match `atom_names` and + `atom_types`; +- SCF, MD, or relax frames have matching coordinates, cells, energies, and + forces; +- cell-relax changes are preserved as per-frame cells; and +- any stress or force sign and unit conversion is explicit. + +ABACUS output formats evolve. Check the producer release and parser contract +for the exact output layout; do not assume a successful parse proves that all +optional labels were present. diff --git a/skills/dpdata-convert/references/producers/cp2k.md b/skills/dpdata-convert/references/producers/cp2k.md new file mode 100644 index 000000000..41fd6d108 --- /dev/null +++ b/skills/dpdata-convert/references/producers/cp2k.md @@ -0,0 +1,19 @@ +# CP2K producer contract + +Use this reference when converting CP2K `*.out`, AIMD, or trajectory output. +Select the dpdata format that matches the producer output (`cp2k/output` or +`cp2k/aimd_output`) and record the CP2K release and the output sections that +were enabled. + +Before accepting the converted data, verify: + +- the coordinate and cell sections describe the same frames as the energy and + force sections; +- the calculation's periodicity and cell vectors are preserved; +- energies, forces, and any stress/virial values use the units and sign + convention expected by the downstream consumer; and +- restarts or truncated runs did not create unmatched frames. + +CP2K output details vary by release and input print settings. Treat the CP2K +manual and the output producer version as the authority. Parser defects belong +in the parser project; this reference defines the validation questions to ask. diff --git a/skills/dpdata-convert/references/producers/qe.md b/skills/dpdata-convert/references/producers/qe.md new file mode 100644 index 000000000..a6edddbe3 --- /dev/null +++ b/skills/dpdata-convert/references/producers/qe.md @@ -0,0 +1,16 @@ +# Quantum ESPRESSO producer contract + +Use this reference when converting Quantum ESPRESSO PWscf or CP trajectory +output (`qe/pw/scf` or `qe/cp/traj`). Record the QE release, input file, output +file, and any trajectory sidecar files used by the parser. + +Before accepting the converted data, verify: + +- the input atom order is the order used in every force and trajectory frame; +- the cell and periodic boundary conditions are retained; +- energies and forces have the same frame count; and +- stress/virial values, units, and sign conventions are checked rather than + inferred from a label name. + +QE output may omit labels when a run stops early. Preserve the raw files and +fail the conversion validation when a required label is missing. diff --git a/skills/dpdata-convert/references/producers/vasp.md b/skills/dpdata-convert/references/producers/vasp.md new file mode 100644 index 000000000..2feded4f4 --- /dev/null +++ b/skills/dpdata-convert/references/producers/vasp.md @@ -0,0 +1,19 @@ +# VASP producer contract + +Use this reference when converting POSCAR/CONTCAR, OUTCAR, or vasprun XML. +Choose the corresponding dpdata format and record the VASP release and whether +the source is a structure-only file or a labeled calculation. + +Before accepting the converted data, verify: + +- atom names, counts, and the coordinate order agree with the selected + `type_map`; +- selective dynamics, Cartesian/direct coordinates, and cell vectors were + interpreted as intended; +- energies and forces are frame-aligned, and stress/virial conversion has + been checked against the downstream convention; and +- an incomplete final ionic step was not retained as a labeled frame. + +Use the VASP documentation and the exact output version for units and stress +semantics. Keep the original POSCAR/OUTCAR/vasprun XML with the converted +system. diff --git a/skills/dpdata-convert/references/semantic-validation.md b/skills/dpdata-convert/references/semantic-validation.md new file mode 100644 index 000000000..c491e57f3 --- /dev/null +++ b/skills/dpdata-convert/references/semantic-validation.md @@ -0,0 +1,43 @@ +# Semantic validation contract + +A parser can succeed while producing a dataset that is unusable for its next +scientific step. Validate the following contract after conversion. Checks that +can be made mechanically should be delegated to +`dpdata-inspect/scripts/validate_labeled_system.py`; conventions that are not +encoded in dpdata must be declared by the producer contract. + +| Invariant | Check | +| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Labels | Decide whether energies are required. If forces or virials are present, they must be complete for the same frames. Missing labels are reported, not filled with zeros or silently dropped. | +| Frame alignment | `coords` and `cells` have `nframes`; required labels cover every frame; optional labels, when present, have matching frame dimensions. Forces have `(nframes, natoms, 3)` and virials have `(nframes, 3, 3)`. | +| Atoms and types | `atom_types` has `natoms`, indices are in `atom_names`, and `sum(atom_numbs) == natoms`. A multi-system dataset uses one documented type-map order. | +| Values | Coordinates, cells, energies, forces, and virials contain finite values. | +| Cell and PBC | Decide whether the source is periodic. For periodic data, cells are present and nonsingular; for molecules, preserve the explicit non-periodic (`nopbc`) choice. | +| Units | Record energy and force units. Record the virial/stress unit and the sign convention separately; they are not inferred from array names. | +| Provenance | Preserve the raw input, producer and version, parser version, conversion command, selected format, type map, and any filtering or dropped labels. | + +`dpdata`'s standard labeled fields are `energies`, `forces`, and `virials`. +Some producers call the last field stress or pressure. Confirm whether the +producer reports stress or virial and whether its sign is the negative of the +convention expected by the consumer. + +Producer references describe where to look for a contract. Project-specific +requirements such as a required print level, convergence threshold, or +scheduler setup belong in the project record, not in this reusable skill. + +## Deterministic validator + +For one labeled input, run: + +```bash +python skills/dpdata-inspect/scripts/validate_labeled_system.py \ + converted -f deepmd/raw \ + --energy-unit eV --force-unit eV/A \ + --producer VASP --producer-version 6.4 --parser-version 0.1 \ + --provenance raw/OUTCAR +``` + +Use `--multi` for a directory of systems, `--type-map C H O` to assert atom +ordering, and `--require-pbc` when periodic cells are required. Add +`--strict-metadata` when undeclared units, stress sign, or provenance should +fail the check. `--json` emits a machine-readable result for CI. diff --git a/skills/dpdata-driver/SKILL.md b/skills/dpdata-driver/SKILL.md index 680bf3734..fdd425c3e 100644 --- a/skills/dpdata-driver/SKILL.md +++ b/skills/dpdata-driver/SKILL.md @@ -1,154 +1,13 @@ --- name: dpdata-driver -description: Use dpdata Python Driver plugins to label systems (energies/forces/virials) via System.predict(), list available drivers, and build Driver objects (ase/deepmd/gaussian/sqm/hybrid). Use when working with dpdata Python API (not CLI) and you need driver-based energy/force prediction, plugin registration keys, or examples of using dpdata with ASE calculators or DeePMD models. -license: LGPL-3.0-or-later -compatibility: Requires dpdata or uv for running dpdata +description: Compatibility entry point for the developer-facing dpdata driver reference. Use dpdata-extend for new driver implementations and System.predict workflows. +compatibility: Requires dpdata; drivers may require optional dependencies or external executables. metadata: - author: njzjz-bot - version: '1.0' + repository: https://github.com/deepmodeling/dpdata --- -# dpdata-driver +# dpdata driver compatibility entry point -Use dpdata “driver plugins” to **label** a `dpdata.System` (predict energies/forces/virials) and obtain a `dpdata.LabeledSystem`. - -## Key idea - -- A **Driver** converts an unlabeled `System` into a `LabeledSystem` by computing: - - `energies` (required) - - `forces` (optional but common) - - `virials` (optional) - -In dpdata, this is exposed as: - -- `System.predict(*args, driver="dp", **kwargs) -> LabeledSystem` - -`driver` can be: - -- a **string key** (plugin name), e.g. `"ase"`, `"dp"`, `"gaussian"` -- a **Driver object**, e.g. `Driver.get_driver("ase")(...)` - -## List supported driver keys (runtime) - -When unsure what drivers exist in *this* dpdata version/env, query them at runtime: - -```python -import dpdata -from dpdata.driver import Driver - -print(sorted(Driver.get_drivers().keys())) -``` - -`import dpdata` ensures built-in plugins are loaded before listing registered drivers. - -In the current repo state, keys include: - -- `ase` -- `dp` / `deepmd` / `deepmd-kit` -- `gaussian` -- `sqm` -- `hybrid` - -(Exact set depends on dpdata version and installed extras.) - -## Minimal workflow - -```python -import dpdata -from dpdata.system import System - -sys = System("input.xyz", fmt="xyz") -ls = sys.predict(driver="ase", calculator=...) # returns dpdata.LabeledSystem -``` - -### Verify you got a labeled system - -```python -assert "energies" in ls.data -# optional: -# assert "forces" in ls.data -# assert "virials" in ls.data -``` - -## Example: use the ASE driver with an ASE calculator (runnable) - -This is the easiest *fully runnable* example because it doesn’t require external QM software. - -Dependencies (recommended): declare script dependencies with uv inline metadata, then run with `uv run`. - -```python -# /// script -# requires-python = ">=3.8" -# dependencies = [ -# "dpdata", -# "numpy", -# "ase", -# ] -# /// -``` - -Script: - -```python -from pathlib import Path - -import numpy as np -from ase.calculators.lj import LennardJones -from dpdata.system import System - -# write a tiny molecule -Path("tmp.xyz").write_text("""2\n\nH 0 0 0\nH 0 0 0.74\n""") - -sys = System("tmp.xyz", fmt="xyz") -ls = sys.predict(driver="ase", calculator=LennardJones()) - -print("energies", np.array(ls.data["energies"])) -print("forces shape", np.array(ls.data["forces"]).shape) -if "virials" in ls.data: - print("virials shape", np.array(ls.data["virials"]).shape) -else: - print("virials: ") -``` - -## Example: pass a Driver object instead of a string - -```python -from ase.calculators.lj import LennardJones -from dpdata.driver import Driver -from dpdata.system import System - -sys = System("tmp.xyz", fmt="xyz") -ase_driver = Driver.get_driver("ase")(calculator=LennardJones()) -ls = sys.predict(driver=ase_driver) -``` - -## Hybrid driver - -Use `driver="hybrid"` to sum energies/forces/virials from multiple drivers. - -The `HybridDriver` accepts `drivers=[ ... ]` where each item is either: - -- a `Driver` instance -- a dict like `{"type": "sqm", ...}` (type is the driver key) - -Example (structure only; may require external executables): - -```python -from dpdata.driver import Driver - -hyb = Driver.get_driver("hybrid")( - drivers=[ - {"type": "sqm", "qm_theory": "DFTB3"}, - {"type": "dp", "dp": "frozen_model.pb"}, - ] -) -# ls = sys.predict(driver=hyb) -``` - -## Notes / gotchas - -- Many drivers require extra dependencies or external programs: - - `dp` requires `deepmd-kit` + a model file - - `gaussian` requires Gaussian and a valid executable (default `g16`) - - `sqm` requires AmberTools `sqm` -- If you just need file format conversion, use the existing **dpdata CLI** skill instead. +The developer reference now lives at +[`dpdata-extend/references/driver.md`](../dpdata-extend/references/driver.md). +Use [`dpdata-extend`](../dpdata-extend/SKILL.md) for new API-extension work. diff --git a/skills/dpdata-extend/SKILL.md b/skills/dpdata-extend/SKILL.md new file mode 100644 index 000000000..4ea3f04c3 --- /dev/null +++ b/skills/dpdata-extend/SKILL.md @@ -0,0 +1,27 @@ +--- +name: dpdata-extend +description: Extend dpdata for developers by adding format plugins, drivers, or minimizers. Use when implementing or packaging dpdata extensions rather than converting user data. +compatibility: Requires dpdata development knowledge; optional dependencies depend on the extension. +metadata: + repository: https://github.com/deepmodeling/dpdata +--- + +# Extend dpdata + +Use this skill for implementation-facing work. Choose one focused reference: + +- [`format-plugin.md`](references/format-plugin.md) for a new format or an + external `dpdata.plugins` package; +- [`driver.md`](references/driver.md) for `System.predict()` and label + generation; or +- [`minimizer.md`](references/minimizer.md) for geometry optimization through + `System.minimize()`. + +An extension must state the data contract it produces. For a format reader, +that includes atom/type ordering, frame count, cells/PBC, labels, units, and +provenance expectations. Add a focused regression test for parser behavior and +keep producer-specific assumptions in the format reference or parser project. + +This skill is deliberately separate from the user workflows. Route requests +to convert or inspect data through `dpdata-convert` or `dpdata-inspect` instead +of loading developer API references. diff --git a/skills/dpdata-extend/references/driver.md b/skills/dpdata-extend/references/driver.md new file mode 100644 index 000000000..680bf3734 --- /dev/null +++ b/skills/dpdata-extend/references/driver.md @@ -0,0 +1,154 @@ +--- +name: dpdata-driver +description: Use dpdata Python Driver plugins to label systems (energies/forces/virials) via System.predict(), list available drivers, and build Driver objects (ase/deepmd/gaussian/sqm/hybrid). Use when working with dpdata Python API (not CLI) and you need driver-based energy/force prediction, plugin registration keys, or examples of using dpdata with ASE calculators or DeePMD models. +license: LGPL-3.0-or-later +compatibility: Requires dpdata or uv for running dpdata +metadata: + author: njzjz-bot + version: '1.0' +--- + +# dpdata-driver + +Use dpdata “driver plugins” to **label** a `dpdata.System` (predict energies/forces/virials) and obtain a `dpdata.LabeledSystem`. + +## Key idea + +- A **Driver** converts an unlabeled `System` into a `LabeledSystem` by computing: + - `energies` (required) + - `forces` (optional but common) + - `virials` (optional) + +In dpdata, this is exposed as: + +- `System.predict(*args, driver="dp", **kwargs) -> LabeledSystem` + +`driver` can be: + +- a **string key** (plugin name), e.g. `"ase"`, `"dp"`, `"gaussian"` +- a **Driver object**, e.g. `Driver.get_driver("ase")(...)` + +## List supported driver keys (runtime) + +When unsure what drivers exist in *this* dpdata version/env, query them at runtime: + +```python +import dpdata +from dpdata.driver import Driver + +print(sorted(Driver.get_drivers().keys())) +``` + +`import dpdata` ensures built-in plugins are loaded before listing registered drivers. + +In the current repo state, keys include: + +- `ase` +- `dp` / `deepmd` / `deepmd-kit` +- `gaussian` +- `sqm` +- `hybrid` + +(Exact set depends on dpdata version and installed extras.) + +## Minimal workflow + +```python +import dpdata +from dpdata.system import System + +sys = System("input.xyz", fmt="xyz") +ls = sys.predict(driver="ase", calculator=...) # returns dpdata.LabeledSystem +``` + +### Verify you got a labeled system + +```python +assert "energies" in ls.data +# optional: +# assert "forces" in ls.data +# assert "virials" in ls.data +``` + +## Example: use the ASE driver with an ASE calculator (runnable) + +This is the easiest *fully runnable* example because it doesn’t require external QM software. + +Dependencies (recommended): declare script dependencies with uv inline metadata, then run with `uv run`. + +```python +# /// script +# requires-python = ">=3.8" +# dependencies = [ +# "dpdata", +# "numpy", +# "ase", +# ] +# /// +``` + +Script: + +```python +from pathlib import Path + +import numpy as np +from ase.calculators.lj import LennardJones +from dpdata.system import System + +# write a tiny molecule +Path("tmp.xyz").write_text("""2\n\nH 0 0 0\nH 0 0 0.74\n""") + +sys = System("tmp.xyz", fmt="xyz") +ls = sys.predict(driver="ase", calculator=LennardJones()) + +print("energies", np.array(ls.data["energies"])) +print("forces shape", np.array(ls.data["forces"]).shape) +if "virials" in ls.data: + print("virials shape", np.array(ls.data["virials"]).shape) +else: + print("virials: ") +``` + +## Example: pass a Driver object instead of a string + +```python +from ase.calculators.lj import LennardJones +from dpdata.driver import Driver +from dpdata.system import System + +sys = System("tmp.xyz", fmt="xyz") +ase_driver = Driver.get_driver("ase")(calculator=LennardJones()) +ls = sys.predict(driver=ase_driver) +``` + +## Hybrid driver + +Use `driver="hybrid"` to sum energies/forces/virials from multiple drivers. + +The `HybridDriver` accepts `drivers=[ ... ]` where each item is either: + +- a `Driver` instance +- a dict like `{"type": "sqm", ...}` (type is the driver key) + +Example (structure only; may require external executables): + +```python +from dpdata.driver import Driver + +hyb = Driver.get_driver("hybrid")( + drivers=[ + {"type": "sqm", "qm_theory": "DFTB3"}, + {"type": "dp", "dp": "frozen_model.pb"}, + ] +) +# ls = sys.predict(driver=hyb) +``` + +## Notes / gotchas + +- Many drivers require extra dependencies or external programs: + - `dp` requires `deepmd-kit` + a model file + - `gaussian` requires Gaussian and a valid executable (default `g16`) + - `sqm` requires AmberTools `sqm` +- If you just need file format conversion, use the existing **dpdata CLI** skill instead. diff --git a/skills/dpdata-extend/references/format-plugin.md b/skills/dpdata-extend/references/format-plugin.md new file mode 100644 index 000000000..818cc8f92 --- /dev/null +++ b/skills/dpdata-extend/references/format-plugin.md @@ -0,0 +1,119 @@ +--- +name: dpdata-plugin +description: Create and install dpdata plugins (especially custom Format readers/writers) using Format.register(...) and pyproject.toml entry_points under 'dpdata.plugins'. Use when extending dpdata with new formats or distributing plugins as separate Python packages. +--- + +# dpdata-plugin + +dpdata loads plugins in two ways: + +1. **Built-in plugins** in `dpdata.plugins.*` (imported automatically) +1. **External plugins** exposed via Python package entry points: `dpdata.plugins` + +This skill focuses on **external plugin packages**, the recommended way to add new formats without modifying dpdata itself. + +## What can be extended? + +Most commonly: add a new **Format** (file reader/writer) via: + +```python +from dpdata.format import Format + + +@Format.register("myfmt") +class MyFormat(Format): ... +``` + +## How dpdata discovers plugins + +dpdata imports `dpdata.plugins` during normal use (e.g. `dpdata.system` imports it). That module: + +- imports every built-in module in `dpdata/plugins/*.py` +- then loads all **entry points** in group `dpdata.plugins` + +So an external plugin package only needs to ensure that importing the entry-point target triggers the `@Format.register(...)` side effects. + +## Minimal external plugin package (based on plugin_example/) + +### 1) Create a new Python package + +Example layout: + +```text +dpdata_random/ + pyproject.toml + dpdata_random/ + __init__.py +``` + +### 2) Implement and register your Format + +In `dpdata_random/__init__.py` (shortened example): + +```python +from __future__ import annotations + +import numpy as np +from dpdata.format import Format + + +@Format.register("random") +class RandomFormat(Format): + def from_system(self, file_name, **kwargs): + nframes = int(file_name) + return { + "atom_numbs": [20], + "atom_names": ["X"], + "atom_types": np.zeros(20, dtype=int), + "cells": np.repeat(np.eye(3)[None, ...], nframes, axis=0) * 100.0, + "coords": np.random.rand(nframes, 20, 3) * 100.0, + "orig": np.zeros(3), + "nopbc": False, + } +``` + +Return dicts must match dpdata’s expected schema (cells/coords/atom_names/atom_types/...). +Here `file_name` is the argument dpdata passes into `from_system`; this example interprets it as an integer frame count. + +### 3) Expose an entry point + +In `pyproject.toml`: + +```toml +[project] +name = "dpdata_random" +version = "0.0.0" +dependencies = ["numpy", "dpdata"] + +[project.entry-points.'dpdata.plugins'] +random = "dpdata_random:RandomFormat" +``` + +Any importable target works; this pattern points directly at the class. + +### 4) Install and test + +In a clean env (recommended via `uv`): + +```bash +uv venv +. .venv/bin/activate +uv pip install -e . dpdata numpy +python - <<'PY' +import dpdata +from dpdata.format import Format + +# importing dpdata will load entry points (dpdata.plugins) +print('random' in Format.get_formats()) +PY +``` + +If it prints `True`, your plugin was discovered. + +## Debug checklist + +- Did you install the plugin package into the same environment where you run dpdata? +- Does `pyproject.toml` contain `[project.entry-points.'dpdata.plugins']`? +- Does importing the entry point module/class execute the `@Format.register(...)` decorator? +- Before choosing a key like `random`, check `Format.get_formats()` to avoid colliding with an existing built-in or previously loaded plugin key. +- If using `uv run`, remember each command runs in its own environment unless you’re in a `uv` project (or you rely on `uv run --with ...`). diff --git a/skills/dpdata-extend/references/minimizer.md b/skills/dpdata-extend/references/minimizer.md new file mode 100644 index 000000000..69eb03da7 --- /dev/null +++ b/skills/dpdata-extend/references/minimizer.md @@ -0,0 +1,86 @@ +--- +name: dpdata-minimizer +description: Minimize geometries with dpdata minimizer plugins via System.minimize(), including how minimizers relate to drivers (ASEMinimizer needs a dpdata Driver) and how to list supported minimizers (ase/sqm). Use when doing geometry optimization/minimization through dpdata Python API. +--- + +# dpdata-minimizer + +Use dpdata “minimizer plugins” to **optimize/minimize geometry** and return a `dpdata.LabeledSystem`. + +## Key idea + +- A **Minimizer** performs geometry optimization (updates coordinates) and returns labeled results. +- dpdata exposes this as: + +```python +System.minimize(*args, minimizer: str|Minimizer, **kwargs) -> LabeledSystem +``` + +## List supported minimizer keys (runtime) + +```python +from dpdata.driver import Minimizer + +print(sorted(Minimizer.get_minimizers().keys())) +``` + +In the current dpdata repo, minimizer keys include: + +- `ase` +- `sqm` + +## Relationship to drivers (important) + +Some minimizers require a **dpdata Driver object**. + +Example: `ASEMinimizer` takes a dpdata `Driver` in its constructor: + +- `minimizer="ase"` requires `driver=` (e.g. the ASE driver wrapping an ASE calculator). + +So you generally do: + +1. Construct a driver +1. Construct a minimizer (or let dpdata do it by passing the right kwargs) +1. Call `System.minimize(...)` + +## Runnable example: ASE minimizer with an ASE calculator + +Use uv inline script metadata so the example runs reproducibly with `uv run`. + +```python +# /// script +# requires-python = ">=3.12" +# dependencies = [ +# "dpdata", +# "numpy", +# "ase", +# ] +# /// + +import numpy as np +from ase.calculators.emt import EMT + +from dpdata.driver import Driver +from dpdata.system import System + +open("tmp.xyz", "w").write("""2\n\nH 0 0 0\nH 0 0 0.74\n""") + +sys = System("tmp.xyz", fmt="xyz") + +# Build a dpdata driver that can provide energies/forces to ASE optimizers. +ase_driver = Driver.get_driver("ase")(calculator=EMT()) + +# Minimize using the ASE minimizer plugin. +# NOTE: ASEMinimizer expects `driver` (not `calculator`) as input. +ls = sys.minimize(minimizer="ase", driver=ase_driver, fmax=0.05, max_steps=5) + +print("coords", np.array(ls.data["coords"]).shape) +print("energies", np.array(ls.data["energies"])) +print("forces", np.array(ls.data["forces"]).shape) +``` + +## Notes / gotchas + +- `System.minimize(...)` accepts either a minimizer key string or a Minimizer object. +- If you previously used `System.predict(driver="ase", calculator=...)`, be aware that minimization is different: you need to pass a **driver** into the minimizer (ASEMinimizer does not accept `calculator=`). +- `sqm` minimizer requires AmberTools `sqm` executable and typically won’t be runnable in CI. diff --git a/skills/dpdata-inspect/SKILL.md b/skills/dpdata-inspect/SKILL.md new file mode 100644 index 000000000..e3168e37e --- /dev/null +++ b/skills/dpdata-inspect/SKILL.md @@ -0,0 +1,62 @@ +--- +name: dpdata-inspect +description: Inspect and validate dpdata systems against labeled-data, shape, atom-order, cell/PBC, finite-value, units, and provenance contracts. Use before training, merging, filtering, or publishing converted data. +compatibility: Requires dpdata for loading inputs; the deterministic validator requires NumPy. +metadata: + repository: https://github.com/deepmodeling/dpdata +--- + +# Inspect atomistic data + +Use this workflow when the user asks whether a dataset is valid, complete, or +ready for a downstream model. Start by identifying whether the input is one +system or a collection and whether labels are required. Then load the +producer-specific contract only if the source producer is known. + +## Procedure + +1. Preserve the raw source and record its producer/version, parser version, + format, and conversion command. +1. Load one `LabeledSystem` when energies are required. Load `System` only for + an intentionally unlabeled structure. Use `MultiSystems` for independent + systems and check that their atom/type order is compatible. +1. Run the deterministic validator below. It checks array shapes, frame + alignment, atom counts and type indices, finite values, labels, and (when + requested) periodic-cell determinants. +1. Review the remaining contract manually: units, stress versus virial and + sign, periodicity, atom ordering, missing-label policy, producer/version, + and provenance. These cannot be inferred safely from NumPy arrays. +1. Stop with an actionable error when a required label or contract declaration + is absent. Do not repair missing labels by inserting zeros. + +## Deterministic validator + +For a labeled system: + +```bash +python skills/dpdata-inspect/scripts/validate_labeled_system.py \ + converted -f deepmd/raw \ + --energy-unit eV --force-unit eV/A \ + --producer VASP --producer-version 6.4 --parser-version 0.1 \ + --provenance raw/source.out +``` + +For multiple systems or an explicitly unlabeled structure: + +```bash +python skills/dpdata-inspect/scripts/validate_labeled_system.py \ + dataset --format deepmd/comp --multi --type-map C H O +python skills/dpdata-inspect/scripts/validate_labeled_system.py \ + POSCAR --format vasp/poscar --allow-unlabeled --require-pbc +``` + +Add `--strict-metadata` to make undeclared units, stress sign, producer/version, +or provenance fail. Add `--json` for CI or another machine consumer. The script exits zero +only when every loaded system satisfies the selected contract. +The validator rejects a supplied `--provenance` path when the file does not exist +or is empty, whether or not `--strict-metadata` is enabled. + +The validator is intentionally small and deterministic. It does not infer +units, transform forces, choose a stress sign, or decide whether a project +should accept partial labels. Those are producer and project contracts and +must be recorded by the caller. diff --git a/skills/dpdata-inspect/scripts/validate_labeled_system.py b/skills/dpdata-inspect/scripts/validate_labeled_system.py new file mode 100755 index 000000000..f46b8d892 --- /dev/null +++ b/skills/dpdata-inspect/scripts/validate_labeled_system.py @@ -0,0 +1,401 @@ +#!/usr/bin/env python3 +"""Validate dpdata systems against the reusable labeled-data contract. + +The validator intentionally checks structure and finite values only. Unit and +stress conventions are producer contracts, so callers must state them with the +metadata options when running in strict mode. +""" + +from __future__ import annotations + +import argparse +import json +from dataclasses import asdict, dataclass, field +from pathlib import Path +from typing import TYPE_CHECKING, Any + +if TYPE_CHECKING: + from collections.abc import Mapping, Sequence + +import numpy as np + + +@dataclass +class ValidationReport: + """Machine-readable validation result for one or more systems.""" + + errors: list[str] = field(default_factory=list) + warnings: list[str] = field(default_factory=list) + summary: dict[str, Any] = field(default_factory=dict) + + @property + def ok(self) -> bool: + """Whether no contract violations were found.""" + return not self.errors + + def to_dict(self) -> dict[str, Any]: + """Return a JSON-serializable representation.""" + return {"ok": self.ok, **asdict(self)} + + +def _as_array( + data: Mapping[str, Any], key: str, errors: list[str] +) -> np.ndarray | None: + """Get *key* as an array, recording malformed values instead of raising.""" + if key not in data: + return None + try: + return np.asarray(data[key]) + except (TypeError, ValueError) as exc: + errors.append(f"{key}: cannot convert to an array ({exc})") + return None + + +def _check_finite(name: str, value: np.ndarray, errors: list[str]) -> None: + """Reject NaN and infinity in a numeric field.""" + if not np.issubdtype(value.dtype, np.number): + errors.append(f"{name}: expected numeric values, got {value.dtype}") + return + if not np.all(np.isfinite(value)): + errors.append(f"{name}: contains NaN or infinity") + + +def _check_shape( + name: str, + value: np.ndarray, + shape: tuple[int, ...], + errors: list[str], +) -> None: + """Check an exact array shape.""" + if value.shape != shape: + errors.append(f"{name}: shape {value.shape}, expected {shape}") + + +def validate_data( + data: Mapping[str, Any], + *, + require_labels: bool = True, + require_pbc: bool = False, + expected_type_map: Sequence[str] | None = None, + strict_metadata: bool = False, + energy_unit: str | None = None, + force_unit: str | None = None, + virial_unit: str | None = None, + stress_sign: str | None = None, + provenance: str | None = None, + producer: str | None = None, + producer_version: str | None = None, + parser_version: str | None = None, +) -> ValidationReport: + """Validate one dpdata ``System.data`` mapping. + + Parameters are deliberately explicit because dpdata cannot infer units, + stress/virial sign conventions, or the producer/version contract from the + arrays alone. Set ``strict_metadata`` to require those declarations. + """ + report = ValidationReport() + errors = report.errors + warnings = report.warnings + + missing = [ + key + for key in ("coords", "atom_types", "atom_names", "atom_numbs") + if key not in data + ] + if missing: + errors.append("missing required fields: " + ", ".join(missing)) + return report + + coords = _as_array(data, "coords", errors) + atom_types = _as_array(data, "atom_types", errors) + cells = _as_array(data, "cells", errors) + if coords is None or atom_types is None: + return report + + if coords.ndim != 3 or coords.shape[-1] != 3: + errors.append(f"coords: shape {coords.shape}, expected (nframes, natoms, 3)") + return report + nframes, natoms = coords.shape[:2] + _check_finite("coords", coords, errors) + + if atom_types.shape != (natoms,): + errors.append(f"atom_types: shape {atom_types.shape}, expected ({natoms},)") + elif not np.issubdtype(atom_types.dtype, np.integer): + errors.append(f"atom_types: expected integer values, got {atom_types.dtype}") + elif np.any(atom_types < 0): + errors.append("atom_types: contains a negative type index") + + atom_names = data["atom_names"] + atom_numbs = data["atom_numbs"] + try: + ntypes = len(atom_names) + counts = np.asarray(atom_numbs) + except (TypeError, ValueError) as exc: + errors.append(f"atom_names/atom_numbs: malformed ({exc})") + return report + if ntypes != len(counts): + errors.append( + f"atom_names/atom_numbs: lengths differ ({ntypes} versus {len(counts)})" + ) + if counts.ndim != 1 or not np.issubdtype(counts.dtype, np.integer): + errors.append("atom_numbs: expected a one-dimensional integer sequence") + elif np.any(counts < 0): + errors.append("atom_numbs: contains a negative atom count") + elif int(np.sum(counts)) != natoms: + errors.append( + f"atom_numbs: sum {int(np.sum(counts))}, expected natoms {natoms}" + ) + if atom_types.size and not ntypes: + errors.append("atom_names: empty type_map for non-empty atom_types") + elif ( + atom_types.size + and np.issubdtype(atom_types.dtype, np.integer) + and np.max(atom_types) >= ntypes + ): + errors.append("atom_types: index is outside atom_names/type_map") + if expected_type_map is not None and list(atom_names) != list(expected_type_map): + errors.append( + "atom_names: does not match type_map " + f"({list(atom_names)!r} versus {list(expected_type_map)!r})" + ) + + nopbc = bool(data.get("nopbc", False)) + if require_pbc and nopbc: + errors.append("nopbc: system is non-periodic while periodic cells are required") + if cells is None: + if not nopbc or require_pbc: + errors.append("cells: missing for a periodic system") + else: + _check_shape("cells", cells, (nframes, 3, 3), errors) + if cells.shape == (nframes, 3, 3): + _check_finite("cells", cells, errors) + if require_pbc: + try: + determinants = np.linalg.det(cells) + except (np.linalg.LinAlgError, TypeError, ValueError) as exc: + errors.append(f"cells: cannot compute determinants ({exc})") + else: + if np.any(np.isclose(determinants, 0.0)): + errors.append( + "cells: singular cell while periodic cells are required" + ) + + label_shapes = { + "energies": (nframes,), + "forces": (nframes, natoms, 3), + "virials": (nframes, 3, 3), + "atom_pref": (nframes, natoms), + } + if require_labels and "energies" not in data: + errors.append("energies: missing required energy labels") + for name, expected_shape in label_shapes.items(): + if name not in data: + continue + value = _as_array(data, name, errors) + if value is None: + continue + _check_shape(name, value, expected_shape, errors) + _check_finite(name, value, errors) + + if (require_labels or "energies" in data) and energy_unit is None: + warnings.append("energy unit was not declared") + if "forces" in data and force_unit is None: + warnings.append("force unit was not declared") + if "virials" in data: + if virial_unit is None: + warnings.append("virial/stress unit was not declared") + if stress_sign is None: + warnings.append("virial/stress sign convention was not declared") + if provenance is None: + warnings.append("raw-data provenance was not supplied") + if producer is None: + warnings.append("producer contract was not declared") + if producer_version is None: + warnings.append("producer version was not declared") + if parser_version is None: + warnings.append("parser version was not declared") + if provenance is not None: + provenance_path = Path(provenance) + if not provenance_path.is_file(): + errors.append(f"provenance: file does not exist ({provenance})") + elif provenance_path.stat().st_size == 0: + errors.append(f"provenance: file is empty ({provenance})") + + if strict_metadata: + metadata_missing = [] + if (require_labels or "energies" in data) and energy_unit is None: + metadata_missing.append("energy unit") + if "forces" in data and force_unit is None: + metadata_missing.append("force unit") + if "virials" in data and virial_unit is None: + metadata_missing.append("virial/stress unit") + if "virials" in data and stress_sign is None: + metadata_missing.append("virial/stress sign") + if provenance is None: + metadata_missing.append("provenance") + if producer is None: + metadata_missing.append("producer") + if producer_version is None: + metadata_missing.append("producer version") + if parser_version is None: + metadata_missing.append("parser version") + if metadata_missing: + errors.append( + "strict metadata is enabled; missing " + ", ".join(metadata_missing) + ) + + report.summary = { + "frames": nframes, + "atoms": natoms, + "types": ntypes, + "labeled": "energies" in data, + "forces": "forces" in data, + "virials": "virials" in data, + "periodic": not nopbc, + "atom_names": list(atom_names), + "producer": producer, + "producer_version": producer_version, + "parser_version": parser_version, + } + return report + + +def validate_systems( + systems: Sequence[Any], + **kwargs: Any, +) -> list[ValidationReport]: + """Validate systems and enforce one shared type map across a collection.""" + reports = [validate_data(system.data, **kwargs) for system in systems] + names: list[str] | None = None + for index, (system, report) in enumerate(zip(systems, reports, strict=True)): + current = list(system.data.get("atom_names", [])) + if names is None: + names = current + elif current != names: + report.errors.append( + "atom_names: differs from system 0 in a multi-system dataset " + f"(system {index}: {current!r}, system 0: {names!r})" + ) + return reports + + +def _parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Validate dpdata frame, label, atom-order, and metadata contracts." + ) + parser.add_argument("input", help="Input path accepted by dpdata") + parser.add_argument( + "-f", + "--format", + default="auto", + help="dpdata format (default: auto)", + ) + parser.add_argument( + "--multi", + action="store_true", + help="Load a directory containing multiple systems", + ) + parser.add_argument( + "--allow-unlabeled", + action="store_true", + help="Allow a dataset without energy labels", + ) + parser.add_argument( + "--require-pbc", + action="store_true", + help="Require nonsingular periodic cells for every frame", + ) + parser.add_argument( + "--type-map", + nargs="+", + help="Expected atom names in type-map order (also passed to the loader)", + ) + parser.add_argument("--energy-unit", help="Declared energy unit, e.g. eV") + parser.add_argument("--force-unit", help="Declared force unit, e.g. eV/A") + parser.add_argument("--virial-unit", help="Declared virial/stress unit") + parser.add_argument("--stress-sign", help="Declared virial/stress sign convention") + parser.add_argument( + "--provenance", + help="Path to a non-empty raw input or provenance record", + ) + parser.add_argument("--producer", help="Producer name, e.g. CP2K or VASP") + parser.add_argument("--producer-version", help="Producer version or release") + parser.add_argument("--parser-version", help="dpdata/parser version used") + parser.add_argument( + "--strict-metadata", + action="store_true", + help="Turn undeclared units, stress sign, and provenance into errors", + ) + parser.add_argument("--json", action="store_true", help="Emit JSON instead of text") + return parser.parse_args(argv) + + +def _load_systems(args: argparse.Namespace) -> list[Any]: + # Import lazily so validate_data can be used in a minimal Python environment. + from dpdata.system import LabeledSystem, MultiSystems, System + + loader_kwargs = {"type_map": args.type_map} + if args.multi: + collection = MultiSystems.from_file( + args.input, + fmt=args.format, + labeled=not args.allow_unlabeled, + **loader_kwargs, + ) + return list(collection.systems.values()) + cls = System if args.allow_unlabeled else LabeledSystem + return [cls(args.input, fmt=args.format, **loader_kwargs)] + + +def main(argv: Sequence[str] | None = None) -> int: + """CLI entry point.""" + args = _parse_args(argv) + try: + systems = _load_systems(args) + if not systems: + raise ValueError("no systems found") + reports = validate_systems( + systems, + require_labels=not args.allow_unlabeled, + require_pbc=args.require_pbc, + expected_type_map=args.type_map, + strict_metadata=args.strict_metadata, + energy_unit=args.energy_unit, + force_unit=args.force_unit, + virial_unit=args.virial_unit, + stress_sign=args.stress_sign, + provenance=args.provenance, + producer=args.producer, + producer_version=args.producer_version, + parser_version=args.parser_version, + ) + except Exception as exc: # noqa: BLE001 - expose parser failures as CLI errors + report = ValidationReport(errors=[f"load: {exc}"]) + reports = [report] + + result = { + "ok": all(report.ok for report in reports), + "systems": [report.to_dict() for report in reports], + } + if args.json: + print(json.dumps(result, indent=2, sort_keys=True)) + else: + status = "PASS" if result["ok"] else "FAIL" + print(f"{status}: validated {len(reports)} system(s)") + for index, report in enumerate(reports): + prefix = f"system {index}: " + for message in report.errors: + print(f"ERROR {prefix}{message}") + for message in report.warnings: + print(f"WARNING {prefix}{message}") + if report.summary: + summary = report.summary + print( + f" frames={summary['frames']} atoms={summary['atoms']} " + f"labeled={summary['labeled']} periodic={summary['periodic']}" + ) + return 0 if result["ok"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/dpdata-minimizer/SKILL.md b/skills/dpdata-minimizer/SKILL.md index 69eb03da7..35d705300 100644 --- a/skills/dpdata-minimizer/SKILL.md +++ b/skills/dpdata-minimizer/SKILL.md @@ -1,86 +1,13 @@ --- name: dpdata-minimizer -description: Minimize geometries with dpdata minimizer plugins via System.minimize(), including how minimizers relate to drivers (ASEMinimizer needs a dpdata Driver) and how to list supported minimizers (ase/sqm). Use when doing geometry optimization/minimization through dpdata Python API. +description: Compatibility entry point for the developer-facing dpdata minimizer reference. Use dpdata-extend for new minimizer implementations and System.minimize workflows. +compatibility: Requires dpdata; minimizers may require optional dependencies or external executables. +metadata: + repository: https://github.com/deepmodeling/dpdata --- -# dpdata-minimizer +# dpdata minimizer compatibility entry point -Use dpdata “minimizer plugins” to **optimize/minimize geometry** and return a `dpdata.LabeledSystem`. - -## Key idea - -- A **Minimizer** performs geometry optimization (updates coordinates) and returns labeled results. -- dpdata exposes this as: - -```python -System.minimize(*args, minimizer: str|Minimizer, **kwargs) -> LabeledSystem -``` - -## List supported minimizer keys (runtime) - -```python -from dpdata.driver import Minimizer - -print(sorted(Minimizer.get_minimizers().keys())) -``` - -In the current dpdata repo, minimizer keys include: - -- `ase` -- `sqm` - -## Relationship to drivers (important) - -Some minimizers require a **dpdata Driver object**. - -Example: `ASEMinimizer` takes a dpdata `Driver` in its constructor: - -- `minimizer="ase"` requires `driver=` (e.g. the ASE driver wrapping an ASE calculator). - -So you generally do: - -1. Construct a driver -1. Construct a minimizer (or let dpdata do it by passing the right kwargs) -1. Call `System.minimize(...)` - -## Runnable example: ASE minimizer with an ASE calculator - -Use uv inline script metadata so the example runs reproducibly with `uv run`. - -```python -# /// script -# requires-python = ">=3.12" -# dependencies = [ -# "dpdata", -# "numpy", -# "ase", -# ] -# /// - -import numpy as np -from ase.calculators.emt import EMT - -from dpdata.driver import Driver -from dpdata.system import System - -open("tmp.xyz", "w").write("""2\n\nH 0 0 0\nH 0 0 0.74\n""") - -sys = System("tmp.xyz", fmt="xyz") - -# Build a dpdata driver that can provide energies/forces to ASE optimizers. -ase_driver = Driver.get_driver("ase")(calculator=EMT()) - -# Minimize using the ASE minimizer plugin. -# NOTE: ASEMinimizer expects `driver` (not `calculator`) as input. -ls = sys.minimize(minimizer="ase", driver=ase_driver, fmax=0.05, max_steps=5) - -print("coords", np.array(ls.data["coords"]).shape) -print("energies", np.array(ls.data["energies"])) -print("forces", np.array(ls.data["forces"]).shape) -``` - -## Notes / gotchas - -- `System.minimize(...)` accepts either a minimizer key string or a Minimizer object. -- If you previously used `System.predict(driver="ase", calculator=...)`, be aware that minimization is different: you need to pass a **driver** into the minimizer (ASEMinimizer does not accept `calculator=`). -- `sqm` minimizer requires AmberTools `sqm` executable and typically won’t be runnable in CI. +The developer reference now lives at +[`dpdata-extend/references/minimizer.md`](../dpdata-extend/references/minimizer.md). +Use [`dpdata-extend`](../dpdata-extend/SKILL.md) for new API-extension work. diff --git a/skills/dpdata-plugin/SKILL.md b/skills/dpdata-plugin/SKILL.md index 818cc8f92..467642a7a 100644 --- a/skills/dpdata-plugin/SKILL.md +++ b/skills/dpdata-plugin/SKILL.md @@ -1,119 +1,13 @@ --- name: dpdata-plugin -description: Create and install dpdata plugins (especially custom Format readers/writers) using Format.register(...) and pyproject.toml entry_points under 'dpdata.plugins'. Use when extending dpdata with new formats or distributing plugins as separate Python packages. +description: Compatibility entry point for the developer-facing dpdata format-plugin reference. Use dpdata-extend for new format plugins and entry-point packages. +compatibility: Requires dpdata; external plugins may have their own dependencies. +metadata: + repository: https://github.com/deepmodeling/dpdata --- -# dpdata-plugin +# dpdata plugin compatibility entry point -dpdata loads plugins in two ways: - -1. **Built-in plugins** in `dpdata.plugins.*` (imported automatically) -1. **External plugins** exposed via Python package entry points: `dpdata.plugins` - -This skill focuses on **external plugin packages**, the recommended way to add new formats without modifying dpdata itself. - -## What can be extended? - -Most commonly: add a new **Format** (file reader/writer) via: - -```python -from dpdata.format import Format - - -@Format.register("myfmt") -class MyFormat(Format): ... -``` - -## How dpdata discovers plugins - -dpdata imports `dpdata.plugins` during normal use (e.g. `dpdata.system` imports it). That module: - -- imports every built-in module in `dpdata/plugins/*.py` -- then loads all **entry points** in group `dpdata.plugins` - -So an external plugin package only needs to ensure that importing the entry-point target triggers the `@Format.register(...)` side effects. - -## Minimal external plugin package (based on plugin_example/) - -### 1) Create a new Python package - -Example layout: - -```text -dpdata_random/ - pyproject.toml - dpdata_random/ - __init__.py -``` - -### 2) Implement and register your Format - -In `dpdata_random/__init__.py` (shortened example): - -```python -from __future__ import annotations - -import numpy as np -from dpdata.format import Format - - -@Format.register("random") -class RandomFormat(Format): - def from_system(self, file_name, **kwargs): - nframes = int(file_name) - return { - "atom_numbs": [20], - "atom_names": ["X"], - "atom_types": np.zeros(20, dtype=int), - "cells": np.repeat(np.eye(3)[None, ...], nframes, axis=0) * 100.0, - "coords": np.random.rand(nframes, 20, 3) * 100.0, - "orig": np.zeros(3), - "nopbc": False, - } -``` - -Return dicts must match dpdata’s expected schema (cells/coords/atom_names/atom_types/...). -Here `file_name` is the argument dpdata passes into `from_system`; this example interprets it as an integer frame count. - -### 3) Expose an entry point - -In `pyproject.toml`: - -```toml -[project] -name = "dpdata_random" -version = "0.0.0" -dependencies = ["numpy", "dpdata"] - -[project.entry-points.'dpdata.plugins'] -random = "dpdata_random:RandomFormat" -``` - -Any importable target works; this pattern points directly at the class. - -### 4) Install and test - -In a clean env (recommended via `uv`): - -```bash -uv venv -. .venv/bin/activate -uv pip install -e . dpdata numpy -python - <<'PY' -import dpdata -from dpdata.format import Format - -# importing dpdata will load entry points (dpdata.plugins) -print('random' in Format.get_formats()) -PY -``` - -If it prints `True`, your plugin was discovered. - -## Debug checklist - -- Did you install the plugin package into the same environment where you run dpdata? -- Does `pyproject.toml` contain `[project.entry-points.'dpdata.plugins']`? -- Does importing the entry point module/class execute the `@Format.register(...)` decorator? -- Before choosing a key like `random`, check `Format.get_formats()` to avoid colliding with an existing built-in or previously loaded plugin key. -- If using `uv run`, remember each command runs in its own environment unless you’re in a `uv` project (or you rely on `uv run --with ...`). +The developer reference now lives at +[`dpdata-extend/references/format-plugin.md`](../dpdata-extend/references/format-plugin.md). +Use [`dpdata-extend`](../dpdata-extend/SKILL.md) for new API-extension work. diff --git a/skills/dpdata-workflows/SKILL.md b/skills/dpdata-workflows/SKILL.md new file mode 100644 index 000000000..f8f24f061 --- /dev/null +++ b/skills/dpdata-workflows/SKILL.md @@ -0,0 +1,25 @@ +# dpdata workflow routing + +Use this page to choose the smallest dpdata skill that matches the user's +intent. The workflow skills own user-facing procedures; API and extension +material lives under `dpdata-extend`. + +| User intent | Skill | Load next | +| --------------------------------------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | +| Convert a producer output, export a dataset, or prepare training data | [`dpdata-convert`](../dpdata-convert/SKILL.md) | [`semantic-validation.md`](../dpdata-convert/references/semantic-validation.md), then the producer reference | +| Inspect, validate, merge, or filter a dataset before it is consumed | [`dpdata-inspect`](../dpdata-inspect/SKILL.md) | `validate_labeled_system.py` for deterministic checks | +| Add a format, driver, or minimizer | [`dpdata-extend`](../dpdata-extend/SKILL.md) | The matching developer reference | + +A conversion request is incomplete until the output contract is checked. Ask +whether the source is labeled, whether it contains one system or many, and +which producer/version generated it. Preserve the raw input and record the +producer contract, units, periodic-cell assumptions, atom ordering, and any +missing-label policy. Do not infer scientific conventions from a file name. + +Load producer references only after the producer is known. They document the +contract to check and link to the parser or producer documentation; they do not +replace parser implementation or project-specific scientific choices. + +The old `dpdata-cli`, `dpdata-driver`, `dpdata-minimizer`, and `dpdata-plugin` +paths remain compatibility entry points. New requests should use the workflow +skills above. diff --git a/tests/test_skill_validate_labeled_system.py b/tests/test_skill_validate_labeled_system.py new file mode 100644 index 000000000..f7b4294bc --- /dev/null +++ b/tests/test_skill_validate_labeled_system.py @@ -0,0 +1,123 @@ +from __future__ import annotations + +import importlib.util +import sys +import unittest +from pathlib import Path + +import numpy as np + +_SCRIPT = ( + Path(__file__).parents[1] + / "skills" + / "dpdata-inspect" + / "scripts" + / "validate_labeled_system.py" +) +_SPEC = importlib.util.spec_from_file_location( + "dpdata_validate_labeled_system", _SCRIPT +) +assert _SPEC is not None and _SPEC.loader is not None +_VALIDATOR = importlib.util.module_from_spec(_SPEC) +sys.modules[_SPEC.name] = _VALIDATOR +_SPEC.loader.exec_module(_VALIDATOR) + + +class TestValidateLabeledSystem(unittest.TestCase): + def _data(self): + return { + "atom_numbs": [1, 1], + "atom_names": ["H", "O"], + "atom_types": np.array([0, 1], dtype=int), + "orig": np.zeros(3), + "cells": np.repeat(np.eye(3)[None, ...], 2, axis=0), + "coords": np.zeros((2, 2, 3)), + "energies": np.array([-1.0, -0.9]), + "forces": np.zeros((2, 2, 3)), + "virials": np.zeros((2, 3, 3)), + } + + def test_valid_labeled_data(self): + report = _VALIDATOR.validate_data( + self._data(), + expected_type_map=["H", "O"], + energy_unit="eV", + force_unit="eV/A", + virial_unit="eV", + stress_sign="virial", + provenance=__file__, + producer="test", + producer_version="1", + parser_version="1", + strict_metadata=True, + ) + self.assertTrue(report.ok, report.errors) + self.assertEqual(report.summary["frames"], 2) + + def test_missing_energy_label_is_an_error(self): + data = self._data() + del data["energies"] + report = _VALIDATOR.validate_data(data) + self.assertFalse(report.ok) + self.assertIn("energies: missing required energy labels", report.errors) + + def test_alignment_and_atom_type_errors_are_reported(self): + data = self._data() + data["forces"] = np.zeros((1, 2, 3)) + data["atom_types"] = np.array([0, 2], dtype=int) + report = _VALIDATOR.validate_data(data) + self.assertFalse(report.ok) + self.assertTrue(any("forces: shape" in error for error in report.errors)) + self.assertIn("atom_types: index is outside atom_names/type_map", report.errors) + + def test_provenance_check_is_independent_of_parser_version(self): + report = _VALIDATOR.validate_data( + self._data(), provenance=__file__, parser_version=None + ) + self.assertTrue(report.ok, report.errors) + + report = _VALIDATOR.validate_data( + self._data(), provenance=None, parser_version="1" + ) + self.assertTrue(report.ok, report.errors) + + def test_empty_cells_are_rejected(self): + data = self._data() + data["cells"] = np.empty((0,)) + report = _VALIDATOR.validate_data(data, require_pbc=True) + self.assertFalse(report.ok) + self.assertTrue(any("cells: shape" in error for error in report.errors)) + + def test_nonnumeric_cells_are_reported(self): + data = self._data() + data["cells"] = np.full((2, 3, 3), "invalid") + report = _VALIDATOR.validate_data(data, require_pbc=True) + self.assertFalse(report.ok) + self.assertTrue( + any("cells: expected numeric" in error for error in report.errors) + ) + + def test_nopbc_is_rejected_when_periodic_cells_are_required(self): + data = self._data() + data["nopbc"] = True + report = _VALIDATOR.validate_data(data, require_pbc=True) + self.assertFalse(report.ok) + self.assertIn( + "nopbc: system is non-periodic while periodic cells are required", + report.errors, + ) + + def test_multi_system_type_map_must_match(self): + first = type("System", (), {"data": self._data()})() + second_data = self._data() + second_data["atom_names"] = ["O", "H"] + second = type("System", (), {"data": second_data})() + reports = _VALIDATOR.validate_systems([first, second]) + self.assertFalse(reports[1].ok) + self.assertTrue( + any("multi-system dataset" in error for error in reports[1].errors) + ) + + +if __name__ == "__main__": + unittest.main()