Skip to content

Proposal: experimental node:json module with lazy views and schema-driven decoding #67504

Open

Description

@lemire

What is the problem this feature will solve?

Node.js programs often parse large JSON documents to read only a few fields: API responses, configuration and manifest files, log records, NDJSON streams. JSON.parse() must build a JavaScript object for every value in the document, even the ones that are never read. Data read from files or sockets also arrives as bytes, so it must first be decoded to a string with buffer.toString(), which adds to the cost.

Node.js already bundles simdjson (deps/simdjson, used for configuration files and the inspector). The simdjson library parses JSON at gigabytes per second straight from UTF-8 bytes, but it is not exposed to users.

What is the feature you are proposing to solve the problem?

An experimental node:json module, behind --experimental-json, with two functions: lazy views over a parsed document, and decoding into objects shaped by a schema. Both ideas are well established in other ecosystems and in npm packages (see Prior work below). The API is closest to the Python library fastpysimdjson (lazy views, typed decoding).

json.parse(input) returns read-only, lazy views. The document is parsed at once, but JavaScript values are created only for the parts that are read:

const { parse } = require('node:json');

const doc = parse(fs.readFileSync('twitter.json'));  // Buffer, no toString()
for (const status of doc.get('statuses')) {
  console.log(status.get('id'), status.get('user').get('screen_name'));
}
doc.atPointer('/statuses/0/user/name');   // JSON Pointer (RFC 6901)
doc.get('search_metadata').toObject();    // convert a subtree, like JSON.parse

JSONObject views are Map-like (get, has, size, keys, values, entries, iteration). JSONArray views have at(), length and iteration. Both convert fully with toJSON(), so JSON.stringify() accepts them.

json.decode(input, schema) builds plain objects shaped by a JSON Schema, straight from the parsed document. It checks types as it goes, applies defaults, and skips the properties the schema does not name:

const { decode } = require('node:json');

const User = {
  type: 'object',
  properties: {
    id: { type: 'integer' },
    login: { type: 'string' },
    followers: { type: 'integer', default: 0 },
  },
  required: ['id', 'login'],
};

decode(data, { type: 'array', items: User });
// TypeError [ERR_JSON_SCHEMA_VALIDATION]: Expected integer, got string at $[3].id

It supports a subset of JSON Schema: type, properties, required, additionalProperties, items, enum, const, default, nullable, and local $ref (so recursive schemas work). Any other validation keyword is rejected rather than silently ignored.

Both functions accept a string, a Buffer, a TypedArray, a DataView or an ArrayBuffer.

Prior work

Lazy access:

  • In Python, pysimdjson and cysimdjson expose simdjson documents as lazy proxies, and fastpysimdjson's parse returns Mapping and Sequence views.
  • In Node.js, the npm package simdjson offers lazyParse(), with values read through valueForKeyPath('foo.bar[1]'). Its last release (0.9.2) dates from 2021 and bundles simdjson 0.9.2.
  • everything-json is a simdjson-based parser that does not block the event loop. It parses to a tape, off the main thread with JSON.parseAsync(), and gives level-by-level access with get(), RFC 6901 pointers with path(), and full conversion with toObject() or toObjectAsync(). Its README notes that reading from a Buffer is about three times faster than from a string, and that full conversion is slower than JSON.parse().

Typed decoding:

  • In Python, msgspec decodes JSON into typed Structs, pydantic validates with its own parser (jiter), and fastpysimdjson decodes dataclasses straight from the simdjson tape.
  • In Node.js, Ajv's compileParser() generates a parser that validates while parsing, and fails on properties not in the schema unless additionalProperties: true is set. It supports JSON Type Definition (JTD) schemas only, and its documentation describes its performance as comparable to JSON.parse() on valid input.
  • typia's json.assertParse<T>() parses then checks the result against a TypeScript type, using a compile-time transform.

The proposal brings these ideas into Node.js on top of the simdjson it already ships: lazy access with JavaScript-friendly views, and a one-pass, schema-driven decoder for standard JSON Schema that projects away the fields it does not need.

Prototype and measurements

A working prototype is at https://github.com/lemire/node/tree/json-lazy-views (C++ binding over the simdjson DOM tape, docs and tests). On an Intel Xeon Gold 6338 (Linux), reading the id and user.screen_name of every status in twitter.json (631 kB):

Method Time (µs)
JSON.parse(buffer.toString()) 1994
JSON.parse(string) 1362
json.parse(string), lazy 481
json.parse(buffer), lazy 384
json.decode(buffer, schema) 246

Decoding six fields of each status and four of its user, the case benchmarked by fastpysimdjson, takes 392 µs with json.decode() against 1703 µs with JSON.parse(buffer.toString()).

This module is not a faster JSON.parse(). To convert a whole document, V8's JSON.parse() is faster (2.7 ms against 4.2 ms on citm_catalog.json), and it should stay the default choice.

Open questions

  • Name and placement. Is node:json the right name, or should this live elsewhere?
  • Projection by default. decode() drops undeclared properties unless additionalProperties: true is set. That is the main source of its speed, but it differs from JSON Schema validation, where an absent additionalProperties allows anything. (Ajv's JTD parser makes the stricter choice and rejects them.)
  • Schema format. A JSON Schema subset is a standard starting point. Should more of it be supported (anyOf, minimum, …), or should JTD, as in Ajv, be considered?
  • View ergonomics. Views use get(key) rather than property access, because a Proxy would cost most of the speed; everything-json reaches the same conclusion about its proxify().
  • Known differences from JSON.parse(). A number whose exponent overflows the double range (1e309) is a SyntaxError, not Infinity; this needs a simdjson change, which I can make. An unpaired surrogate escape is a SyntaxError, and a leading UTF-8 BOM is accepted.

I am a simdjson maintainer and would be glad to do the work, including the simdjson changes. Is there interest in this feature?

What alternatives have you considered?

  • Leaving it to npm packages. The packages above show the demand, but native addons need a compiler at install time or prebuilt binaries for each platform and Node.js version, and the most direct binding, simdjson, has not been released since 2021. Node.js already ships simdjson, so exposing it adds no new dependency and keeps it up to date.
  • Validating after JSON.parse() (Ajv's validate(), Zod, TypeBox, typia's assertParse). These check the objects that JSON.parse() has already built, so they add to its cost rather than avoid it, and they cannot skip the fields they do not need.
  • Ajv's JTD compileParser() is the closest one-pass design. It is written in JavaScript, which makes it portable but bounds it near JSON.parse() speed, and it does not accept JSON Schema.
  • Streaming parsers reduce memory use but are slower than JSON.parse() for documents that fit in memory.

Activity

  1. Renegade334 commented on Nov 16, 2026

    @Renegade334
    Member

    #61577 staled out, but there was an associated PR at #61855 that did not make it to fruition. #62598 is related.

  2. anonrig commented on Nov 16, 2026

    @anonrig
    Member

    Sounds like a great idea.

  3. lemire commented on Nov 16, 2026

    @lemire
    MemberAuthor

    @Renegade334 Thanks.

    It has been tried before, but I think we can get it done this time.

  4. mcollina commented on Nov 16, 2026

    @mcollina
    SponsorMember

    I support the idea! I'm not sure if I would use a module or just place it in util... but this is a detail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

feature requestIssues requesting new Node.js features.

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions