A primitive token library used to read and write from a Uint8Array.
Although it is possible to use this module directly, it is primarily designed to be used with strtok3 tokenizer.
Node.js 18 is supported for running the published package; it is not a supported
build environment. The test toolchain requires Node.js 22.12 or later. CI builds
on Node.js 24 and runs the tests on Node.js 22, 24 and 26.
Development
Use Node.js 24 and the Yarn version pinned in package.json to install development
dependencies and build from source:
corepack enableyarn install --immutableyarn test
yarn test type-checks the sources and runs the Vitest suite against the
TypeScript sources. yarn test:watch runs Vitest in watch mode, and
yarn test-coverage writes text and LCOV reports to coverage. yarn clean
removes dist and coverage.
yarn lint checks the Markdown and Biome rules. yarn format:check verifies
Biome formatting, and yarn fix applies formatting, import organization and
safe lint fixes.