Test, lint, and report on AI agent tool use before shipping.
npm install failsCheck your Node.js version first:
node --version
ToolSmith currently requires Node 20 or newer. If dependencies still fail to install, remove any partial install and retry from a clean checkout:
npm install
ToolSmith is published to npm as @landon-personal/toolsmith. If you are testing an unpublished patch from this repository, use the local development commands instead.
If commands fail with syntax or module errors, use Node 20 or newer. The package metadata declares:
"engines": {
"node": ">=20"
}
Install the published CLI or run it with npx:
npm install -g @landon-personal/toolsmith
toolsmith --help
npx @landon-personal/toolsmith@latest --help
For the local checkout, run commands through npm:
npm run dev -- --help
package:check Failsnpm run package:check compiles the CLI, creates a local npm tarball, installs it into a temporary directory, runs toolsmith --help, runs toolsmith --version, and cleans up on success.
If it fails:
npm run compilenpm testdist/cli.js was generated locallyRun an eval before reporting:
toolsmith eval .
toolsmith report
If you do not have tools.json and tasks.json yet, run toolsmith init first.
Markdown and HTML reports are generated only when requested:
npm run dev -- report --format markdown
npm run dev -- report --format html
By default these write report.md and report.html. They are generated files and should not be committed unless a future task explicitly asks for fixtures.
compare needs two saved eval run JSON files:
npm run dev -- compare baseline.json .toolsmith/runs/latest.json
Create .toolsmith/runs/latest.json by running:
npm run dev -- eval examples/calendar-email
Do not commit .toolsmith/runs/; it is generated local output.
--fail-under Exits 1This can be intentional. eval --fail-under <score> exits non-zero when the current score is below the threshold:
npm run dev -- eval examples/calendar-email --fail-under 80
The starter example currently includes intentionally tricky failures, so high thresholds may fail.
Use a JSON file path and an explicit output path:
npm run dev -- import openapi examples/openapi/tiny-api.json --out examples/openapi/tools.generated.json
The importer reads and writes local JSON only. It does not call imported APIs.
Prefer npm scripts and Node paths instead of Unix-only shell assumptions. Commands in this repo use forward slashes in examples because npm and Node handle them well in typical shells, but source and tests should use Node path and fs APIs.
ToolSmith is published to npm as @landon-personal/toolsmith:
npm install -g @landon-personal/toolsmith
npx @landon-personal/toolsmith@latest --help
toolsmith --help
The currently published npm package is v1.0.7. Publishing future patches requires explicit approval.