ToolSmith

Test, lint, and report on AI agent tool use before shipping.

View the Project on GitHub landon-personal/toolsmith

Troubleshooting

npm install fails

Check your Node.js version first:

node --version

ToolSmith currently requires Node 20 or newer. If dependencies still fail to install, remove any partial install and retry from a clean checkout:

npm install

ToolSmith is published to npm as @landon-personal/toolsmith. If you are testing an unpublished patch from this repository, use the local development commands instead.

Node Version Problems

If commands fail with syntax or module errors, use Node 20 or newer. The package metadata declares:

"engines": {
  "node": ">=20"
}

Command Not Found

Install the published CLI or run it with npx:

npm install -g @landon-personal/toolsmith
toolsmith --help
npx @landon-personal/toolsmith@latest --help

For the local checkout, run commands through npm:

npm run dev -- --help

package:check Fails

npm run package:check compiles the CLI, creates a local npm tarball, installs it into a temporary directory, runs toolsmith --help, runs toolsmith --version, and cleans up on success.

If it fails:

Generated Reports Not Found

Run an eval before reporting:

toolsmith eval .
toolsmith report

If you do not have tools.json and tasks.json yet, run toolsmith init first.

Markdown and HTML reports are generated only when requested:

npm run dev -- report --format markdown
npm run dev -- report --format html

By default these write report.md and report.html. They are generated files and should not be committed unless a future task explicitly asks for fixtures.

Compare Cannot Find Run Files

compare needs two saved eval run JSON files:

npm run dev -- compare baseline.json .toolsmith/runs/latest.json

Create .toolsmith/runs/latest.json by running:

npm run dev -- eval examples/calendar-email

Do not commit .toolsmith/runs/; it is generated local output.

--fail-under Exits 1

This can be intentional. eval --fail-under <score> exits non-zero when the current score is below the threshold:

npm run dev -- eval examples/calendar-email --fail-under 80

The starter example currently includes intentionally tricky failures, so high thresholds may fail.

OpenAPI Import File Path Issues

Use a JSON file path and an explicit output path:

npm run dev -- import openapi examples/openapi/tiny-api.json --out examples/openapi/tools.generated.json

The importer reads and writes local JSON only. It does not call imported APIs.

Windows Path Notes

Prefer npm scripts and Node paths instead of Unix-only shell assumptions. Commands in this repo use forward slashes in examples because npm and Node handle them well in typical shells, but source and tests should use Node path and fs APIs.

Published npm Package

ToolSmith is published to npm as @landon-personal/toolsmith:

npm install -g @landon-personal/toolsmith
npx @landon-personal/toolsmith@latest --help
toolsmith --help

The currently published npm package is v1.0.7. Publishing future patches requires explicit approval.