ToolSmith

Before shipping your AI agent, test whether it knows how to use its tools.

ToolSmith is a local-first CLI for testing, linting, scoring, comparing, and reporting on AI agent tool use before real tool behavior reaches users.

Repository: github.com/landon-personal/toolsmith

Quickstart

ToolSmith is published to npm as @landon-personal/toolsmith, and the npm latest dist-tag points to v1.0.7. The GitHub repository is public, GitHub Pages is live at https://landon-personal.github.io/toolsmith/, and the CLI binary command remains toolsmith.

Fresh user flow:

npm install -g @landon-personal/toolsmith
mkdir demo-agent-tools
cd demo-agent-tools
toolsmith init
toolsmith lint .
toolsmith eval .
toolsmith report

Local development:

git clone https://github.com/landon-personal/toolsmith.git
cd toolsmith
npm install
npm run compile
npm test
npm run dev -- --help
npm run dev -- lint examples/calendar-email
npm run dev -- eval examples/calendar-email
npm run dev -- report

Commands

npm run dev -- --help
npm run dev -- --version
npm run dev -- init
npm run dev -- lint examples/calendar-email
npm run dev -- eval examples/calendar-email
npm run dev -- report
npm run dev -- report --format markdown
npm run dev -- report --format html
npm run dev -- compare .toolsmith/runs/latest.json .toolsmith/runs/latest.json
npm run dev -- import openapi examples/openapi/tiny-api.json --out examples/openapi/tools.generated.json

Stable CLI commands:

• toolsmith --help
• toolsmith --version
• toolsmith init
• toolsmith lint <path>
• toolsmith eval <path>
• toolsmith report
• toolsmith compare <baseline-run> <current-run>
• toolsmith import openapi <path> --out <path>

Documentation

• Quickstart
• Installation
• Writing tools
• Writing tasks
• Running evals
• Reports
• CI mode
• Importers
• Coding agents
• Safety
• Roadmap
• Public release plan
• v1.0.0 release notes
• Troubleshooting
• Schema

Safety

ToolSmith uses a deterministic local mock agent by default. v1.1.0 adds an optional OpenAI provider for tool-selection evals only. It does not execute imported APIs, send emails, edit calendars, charge money, delete data, deploy code, publish packages, print secrets, or perform real tool side effects.

Future Install Goal

Future npm install commands are expected to be:

npm install -g @landon-personal/toolsmith
npx @landon-personal/toolsmith@latest --help
toolsmith --help

These commands should not be used until npm publishing is explicitly approved and completed.