Nick Evans

AI vs Baseline Testing Study: LLM-assisted failure triage

A published study · TypeScript, Playwright, Python

A test fails overnight. Someone has to decide whether the product, the test or the environment is at fault, or whether the test is simply flaky.

I wanted to know whether a language model does that better than a simple baseline, so I first built a test application: a deliberately defective application and API with twenty-two cataloged defects and four injectable flakes. Each defect carries the classification a correct triage system should assign, so every failure it produces has a known right answer. That repository contains no AI, so the application shares no components with the classifiers being measured.

The model was also weakest on the deliberately ambiguous cases, where a model would be expected to help most. Two cross-run frequency hypotheses that seemed reasonable contributed nothing, measured three separate ways.

The labeled corpus is published as a checksummed release and cited by tag, so the numbers can be reproduced. The deterministic baseline runs from a clean checkout with one command and no credentials. An earlier version of this result was measured against a corpus that was later regenerated, which made the published figure unverifiable. The repository documents the fix alongside the superseded numbers.

ai-qa-proving-ground (the defective application and its corpus) · ai-qa-failure-triage (the classifier and the scoring)

Job Tracker: full-stack and self-hosted

Built with Anthropic Claude Code · FastAPI, MariaDB, React

Tracks job applications end to end: company, role, source, status, salary range, dates, notes, next action and contacts. A FastAPI service in Python with SQLAlchemy models and Alembic migrations, running against MariaDB over the MySQL wire protocol, with a React single-page front end built in Vite.

Two repositories deployed independently, with requirements and architecture in a third that both consume as a git submodule, so there is one copy rather than several that can drift. Two Chrome extensions sit alongside them: one scrapes a posting and files it, the other marks an existing application closed in a single click. The whole thing runs on a self-hosted Debian machine on my local network with a nightly backup to Backblaze, with work tracked in Jira.

backend · frontend · shared docs · import extension · close-posting extension

Jira Command-Line Client: replacing a metered integration with a small script

Built with Anthropic Claude Code · Python, no dependencies

I was driving the Jira board for the Job Tracker project through Atlassian's hosted Rovo MCP server. It worked, but it meters against Rovo credits and is not offered on the Free plan, so keeping one integration alive meant paying for a tier I had no other use for. The REST API underneath it is available on every plan, unmetered.

So I built, with Claude Code, a short Python program that talks to it directly: a single file, no dependencies, any Python 3.9 or later. It also turned out better in two ways. It runs anywhere there is a shell (a terminal, an editor, a CI job, more than one AI coding agent), whereas an MCP server only works where it has been configured. And it can be read, tested and changed, which a hosted service cannot.

I chose no dependencies because a tool I use every few weeks can break between uses if its dependencies change. transition takes a status name rather than Jira's internal transition id and makes the lookup round trip itself. When the status is not reachable from where the issue currently sits, the error lists the ones that are.

Jira Command-Line Client on GitHub: the repository, with unit tests run in GitHub Actions

Job Search Workflow: a manager agent and its subagents

Built with Anthropic Claude Code · skills, subagents, a job board that refreshes itself

Applying for a job is a set of repeatable tasks: check whether a posting is the employer's own, judge whether the role fits, tailor a résumé, draft a cover letter, finalize it. I built a Claude Code skill for each one, and a manager agent that hands each step to a subagent running the matching skill and collects what comes back.

The manager keeps a small state file for every job, and it is the only thing I talk to. When a subagent has a question or needs a decision, it goes back to the manager, which puts it on an HTML job board I keep open in a browser. The board refreshes itself and sorts jobs into waiting on me, agents working and finished, so I can see what needs me without scrolling through a chat. I can hand it several postings at once and each one moves through the steps on its own.

Nothing is applied for, published or sent without my explicit approval at that step.

job-application-workflow : the manager, subagents, skills and board, with a demo and no personal data