All projectsProject 03 of 22 · Build Agents That Remember

PipeOps

PipeOps is a test bench for screenpipe's Workflows.

The project

PipeOps is a test bench for screenpipe's Workflows.

screenpipe records your screen. Its Workflows feature uses AI agents to read that recording and spot the tasks you repeat, so you can automate them. But when someone changes those agents, nobody can tell whether they got better, worse, or just more expensive.

So we built an exam. We made up a person's week on a computer, so we know exactly which tasks the agents should find. screenpipe's real agents take the exam, we grade their answers, and we count what it cost.

Then we used it to improve screenpipe. With our fixes, the agents got the same score with half the AI calls and less than half the time, and they no longer run at all when nothing new has happened. Whether the workflows they write are better is too early to tell: two test runs are not enough.

Everything goes back to screenpipe as five pull requests, the exam included.

How it’s built

The exam is wfeval (Python, uv). It builds screenpipe at a8aaee2d6 with one added route, /eval/seed (404 unless SCREENPIPE_EVAL_SEED=1), and writes a deterministic synthetic history into an isolated engine (own data dir, 127.0.0.1:3036, capture off). The four shipped Workflows agents run through Pi on Claude Sonnet 5 via screenpipe cloud, with tools in a Docker container that has no network. Grading covers 14 checks against the answer key, plus every cited capture must exist. Cost comes from the request logs (calls, time, tokens, refused saves, cut replies). Deliberately damaged workflows confirm the checks catch errors. A blind judge (Claude Sonnet agents, both orders) compares the documents. Fixes found this way: whole-second citations were refused, text was cut at 400 characters, screenpipe cloud ignores Pi's max_completion_tokens, and full agent cycles ran with nothing new. Our engine changes are 12 commits under screenpipe/. The PipeOps app (React, TypeScript, Tailwind, screenpipe's design system) presents it offline from the real run data: npm run dev.

🇫🇷 made with love <3 by @nass