Project500
2024iamwil

forestfriends

We wrote a zine on System evals for LLM-driven apps. Lots of people building impressive demos with AIs, but to get it working well in production over time (maintainable), you need some kind of system eval. It's like some kind of open secret, since lots of people are still floating on vibes-based engineering and looks-good-to-me@K metrics. https://forestfriends.tech Sri and I wrote it as a way to collaborate after doing a podcast together, which made no money. Picked a topic that people seemed to be interested in. Did the whole customer dev thing, and honestly, we were unsure if it'd make any money at all. Representing the AI as a shoggoth is from that meme, and we merely thought juxtaposing it with some furry animals was funny. But it turns out people like it. It introduces system evals without jargon, and frames how to get started with evals for AI engineers that moved into the space from other kinds of engineering. It feels pretty good when people buy it and say they like it.

AppAIFinance