Testing new tech in public: the one-week spike, decided
New tools arrive faster than anyone can evaluate them, so most teams either chase everything or freeze and chase nothing. We do neither. We run a one-week spike: a tightly boxed test of a new tool against a real task, in public, ending in a written keep-or-kill decision. This post is the method, including the part where we publish the failures.
What you'll learn
- Why "evaluate it later" means never, and why chasing every release is worse
- The one-week spike: scope, box, and the single question it answers
- How we test in public without leaking client work
- The keep-kill-park decision and why "park" is the most useful verdict
- What a steady spike habit makes possible over a year
The old way: hype, then nothing
A new model drops. The timeline calls it the end of everything. A team either spends two weeks playing with it on toy problems and learns nothing transferable, or bookmarks it for "when we have time," which is a date that never arrives.
Both paths share a flaw: no decision. The toy-problem path produces a vague good feeling and no commitment. The bookmark path produces guilt and no commitment. A year later the team has opinions about forty tools and a workflow that uses three, none of which they chose deliberately.
The honest problem is that evaluation feels expensive, so it gets deferred, so it competes with shipping and loses. The fix is not more discipline. It is making evaluation cheap enough that it stops competing.
The new way: box the test to one week
A spike is a time-boxed experiment with a fixed end and a forced decision. Ours run one week, never longer, and they answer exactly one question: does this tool do a real task we already do, better than what we use now.
The constraints are the point.
- **One real task, not a demo.** We pick something we genuinely do every week. A real transcript to clean, a real image to generate, a real scrape to run. Toy problems flatter every tool. Real tasks expose them.
- **A baseline we already trust.** We do not test a new tool in a vacuum. We run it against the thing it would replace, on the same input, and compare. "Impressive" means nothing. "Better than our current step, on our actual work" means something.
- **A hard stop on Friday.** When the week ends, the spike ends, decision or no decision. A spike that runs long is no longer a spike, it is a distraction wearing a lab coat.
We keep a running queue of tools to spike, in **Airtable**, ranked by how much pain the target task causes us. The most painful task gets the next slot. That keeps the spikes pointed at things that matter instead of things that are merely shiny.
How we test in public without leaking anything
Testing in public is part of the brand, but client work is not ours to broadcast. So every spike runs against Temerarii Media's own task, never a client's.
When we spiked a new image generation pipeline, we ran it on our own blog header art. When we tested a new transcription model, we ran it on our own podcast cuts. The input is always something we own, so the test is real and the publishing is clean.
We write the spike up the same way every time: the task, the baseline, the tool, what we fed it, what came back, and where it broke. The "where it broke" section is mandatory and it is the most-read part, because a public failure is more useful to a reader than a polished win. Anyone can find a glowing review. Almost nobody publishes the thing that made them put the tool down.
A worked example: the spike that ended in park
We spiked a new image generation model the week it launched. Everyone called it the one that finally made the older pipelines obsolete. Our real task was our own blog header art, which our current pipeline handles in a known, repeatable way.
We ran the new model against the same five briefs our current step handles weekly. The new outputs were, in isolation, striking. Against the baseline, on our actual task, they lost on the one thing that matters to us: consistency. Our brand needs the same visual language across a run of headers, and the new model produced five beautiful images that did not look like they came from the same studio. The old pipeline produced five plainer images that did.
That is a park, not a kill. The model is not worse forever, it is worse at the specific thing we need now. So we wrote the re-test trigger next to it: re-spike when it supports a locked style reference across a batch. When that ships, we will know in a week whether it beats the baseline, and we will not have to re-argue the whole thing from hype.
We published the spike, failure section included. The most common reply was from people who had assumed the new model was a clear upgrade and had not tested it on a consistency-sensitive task. The public failure was more useful to them than any of the glowing launch-day threads, which is exactly why we write that section.
Reading the queue: which task gets the next slot
The queue is ranked by pain, not by buzz, and that ranking is the quiet discipline of the whole method. The most-requested tool is rarely the most painful task. The painful tasks are the boring, recurring ones: a transcription step that needs cleanup, a scrape that breaks monthly, a render that takes too long.
So the next spike slot goes to whichever real task currently costs us the most time or annoyance, and we test whatever tool credibly targets that task. This keeps spikes pointed at our actual workflow instead of at the trend cycle. A tool can be the talk of the timeline and never get a slot, simply because the task it improves is not one that hurts us. That is the correct outcome, and the ranked queue is what makes it automatic instead of a debate.
The verdict: keep, kill, or park
Every spike ends in one of three written decisions, and we have learned the third is the most valuable.
- **Keep** means the tool beat the baseline on our real task and we are wiring it into the workflow this month. Rare, and it should be.
- **Kill** means it lost, and we write down exactly why so we do not re-litigate it in three months when it trends again.
- **Park** means it is not better yet, but the gap is closing, and here is the specific condition that would make us re-spike it. A parked tool gets a trigger written next to it: "re-test when it supports X."
Park is the verdict that saves the most time, because most new tools are not worse forever, they are worse now. Writing the re-test trigger means we revisit on evidence instead of on hype, and we never have to hold the whole landscape in our heads.
When we adopted this on ourselves, the surprising result was how often the answer was kill or park, not keep. The tools that survived a spike against a real baseline were a small fraction of the ones that looked unbeatable on launch day. That gap, between launch-day impression and real-task performance, is the entire reason the method exists.
What this makes possible
A team that spikes one tool a week, every week, builds something most teams never have: a documented, honest map of what actually works for their specific work, updated continuously, with the reasoning attached.
That map compounds. New hires read it instead of re-discovering dead ends. Decisions get faster because half the candidates are already parked with a reason. And the workflow stays current without ever stopping to chase, because evaluation is a steady background rhythm, not a panic every time the timeline loses its mind.
The goal is not to use the newest thing. It is to know, on the record, why you use what you use, and to find out cheaply when that should change. A boxed week and a written verdict is how you get there without drowning.
We run these spikes in public on purpose. The experiments at [temerarii.xyz](https://temerarii.xyz) are where they live while they run, and the rest of the [blog](/blog/) carries the verdicts, failures included.
Written by Dominique Davis, Founder, Temerarii Media.