Here’s one absolutely massive AI fail I had this weekend.

I thought I’d share it because AI has become a world of look at these amazing things I achieved.

But (like everything) getting good at AI involves a lot of failing before you eventually succeed. And you see far less of that side of it.

I’d been seeing the “I no longer prompt, I create loops” stuff everywhere.

I’ve been desperate to learn loops since Ralph Wiggum came out last year, and I’ve had some really great successes with them too.

But I wasn’t doing the whole:

Here’s my prompt, Fable. You go away for three days, and when we next speak my world will have changed.

And, well, I wanted my world to change.

Subscribe now

What I was trying to do

Over the last few weeks, I’d built a particular data science model quite slowly with AI in the “old” way.

I went through long brainstorming and planning sessions. I interrogated those documents through a mixture of my own reading, AI evals and our data team.

I built a frontend for it using Claude Design, and suddenly had a beautiful model and report to share with a client.

It took a lot of hands-on time, but the output was there.

Over the weekend, I thought I’d use loops to extend it across the whole company roster.

I was using Claude Code, set to Fable 5. I told it that it could use a mixture of Fable 5, smaller Claude models, and all the new GPT-5.6 models it could access through Codex.

I set the goal that it needed to reach parity with the model we’d already built:

• same safety precautions

• same checks

• same planning decisions

• the output should contain all the same aspects as the original

I’d pre-empted a few of the conditions it was likely to hit because the data was slightly different from client to client.

I gave it permission to make judgements where one client’s data wasn’t sufficient.

I gave it evals.

And then I set it on its way.

So, what went wrong?

First, it overengineered almost everything.

The judgement around client data became a whole rabbit hole. It started creating its own models to judge whether each dataset was right or not.

It spent more than half a day on this, going through two five-hour context windows.

But perhaps the bigger failure was that my definition of “parity” around the output wasn’t clear enough.

There were around 100 variables that the original model produced. Part of my original work had been turning a dozen of those into a usable client report with actual insights.

The loop took the 100 variables as the goal.

It deleted my original frontend and got every client through to completed data. But the variables that were actually important were invalid for most of them.

It had focused on output rather than the right eval.

The other big problem was how it broke down the work.

It created six phases. But instead of doing one client all the way through to completion, it took all 21 clients through phase one, then all 21 through phase two, and so on.

The result was that when those six phases were wrong, they were wrong for everyone.

I’d been nudging it with feedback all weekend, but I didn’t notice the full extent of the failure until Monday morning. I eventually interrupted it and asked it to show me where it had got to.

That was when I realised:

1. The original client output had been deleted. Fine, it was recoverable from the commit history, but the whole new workload had been based on false data.

2. I’d used 80% of my weekly Claude tokens (mine reset on Saturday mornings), plus two of my OpenAI reset windows. I’d also spent quite a few API tokens after deciding to force some production work through in an effort to get it live.

3. Almost all of that work was wasted.

4. I’d given up six hours of my weekend for something that was a waste of time.

It didn’t create the best motivation for a Monday morning.

What I learned

There are learnings in all failures.

I’ve had success with loops like this before, but on quite different projects. It’s clear I need to refine my process if I attempt something like this again.

The obvious change is to take one client all the way through to completion before allowing a loop to repeat the process across 21 of them.

But one of the bigger lessons is the same as ever: don’t just follow the trend.

There’s new AI stuff every week. You don’t need to try it all, and you don’t need to replace something that already works just because there’s a new way of doing it.

Your objective is the most important thing, not the tool.

It’s only in the last few years, as auto-save has become dominant, that we no longer ‘click save’.

But for a good 20-year run, people clicked on a little icon of a floppy disk not knowing why that button represented the action of ‘saving.’

We became an AI-native agency a few months ago. I’d also define that a few of our clients are AI-native versions of …

Read more

When I was a teenager I wanted to be a video games programmer. I bought C++ for Dummies and tried to learn how to code. I couldn’t, for some reason my mind just wasn’t set up for it.

Over the years I’ve …

Read more

It’s everywhere. X, LinkedIn, Substack, even the NYT got in on it.

The truth is nobody really knows what’s coming. But the possibilities are starting to feel genuinely endless.

At the beginning of March, I published Becoming AI Nati…

Read more

“The way I interact with a computer has completely changed forever.”
– Mark Holland, our Head of Creative Strategy

Something changed over the last month. I and another colleague have become AI-native.

My colleague Mark, our head of creative strategy, told the team at Friday Beers & Cheers “the way I interact with a computer has completely changed forever.”

Read more