This week I spent two days at the Products and Platforms planning event. The fifth one of these I’ve attended – they’re starting to feel less overwhelming. One of the unconference sessions was on test and learn. The discussion focused on problems people face trying to be agile in a deeply risk‑averse organisation.
For services that help users form habits or manage conditions, governance that doesn’t support us to test and learn is a blocker. The normal moderated research sessions we can do in the alpha phase aren’t sufficient. In an interview a participant might tell you they’d quit smoking or start exercising, but would they? What would their actual experience be like?
We need a way to test the purpose of the service, to see if we can impact behaviour over time. But putting something unmoderated out into the world without following the rigorous standards designed for scaled services has proved so hard that teams have given‑up trying to do the necessary testing in alpha and had to move to private beta to get anything done. Quick, small‑scale pilots are extremely difficult.
‘Pilot’ has become a banned word.
I think for good reason. Pilots are often an end in themselves, or can be seen as a way to get a service live without going through the necessary rigour.
Last year, I went to a meeting about running a pilot. A big meeting, lots of investment in time, lots of expertise in the room. I asked a few times what the pilot was looking to learn, the team said that they were going to get to that. I still don’t understand why anyone would decide to run a pilot before knowing what they need to learn from it.
As an assessor I’d struggle to pass a team through an alpha assessment without proving the core purpose of their service worked. I think part of the solution might be as simple as working with the various governance boards to create a new phase – between testing prototypes in alpha and building the MVP in private beta.
A ‘live alpha’.
Irina and I wrote a definition of pilots last year. A working service, tested with a limited group of real users, for a defined period – not built to scale, with clear success criteria and a decision point at the end. The idea was that it would sit within the second phase of an alpha.
Live testing not tied to production ways of working will become even more important as we start to use AI in our services. With AI you can’t fully specify the service, which means the normal model of ‘build it, then test it’ stops working.
The relationship is the service. You can test a relationship in a prototype, but to understand if it’s safe and has any real impact, you need real users, in real moments. A live alpha is the only honest way to test something that can’t be fully specified before you commit to an expensive build.