Product testing exists to make sure that things we build, work as intended. If something about your customer’s experience is broken, you might lose them forever. Companies obviously want people to use their products reliably and avoid this — product testing makes for a logical solution.
Yet despite its importance, product testing has plenty of issues. Software teams are routinely subject to short timelines (QA teams regularly hear ”Can you test this last minute?”). Regression testing, which makes sure new code does not impact existing functionality, is often the first thing cut when resources are tight. This combination of resource constraints plus technical debt have made adoption of best practices (like continuous testing) and new AI / ML techniques, damn near impossible for most companies to implement.
When over half of software companies suck at something they all acknowledge is critical, something is amiss. Either people are lying about how much they value product testing or the model is broken.
Whats wrong with product testing?
My take is the incentives are all off.
Businesses don’t get a clear idea of what or how to spend a testing budget, so they only invest a fraction of what’s really needed. Problems don’t immediately surface, so issues are able to compound quietly. After enough time passes, they resurface as customer problems and become a focus for leadership. More budget is allocated and because those customer issues are pressing, teams rush and can only scope for a fraction of the true cost. The cycle repeats.
You can see how ineffective product testing is a self-perpetuating loop. Getting it right is hard.
The way of blockchain protocols
Blockchain protocols are experimenting with a new concept for launching products. These decentralized networks (e.g. Ethereum) have entire ecosystems of products (called “decentralized applications” or “dapps”) that need to reliably interact. In order to effectively test these networks, protocols rely on the contributions of users rather than paid workers.
Product testing is largely done through a rewards-based concept called airdropping. An “airdrop” is when a token is delivered to an individual’s crypto wallet, essentially their cryptocurrency bank account. New protocols will encourage users to contribute to the network by performing certain actions. Users willingly perform those actions and more because (a) they might receive some future token by airdrop, and (b) they have a reason to use the products bring offered. Protocols can choose to reward users who used the network with tokens, which typically represent ownership and protocol governance.
Airdrops are an incentive that still has a lot of legal / regulatory unknowns. However, by minimizing product testing costs with potentially better results, it’s a concept that is worthy of further exploration. I see it as an alternative, more sustainable way of product testing.
Incentives based product testing
The early experiments with blockchain protocols are far from complete, but the size and scale reached is remarkable. Look at how many unique users were using the 4 protocols below, 30d prior to an airdrop.
Hundreds of thousands to millions of users actively testing and using your products — that’s insanely valuable to software teams. Not to mention a few other benefits:
Easier tracking of bugs. Every transaction made on blockchains are recorded to a public ledger. This means issues are automatically captured, no need for bug reports.
Ability to target edge cases. Incentivize users to focus on your most critical or challenging issues.
Long-term customer acquisition funnel. Encourage behavior that creates stickiness.
Paying users to do most of your tests, then building what you need to fill in for the rest, could be a model for effective product testing. Who knows, maybe it's effective enough to disrupt the $38B testing industry.
-----
* Sources: Blast leaderboard, ZKsync data via Nansen, LayerZero data via Dune, Arbitrum report via Coindesk.