I am writing this post to those who are fundoo. They are elite. They are either VP Engineering or QA Head who mostly think QA means writing test code. Mostly. Not completely. I am no coding expert to teach people how to write robust code – be it product code or test code. That disqualifies me from advising them how to do their job.
I have found them to be hard to convince people unless you talk code. I can’t convince them. Data can. Story telling can. So – I am trying to combine them in this post.
App uninstall rates of top 100 apps for instance
Good news to engineering teams about the numbers that look this bad – not all the uninstalls or lack of usage is because of app quality. The not so good news is about 12% of the uninstalls are because of app crashes, functional robustness problems and app performance issues.Note – I didn’t call that bad news because I have the bad news coming.
What is the bad news for engineering teams?
The bad news is – super fundoo engineering teams – have reached their end of wisdom in testing after all the fundoo testing they do. They implement all frameworks – mostly open source – use tools provided by Google or use cloud based solutions available in the market. They hire a bunch of people who are called SDET’s and then a few QA who are actually testers. Everything looks good. Why have 25% uninstalls due to app performance and bugs despite throwing in so much money to test? 25% means – for every dollar spent to get the user the company is losing 25 cents.
Where does the blind spot come from?
The blind spot comes to those who don’t have time to listen to anything that does not come from whom they consider not so elite to advice them. The blind spot comes from thinking, “We are doing the best testing we can already”. The blind spot comes from the fact that this is not their problem because 75% of users are not uninstalling because of app quality.
I agree that 75% of users not uninstalling the apps because of app quality but looking at the 90 day DAU drop to 19% even for the top 100 apps is dangerous. As engineering teams you can only do what is in your control. What is in your control is to bring the 25% number down to as low as you can.
No – fragmentation is not the problem
People have sold you an idea that there are so many mobile phones and it is hard to test on all of them. So many people quote this as convent answer to why their app does not work on a specific phone. Hey wait! We are talking about fundoo engineering teams who have access to everything they want. They have the test frameworks – they have scripts – CI and the fancy setups of mobile devices like this one. Yet – 25% uninstall due to app quality problems? Seriously. What is wrong?
Tests are the problem
Not testing. Tests. The kind of tests you run matter. Most people in your engineering team including your testers and QA are doing what is obvious to do. The happy paths. The corner cases. The flows that your users take. That is all covered. I know that. What is not covered is testing in user conditions. Are you testing in Low RAM conditions? Are you testing in varying RAM condition? We have seen that for our customers testing in those real world conditions helped us find more crashes and performance risks even for apps that have very high Memory and CPU scores.
We initially thought it was a value add but after seeing the data we have come to discover it is such a core value that we are doing a free analysis to all our signed up customers.
A report of an app built by Google Android Developers themselves
GoogleIO 17 app was built by their own teams. Now wouldn’t they have tested it with all the super fundoo Firebase Test Lab? They definitely would have.
Following is the report of the same app under Changing Orientation During Test
Now – what happens when a Change Orientation Test is run in Combination with a 2G Network? More crashes and performance issues can be uncovered. This is what is happening to real users. And all our testing is in normal condition in lab and works mostly but not when conditions fluctuate.
You could either bash me for this post or could upload your app – get testing of it in Normal and in Real World User Conditions and then choose if you should bash me or work with me.