When your app crashes on a user’s device, you need to know about it. It’s as simple as that. Users who experience severe performance issues aren’t going to write in and let you know—they’re going to leave cryptically bad app store reviews: “Doesn’t work.” “Useless.” “Paid for this app and it did nothing.”
These are the kinds of reviews that impact app downloads, and they can cause serious internal stress if you don’t have the tools to understand _why _users are having problems and where.
Properly testing your app isn’t as simple as setting up Crashlytics or PagerDuty and sitting back, however. Here are five crucial things to remember to get the most out of your app performance tracking.
1. Test your app on as many different devices as possible.
When it comes to the problem of multiple-device testing, the word to remember is “fragmentation.”
It’s more of a problem with Android devices than iOS. There are 1.4 billion active Android phones and tablets in the world, and at least 24,000 unique types.
The image below from OpenSignal visualizes the market divided by device type, showing just how fragmented the Android ecosystem is.
You don’t need to buy 24k+ Android devices and test your app on all of them, but you do need to be vigilant about testing on each different device that you plan to support.
When you’re thinking about which devices to support, refer to Android’s official page listing usage by device and operating system. To prioritize, find the most common ones that you want to support with your app and focus on testing those.
Services like Testdroid and Xamarin Test Cloud, which run on your app on real physical handsets, can then help you actually perform the tests and integrate your changes continuously.
2. Your whole team needs to be on the same page.
The worst way for your team to handle a major outage is to keep the details of that outage confined to engineering. The fact that your app crashed isn’t necessarily something you want to shout from the rooftops, but you need the details of the incident and the plan for getting back on track, to be out in the open.
If your app goes down and your customer service team is suddenly deluged with complaints, they need to know:
- What’s wrong
- Whether any user data or personal information has been compromised
- The timeline for getting the app back online
To get this kind of information to your team, you need to set up the proper distribution channels and train everyone in engineering to use them. Internal dashboards, email distribution lists, and status pages are your most powerful tools here.
An example of a dashboard from PagerDuty.
A lot of companies will wait until they _have _an outage to set up these channels, but having them at the ready is the only way to respond to your first crash in an efficient manner. And as a side benefit, setting up those cross-disciplinary channels is a way to lay the framework for more cross-disciplinary work in the future—most importantly, establishing a data-informed culture.
3. Unfixed bugs will tank your app store reviews.
App store reviews are a fickle beast. It’s unlikely that getting a good score is going to seriously boost your acquisition efforts—because many users are looking for mobile apps with 4-5 stars at minimum—but getting a run of 1-star reviews right when your app goes down could forever cripple your place in the app store.
And it is usually problems with an app that can bring down an app’s reviews the hardest. Within one-star reviews, some of the most common keywords were crashes, useless, nothing, didn’t, and stupid.
The key to avoiding negative mobile app reviews is responding to crashes quickly—but the reality is that you will, at some point, get 1 and 2-star reviews. That’s part of why developers put so much work into asking for positive reviews from users. Just make sure you’re asking at the right time – best to use your crash reporting tools to keep from asking users for whom your app is highly crash-prone.
Another tactic, embraced by the Ember team, is to give users with complaints about bugs or outages another outlet.
If you give annoyed users the option to get in touch with your team—assuming your “Contact the team” button doesn’t trigger an infuriating, Safari-opening mailto link—you can divert some potential 1-2 star reviews.
4. Your app’s performance depends on priorities
No matter how diligent and talented your team is, you will have bugs, and the speed at which you take care of them is going to depend largely on how your team prioritizes bug fixes vs. new features.
Marc LaFountain, support engineer and first employee at Tumblr, found that most engineering teams focused on the latter. “It may not be perceived as sexy as building the new features that the company is working on,” LaFountain says of fixing bugs and issues, “but it’s important.”
Without at least some effort dedicated to fixing bugs, you’re going to wind up with a product development process that resembles an ouroboros—where engineering’s focus on the New constantly undermines efforts to maintain what’s already There.
Especially for a company that’s growing fast, technical debt can rack up quickly. Over time, that debt gets more and more embedded in a product’s architecture and it becomes harder to cut it down. Without top-down insistence on going through the codebase and fixing bugs, that debt will just pile up.
5. You need both crash reporting and analytics.
Bad crash reporting can distort the results of your analytics even if the event data you’re pulling is completely clean.
Think of one of your app’s critical flows—say you’re looking for your users to go from a product page, to entering their address, to check-out. If there’s a significant drop-off between one or more of these steps, you need to be able to rule out a malfunctioning UI/UX if you want to take accurate conclusions away from your funnel analysis.
If you think that people aren’t getting from address information to check-out because there’s too much text entry, or your buttons are the wrong color, and the real reason is because your check-out button is throwing an error, then you’re going to waste a bunch of time optimizing the wrong thing.
The upside of uptime
According to Aptiligent (formerly Crittercism), most people spend 80% of their app time using their favorite 3 apps, so if your app belongs in the other 20%, a crash can be a serious problem to creating loyalty with a new user.
If you want your app to be something new users can imagine coming back to time and time again, beat the odds—make it reliable.
For more on setting up, launching, and making your mobile app sticky, check out our Mobile Analytics resource page.
photo credit: WOCinTech Chat
