The following is a guest post from Brian Capps at Lickability, a premier mobile software studio.
"If you can't measure it, you can't manage it."
- Peter Drucker
Emerge Tools can measure a lot about your app — size, performance, UI changes, and more. However, only you can measure what success looks like for your product. Number of users, retention, conversion: these success metrics are often critical to your business and can be measured using analytics. But with a head-spinning number of analytics libraries available, which one should you choose for your app?
appFigures has analyzed millions of apps in the App Store and provides a handy list of the most popular analytics libraries. It's easy to find a list of features for each library and why you might want to use it. But how does adding an SDK affect your app? What type of overhead do these libraries bring?
We can't choose an analytics library for you, but we can compare the size and performance implications of different SDKs to your app on iOS. Let's see how they all stack up!
The Comparison
This post will compare the top 6 analytics SDKs on iOS. Every SDK will be measured against a newly created Xcode project with the default iOS app template (base app). This simple, single-view app is a mere 118.5kb in size with a clean startup path.
We'll add each library as a dependency through Swift Package Manager and initially configure it as directed (generally calling a setup method from the app delegate). Then we'll compare the size and startup performance to the base app.
Startup performance comparisons will be done using Emerge's Performance Analysis. Some notes on how it works:
- Testing is done on physical devices (iPhone SE 2020)
- For this comparison, startup time is defined as process start to
didFinishLaunching(Emerge users can customize markers) - The primary metric is the relative change in startup time between the two builds. While we often think of startup in absolute time, the percentage change is more apt for comparisons
- To come to a statistically significant conclusion, testing is done in a controlled environment and consists of numerous iterations of both the head and base builds (below image)
- You can refer to our docs for full details on how testing is done
Links to our complete analysis are included for each SDK. The following are the 6 most popular analytics SDKs from most to least popular.
Firebase from Google is by far the most popular analytics library in use on the App Store. This makes sense, with how many features Firebase offers as an all-service app development platform (database, push notifications, A/B testing, etc.). Our focus is testing the implications of only adding the FirebaseAnalytics product from the package. We are using the latest version, 10.14.0, and here are the results of the analyses:
Note: While appFigures lists Fabric and Google Analytics as popular SDKs, Google has rolled both products into Firebase Analytics, so they have not been included.
The next most popular SDK is Facebook Analytics. This popularity is also no surprise: Meta and Google are the largest companies on this list, and Facebook has long been a presence in the iOS community (anyone remember Three20?). To set this up, we added the Facebook SDK version 14.1.0 with only the FacebookCore package product for analytics. Here's what we found:
Looking at the flame chart, we can see Facebook accessing the Keychain on the main thread. While these are quite fast, best practice is to put these calls in the background.
Like many of the top analytics SDKs, Flurry has been around for a long time. The configuration instructions show their age, referencing Objective-C code by default with no mention of SwiftUI or any newer technologies. After some initial trouble with website load issues, we installed version 12.4.0 and got to work on the comparison. You can see it below:
Amplitude is an analytics platform with detailed insight and experimentation capabilities. Founded in 2011, it's the newest company on the list. They offer several iOS SDK options, but for this comparison, we looked at the newer, purely Swift beta iOS SDK, version 0.4.14:
Segment is another platform for analytics and customer data, with a focus on data insight. They were acquired in 2020 and are now a division of Twilio. They offer several iOS SDKs, but we chose their Swift SDK for this measurement. Here's what we saw on Analytics-Swift version 1.4.7:
Part of Emerge's analysis identifies protocols without conformances — dead code. Of all the SDKs, Segment SDK had the most of this type of dead code, but not a significant amount.
Finally, we have Mixpanel, our last (but certainly not least!) analytics provider. Mixpanel has been a staple of iOS analytics for quite a while, and they offer similar features to Segment and Amplitude. We selected the Swift SDK and tested the latest version, 4.1.4:
Results
So how did they all do? Let’s see all the results in a table for easier comparison:
| SDK | App Size Increase | Startup Time Increase (Relative) |
|---|---|---|
| Firebase Analytics | 1.6 MB | +30.2% |
| Facebook Analytics | 1.4 MB | +18.7% |
| Flurry | 1.0 MB | +10.8% |
| Amplitude | 468.5 kB | +23.6% |
| Segment | 718.8 kB | +14.6% |
| Mixpanel | 422.2 kB | +31.0% |
As you can see, there are significant differences in app size and startup time between the SDKs. The most popular SDKs with some of the oldest repositories and many features add the most to your app size, while some of the more focused, less popular entrants are more compact and nimble. This may have been expected, but now we have the data to dig in more.
Firebase, with many years of development and shared core code with its myriad other features, tops the list as the most expensive in size and second in startup time. On the other end of the spectrum, Mixpanel's SDK is nearly 75% smaller than Firebase, while still adding just as much time to your app’s startup. All the others fall somewhere in between, with a few surprises like Flurry’s minimal impact on startup.
Increased size and runtime can affect your bottom line in many ways (and these SDKs will help you confirm that 😉). Aside from impacting the user, adding a megabyte of app size can have the equivalent CO2 emissions for 5 round-trip flights from London to LAX for a globally-scaled app, so it's important to understand what's going into your app.
More than Metrics
While these metrics provide helpful information, they can’t tell you everything. There may be features that your product team needs that only some SDKs offer, cost considerations, and many more factors. For example, Firebase has many other features that are incredibly useful for mobile developers (along with it being mostly free!), and that may be a much greater consideration than any app size or startup cost.
The analytics SDK you choose for your app will be personal for you and your business. Hopefully, these metrics provide helpful context for your team to decide. If you'd like to evaluate the performance implications of a specific SDK, you can create an Emerge Tools account to see our analysis!
You can reach out to Lickability for help implementing your Emerge Tools analysis recommendations!