Speeding up Codecov Analysis for Xcode Projects
Tandem Diabetes Care develops and manufactures insulin pumps for use by people with diabetes. These pumps deliver a precise amount of insulin directly into the body and are one of the few on the market that can be controlled by a mobile app. Our customers need our products and services to be reliable and performant at all times. We have little margin for error and need to ship the highest quality code we can.
As a member of the mobile infrastructure team, I focus on developer automation and tooling of Tandem’s mobile apps. One of the newer tools we’ve been using at Tandem is Codecov. We love that it can track and report on our code coverage in total and on pull requests.
Codecov helps us understand which parts of our codebase are being tested by our unit tests and also helps us keep track of how the percentage of our codebase being tested changes over time. For example, if a pull request would introduce a greater than 1% change to our total code coverage, merging it will be blocked until that is addressed.
The Problem
One of my focuses recently was finding ways to optimize our CI workflows. I noticed that for our iOS app, the Codecov step of our workflow was taking much longer than expected, so I decided to find out why and if there was anything we could do to improve it.
Apple’s integrated development environment Xcode collects code coverage data and can display it in the IDE for developers. It can also be exported as an .xcresult file when using xcodebuild from the command line, like so:
|
1 2 3 4 5 6 7 |
xcrun xcodebuild test \ -project tconnect.xcodeproj \ -scheme tconnect \ -testPlan tconnect \ -destination "platform=iOS Simulator,name=iPhone 14" \ -derivedDataPath DerivedData \ -resultBundlePath artifacts/ResultBundle.xcresult |
These xcresult files are great and can be useful in lots of different ways, but like many things Apple, they can be difficult to use outside the Apple ecosystem. While they can be converted into a JSON representation using the xccov binary included with Xcode, the resulting JSON is not in the standard coverage formats that Codecov can ingest. It also has been known to change without warning with new Xcode releases.
And so, in order for Codecov to use the coverage results from Xcode, they have to be converted into another format. Codecov’s official GitHub Action can do the conversion, but it handles this conversion by analyzing the coverage for each file one by one, which can take up to a second for each file. This is a fine enough approach for some projects, but when working with a large codebase like ours, that can take quite some time.
Enter xcresultparser
The open source Swift tool xcresultparser can parse xcresult files and convert them into various other formats. One of these formats is Cobertura XML, which Codecov supports.
The big advantage xcresultparser brings is that, because it is a compiled program and not a script, it can use multiple threads to do the conversion. This speeds up the conversion process immensely.
After running the xcodebuild command above to generate the xcresult file, we tell xcresultparser to convert it like so:
|
1 2 3 |
xcresultparser \ --output-format cobertura \ "artifacts/ResultBundle.xcresult" >"artifacts/coverage.xml" |
And finally, we tell the Codecov GitHub Action to upload that XML file instead of the xcresult file.
Results
So, just how much time savings are we seeing?
We run these builds in parallel, so the total real-time savings for each build is the delta of the longer packages unit tests build: about seven minutes. This might not seem like much, but when you factor in that we’re running these builds upwards of 20 times a day, it’s a considerable time (and cost) savings. That’s over two hours of total developer time saved per day, almost 12 hours per week!
Bonus
While implementing xcresultparser for our project, I learned that it can also print a summary of test results to the command line. For our packages unit test build, we run eight separate test suites in serial, so if a test fails further up the log, it can be difficult to find it.
Now, at the end of each test run, we print out a summary like so:
|
1 2 3 4 |
xcresultparser \ --output-format cli \ --failed-tests-only \ "artifacts/ResultBundle.xcresult" |
This produces output that looks like this:
Now, it’s very easy for our developers to see which specific tests had failures just by looking at the end of the test log in CI.
Conclusion
Using xcresultparser has improved the lives of our developers quite a bit. And the fact that it is open source means that we as a developer community can help improve it for the benefit of ourselves and others.
