Why Xcode tools are slow after reboot

Why Xcode tools are slow after reboot

August 28 2021 by Jeff Johnson
Support this blog: StopTheMadness, Tweaks for Twitter, Link Unshortener, PayPal.Me

I ran git status on a newly created, very small repository, but the command took more than 10 seconds to finish. This was highly unusual, as git status is mostly instantaneous for me. Indeed it was instantaneous the next time I ran it on the same repository. Puzzled, I could only think of one thing out of the ordinary: I had just rebooted my Mac. So I tried rebooting again, and then the issue occurred again!

For the impatient who don't care to hear the story of how I discovered the cause, I'll cut to the chase. The /usr/bin/git executable on macOS is just a stub that calls the _xcselect_invoke_xcrun function in the /usr/lib/libxcselect.dylib library, which you can see by using the /usr/bin/otool command-line tool — which coincidentally is also just a stub that calls the _xcselect_invoke_xcrun function. By the way, if you're looking for /usr/lib/libxcselect.dylib on disk you may not find it, because Big Sur. Is your head spinning yet? The actual tools are inside the Xcode app bundle:

$ xcrun --find git
/Applications/Xcode.app/Contents/Developer/usr/bin/git

When you attempt to run one of the developer tools, the _xcselect_invoke_xcrun function must look up the actual path of the tool. The paths of Xcode and the developer tools are cached on disk in a database file named xcrun_db located in your $TMPDIR. (In Terminal, call echo $TMPDIR to see the temporary directory path.)

Perhaps you already see the problem here: the contents of $TMPDIR are emptied on every reboot! Thus, the first time you run a developer tool after reboot, the xcrun_db cache needs to be regenerated. Sigh.

As an experiment, I saved a duplicate of my xcrun_db file, rebooted, and then put the duplicate in $TMPDIR. With the cache restored, there was no longer any slowness on the first run of developer tools after reboot. Q.E.D.

Why does it take so long to regenerate the cache? While I was reproducing and diagnosing the issue, I noticed that when I ran a developer tool after reboot, the process syspolicyd went crazy and used almost 100% CPU until the command finished. I took samples of syspolicyd when this happened, and the process seemed to be spending a lot of time in the security framework checking code signing. Running fs_usage showed a lot of calls to getattrlist on files inside the Xcode bundle. (Not all the files, though; verifying the code signature of the entire gigantic Xcode bundle typically takes much longer, more like 10 minutes than 10 seconds.) So it appears that some limited kind of code signature verification is done before the xcrun_db file is written. I should mention that neither Xcode nor the files inside its bundle have any com.apple.quarantine extended attributes. The only xattr are a few com.apple.lastuseddate#PS on files that I've opened in a text editor. (Again, sigh.)

You may remember syspolicyd as the culprit in checking notarization of unsigned executables, but this is different. I can reproduce the developer tools issue with my wifi turned off, and I've verified with Little Snitch that syspolicyd does not even attempt any network connections in this case. Moreover, the tools in /usr/bin/ and Xcode itself are not unsigned, they're validly signed by Apple. Clearly, the slowness here is caused by checking local files and not by waiting on a network connection.

Why haven't I noticed the issue until now? I'm not sure, but I did recently update from Mojave to Big Sur and from Xcode 11 to Xcode 12. I can reproduce the slowness on Catalina, Big Sur, and Monterey, with Xcode 12 and 13. I haven't tested with Xcode 11, because it takes practically forever to download and install Xcode. It's not just an app anymore, it's an operating system! Actually four operating systems, since Xcode includes simulators for iOS, iPadOS, tvOS, and watchOS.

Do I have a workaround for the slowness? Yes and no. I don't know how to stop xcrun_db from getting deleted on reboot. However, I discovered a way to accelerate the cache regeneration to around 3 seconds, down from over 10 seconds: disable System Integrity Protection. Seriously, that works.

Support this blog: StopTheMadness, Tweaks for Twitter, Link Unshortener, PayPal.Me