The Sad State of Logging Bugs for Apple

You find a bug in macOS or iOS. Something in the software isn’t working the way it should work. You decide to let Apple know, and head over to http://bugreport.apple.com and login with your Apple ID. You fill out the required form, list the reproduction steps, hit the special key chord to generate a fat sysdiagnose, attach some screen shots, and hope they fix it. 

What happens to your bug?

All bugs are tracked with an internal application called Radar; internal Apple employees simply log bugs directly into this application. External people use the website “Bug Reporter”, which only shows specific developer interactions with the bug, and hide all other details of the bug. Those details are engineer comments, progress on the bug, when they “expect” to fix it or which build it is fixed in, and priority. Priority is important for bugs; they are considered “unscreened” if they don’t have a priority set. Bugs with a high priority (1) are highly likely to get fixed, whereas bugs with a lower priority (3 or 4) are not likely to ever get fixed.

Each bug is placed into a particular top level component, which can have sub components. Savvy internal developers will log bugs directly into the component that the bug corresponds to, such as “AppKit / NSTableView”. This will get it directly to the right person who needs to look at it. Top level components have a large dropping areas that generic bugs can fall into, such as “iTunes / All”. These components get saturated with bug reports.

Once your bug hits Radar someone will look at it. Developer Support will often take a first look at bugs that are related to SDK/programming issues. They will then pass it on to a more specific top level component, or directly to a more specific component (ie: NSWindow).

This is where things get screwy depending on the component your bug lands in, since bug management is group dependent. Many groups will have only one or two QA people to do the initial screening of those large drop areas for bugs. QA engineers are sometimes instructed to screen bugs with a priority and “fix period” before passing them off to the engineer responsible for the code. This is terrible because many engineers will not look at bugs with a low priority. It is much better for the engineer who “owns the code” to look at a bug and determine the priority. The QA engineers will frequently get a huge back log of bugs to screen, and it can take weeks, or even months, for some bugs to get screened. Sometimes this leads to a mass screening of bugs, marking them all with a low priority . Bug originators have to notice this, and complain about it for the priority to get increased. Worse yet, some groups mass close bugs older than a year or so, and ask the originator to re-open the bug if the issue still exists. A lot of people don’t pay attention to bugs that need verification, and they simply become lost. 

From an internal developer’s perspective, it was quite frustrating to take the time to log a detailed bug, only to see it sit “unscreened” for weeks and weeks. It is demoralizing to see this happen to your bugs. But worse yet, sometimes your bug would get forward-duped. Someone else at a later time logs the same bug, and your older bug report gets duplicated to it. More often than not, this is simply because of the delayed screening of bugs.

From an external developer’s perspective, it is even worse. There is very little feedback on bugs when they are in progress. You don’t know what priority the bug has, or if it has even been looked at and screened. This is Apple internationally being opaque. More transparency could easily leak features or products, so this is expected.

Hopefully a bug ends up with an individual engineer who works on the code. A lot of these bugs will still be unscreened, and Apple engineers seem to loathe the process of screening bugs. So they let them sit around for a while, sometimes for weeks or months. People would rather focus on coding something new up, or working on already screened bugs that they have to fix , as opposed to taking the time to read a bug report and figure out what needs to be done with it. They don’t have to read it and fix it right then, but just do a screening: determine priority, when to fix it (this is sometimes nebulous), and possibly ask the originator for more information (or mark it as a duplicate.)

Engineers also dislike screening bugs because sometimes they have to add them to their queue for the current release. This increases their required workload for that release, which is something people don’t like doing. So, instead, many bugs stay unscreened.

Unscreened bugs indicate potential software problems that need to be fixed. Obviously this is bad, and management doesn’t like to have high unscreened bug counts. Managers will frequently instruct engineers to screen bugs, or ask QA engineers to do it (which is bad).

Sometimes QA screen bugs with a low priority and holds onto them. They never  get moved to the appropriate code engineers, and effectively become lost in the system. Sadly, I had seen this happen way too often.

Bugs screened with a low priority and a nebulous-future fix date are also lost. They rarely will be revisited and fixed. Sometimes this is okay. Some bugs are just too obscure or rare to spend time fixing.

Some bug owners don’t want to take the time to go back and forth with the originator to find out some piece of information they may need. They sit on bugs for a long time, and eventually try to reproduce a bug on a later version of the un-released OS. They can’t reproduce it, so they just assume it is fixed, and send it back to the originator. This is particularly bad for external developers, and I’ll describe why below.

When a bug is sent back as fixed, the internal developer who originated the bug is supposed to verify the problem is resolved. They can send it back if the problem isn’t resolved. However, internal developers don’t really have an incentive to verify bugs. Management doesn’t keep track of bugs that need verification or really require developers to verify them. Most engineers do verify bugs; they like to make sure problems are resolved. But external developers are left in a more sad state. The bug becomes closed for them, and is dead.

External developers do get notified when bugs are fixed, but only after the fix is available in a public release of the OS. This means it could take about a year for some bugs to get back to the external developer, since bugs start to get fixed *before* an OS is released to the public.  Generally, the bug will ask the external developer to verify the bug by installing the latest OS. For some people, including me, this just isn’t possible; they need to run and support an older OS, and don’t have the ability to install a newer OS on a particular machine. Or, they just won’t want to install the latest OS because it doesn’t have anything they want or need (that’s me, I run 10.13). Even if they do install the OS, they may find the bug isn’t fixed. At this point, it is simply too late; engineering has likely closed the bug, and it is lost in the ether.

What can Apple do? 

It is simple: Internal engineers need to take more responsibility in promptly screening bugs. Management needs to allow engineers to have more time to do this, which is at the expense of working on features or fixing already screened bugs. Engineers should always be expected to have a very low unscreened bug count.

Internal engineers need to promptly verify bugs when they are sent back to them. They need to install beta versions of the OS and do some testing. If it isn’t fixed, send it back.

If a bug report refers to some problem, then the developer should look at the code that interacts with the area described and try to think about what would cause such a problem to happen. For instance, if a button is incorrectly disabled in iTunes, then the developer should try to figure out why that could happen instead of just asking for “how to reproduce this?”

When I worked at Apple I practiced what I am now preaching. I would screen all my bugs within a day or so. I would verify bugs within a week or so. I would poke around at code and attempt to make theoretical fixes for bugs I couldn’t reproduce. I felt like it was part of my job, and I just did these things as a daily task.

What can you do?

Obviously, you have to log bug reports. It’s okay to log bugs that are missing information; developers can ask for specifics of what they need. In general, for AppKit related bugs, I would find that a “sysdiagnose” rarely would provide any useful information. I would hate it when people (usually QA, and not the engineers who worked on the code) would ask me for a sysdiagnose, when I know it wouldn’t help them figure anything out.

Log duplicates. If you know someone else logged a bug, but you are encountering the same issue, then log the same bug. It will get marked as a duplicate, but this will help increase the priority of a bug. It is sad, but Apple does look at this, and bugs with “high dupe counts” are looked at.