Big Tech and the reality of scaling customer support

There are plenty of good examples to take from Big Tech when it comes to software engineering: from how they empower engineers, through how they run projects or even how they architect systems.

However, customer support is not one where Big Tech has much thought leadership to offer. Take this full-page ad posted in the JS Mercury News on Monday, 22nd January, which was a call for help from an agency that spent more than $800K on ads with Google:

A full-page ad serving as a plead to Google on 2022, 22 January. See a thread of comments here.

Customer complaints handling is at scale is broken at most tech companies. The easiest way to experience this is by starting to work at a consumer-facing company. Choose Google, Meta, Bytedance, Spotify or any other one.

As soon as you update your LinkedIn profile to the new gig, you start to get messages from friends of friends asking to solve one of their problems. They message you because you're the only human they can get hold of at these companies.

Every now and then you'll see how broken the process is in public.

Down the rabbit hole of customer support

When working at Skype at Uber, I got plenty of these inbound messages. An account blocked. A video call not working with a contact on a regular basis. A payment stuck.

At first, I was eager to help. I'd open a support ticket and try to route it to the right internal teams. I sometimes would be able to find someone working on the functionality that was broken.

However, none of my tickets got resolved. Some sat idle. Some got closed as duplicates of known issues. And some resolved as "won't fix."

What was going on? Why was no one fixing these issues? The longer I worked at these companies, the more I understood the rationale. There were two underlying reasons:

1. ML black boxes. For complaints related to fraud, this almost always came from a machine learning system flagging an account for suspicious activity. The fraud teams had to balance false positives (tagging people as bad actors who were, indeed, not, causing users churning) and false negatives (tagging bad actors as good users, causing a loss to the company).

When an ML model decided to ban an account I was told by a friend of a friend was legit, I had no options but to hope the team owning the ML model would do something: either whitelist the account or tweak the model. For the most part, nothing changed as a result of my report.

2. Functionality broken for some users. Several bug reports indicated issues that were happening in certain edge cases, often in a non-deterministic way.

I spent countless hours trying to reproduce some of these issues. Unfortunately, what I've seen is there's an 80-20 rule in play here. 80% of customer reports were useless in the sense that they were really either hidden asks for new features, exceptions or were non-issues that were resolved by using the feature in the way it was intended.

About 20% of the reports pointed to something fundamentally broken which might have impacted more users. Getting to this 20%, and figuring out what the issue really is, took up a lot of time.

And yet, even when I figured out the root cause, made a well-written bug report with reproduction steps and estimated impact, almost none of these got fixed.

The carrot and the stick at Big Tech

To understand why customer complaints are rarely addressed at large companies, it's important to understand the incentives structure.

Carrots are things that these companies reward. Stuff that gets people better performance reviews and promotions - and, at the end of the day, better financial and career rewards.

At most of Big Tech, carrots are:

• Business impact
• Increase revenue significantly
• Decrease costs significantly

Fixing customer issues impacting a few customers - and well below 0.1% of the population - is not in this box. So it won't be rewarded.

Alright, but won't anyone get in trouble for not attending these issues? Let's look at what gets people into trouble.

Sticks are the opposite of carrots: things that pull down your performance ratings, block you from promotions, and can get you fired. At most of Big Tech, sticks are:

• Not delivering adequate business impact
• Unprofessional behavior
• Poor feedback from your manager, peers, or direct reports (the last one only if you're a manager)

Not spending any time on one-off customer issues won't trigger any of the "stick" parts - unless your business impact is measured by doing so. Unless you work in customer support, this is unlikely to be the case.

Amusingly, spending too much time on customer issues might even get you in trouble. If you spend a considerable time fixing customer issues, you will likely only deliver a small impact. This, in turn, can activate the "stick" that comes when you don't deliver adequate business impact, and result in a poor performance review and no promotion.

The Amazon exception

Even though I've been generalizing Big Tech, one company does not fit the above. This company is Amazon. And it's for a simple reason.

Amazon has made Customer Obsession the single most important company principle. The company has 16 Leadership Principles, but - as I've covered in Inside Amazon's Engineering Culture - has insisted that Customer Obsession is always the most important one.

At Amazon, Jeff Bezos built an efficient escalation chain. Customer complaints could be routed upwards, or sent directly to him. Jeff would review many of these himself, and when he found it an issue where the customer was right, and it should have been fixed, he would forward the issue to the team owning the problem area, typing a single message along, which was this:

?

This is a message Amazon teams dreaded to get. It meant the Eyes of Sauron were on them and needed to fix the underlying issue, and respond with the fix.

If Jeff Bezos was not happy with the fix, he would summon the leader of the team to him to present a better fix in two week's time. I know this because I talked with a senior Amazon leader who confessed the trip to Jeff's office was the single most stressful walk he'd done over his career, and he sweated through his shirt by the time he got to present.

Amazon has a very powerful "stick" in place that most Big Tech does not. There is no option to ignore customer complaints: and this attitude starts at the top.

Newsletter

Subscribe to my weekly newsletter for advice, observations and inspiration across the software engineering industry. Especially relevant for those in big tech and at high-growth startups.

The Google approach of automating customer support

Google is on the other scale of customer support. The company has been a pioneer in launching projects with no customer support, and filling the gaps with self-service tools.

In 2011, when I was a paying user for the beta version of Google Apps Engine, I spent more than $1,000/month on the product. Still, the only support channel was a Google Forum where fellow customers would advise on problems. If we were lucky, a member from the Google Apps team would show up - but there was no guarantee of ever being able to interact with anyone from the team.

Even after building the largest email system in the world, Google does offer human support for customers. Gmail has more than 1.5 billion users, and has scaled up thanks to great engineering and with the help of self-service tooling.

And yet, if you get locked out of your email, you have no options to contact a human, even if you wanted to pay for this. The support page for getting locked out of your Google Account ends with no option to contact any support channel. The page for requesting to restore a Google Account is a form where someone at Google might review your account one final time.

Google hides the fact that they do offer paid support from most of its customers. Reading through all official Gmail help pages, there is no indication that Google offers any means to purchase paid support for Gmail. However, this is not entirely true. Google launched a new product called Google One which is a plan to get more storage, and a vague promise of "access to experts". It's unclear if this means customer support versus Google enthusiasts, and Google, seemingly deliberately does not clarify what "experts" mean beyond someone who "speaks fluent Google", whatever this means:

You have access to "Google experts" with Google One. Is this customer support? Or just people who can read out the FAQ for you? Google makes this unclear, likely on purpose.

Will Big Tech change?

Most of Big Tech is going down the same path Google has chosen. Build self-service systems and minimize the customer support surface. Even when offering customer support, limit the surface area, and have no incentives for engineers to tackle one-off customer problems.

And you know what? Not investing much in customer support is profitable. Google, Meta, and other companies following the self-service automation model are doing great profit-wise. So much that companies which have historically offered good customer support are cutting down on these costs.

Booking.com has been known for its decent customer support. Now they are  firing most in-house customer support people to replace them with an outsourcing agency. Given that Big Tech rewards decreasing costs, it's little surprise that cutting down on customer support will be an initiative that frequently comes up - and a temptation hard to resist on the long-term.

As a former Amazon engineer put it, it's hard to measure the dividend of customer support. This is why most of Big Tech de-prioritized investing in this area.

But it doesn't have to end gloomy for all of Big Tech. As I was researching how Big Tech does customer support, another example stood out: Apple. I have heard relatively few horror stories on Apple customer support. Apple seems to offer a mix of self-service, but also human touchpoints in their support. Turns out, Apple has incorporated customer satisfaction scores - CSAT - as a target for teams to focus on. And so they do.

And it's not just Apple. Since I tweeted about how Big Tech does a terrible job at customer support, I've heard that Meta is launching a new customer support organization, sponsored by Mark Zuckeberg.

I'm not holding my breath, but we could see customer support across Big Tech improve.

Voting with your feet

While we wait to see meaningful customer support changes, is there anything we can do?

There is.

Vote with your feet, and with your wallet. Gmail might not offer support for customers, but plenty of email providers which charge for their service do. Both Fastmail and Protonmail are examples where customers have clear support paths, should they run into any issue.

Instead of getting frustrated for being unable to reach support at Google Analytics even when paying, choose a smaller tool like Plausible analytics - like I am doing - and get a direct line to the team for questions and queries.

While Booking.com might be cutting down on customer support costs, the smaller travel agents and hotels and accommodation providers depend on providing this support to scale.

Going small, of course, always carries risk. Smaller companies might go out of business - or they might grow, and start to optimize on their customer support costs.

Still, you're more likely to get the service you expect when things go wrong, versus using Big Tech companies that are deliberately under-investing in this area.

And, who knows: these competitors gaining business might be just that nudge that Big Tech companies need to up their game. Perhaps with enough people switching email providers, Google might even one day change the term "Google expert" to "customer support" and include it in their help pages.

Featured Pragmatic Engineer Jobs

1. Senior Product Engineer at Casual. $150-250K + equity. Remote (US, EU).
2. Senior Software Engineer at Patina. Remote (US).
3. Founding Software Engineer at Teero. €70-100K + equity. Amsterdam, Remote (EU).
4. Senior Backend Engineer (Ruby) at Rise Calendar. €80-120K + equity. Remote (EU). I'm an investor.
5. Engineering Manager at Spill. £80-100K + equity. London, Remote (EU).
6. Senior Backend Engineer at Spill. £80-100K + equity. London, Remote (EU).
7. Full Stack Engineer at Relive. €55-75K + equity. Remote (EU).
8. Founding Software Engineer at Keel. £90-130K + equity. Remote (EU).
9. Senior Backend Engineer at Waybridge. £60-110K + equity. London.
10. Senior Full Stack Engineer at Packfleet. $105-240K + equity. NYC, Remote (US, EU).
11. Engineering Manager at Clipboard Health. Remote (US, EU).
12. Senior Full Stack Javascript Engineer at Clevertech. $60-125K. Remote (Global).
13. Senior Product Engineer at Barsala. $100K+ + equity. Remote (Global).
14. Engineering Manager at Raycast. $140-170K + equity. Remote (EU).
15. Senior Software Engineer on the Autoscaler team & the Core Platform Team at CAST AI. Remote (EU) €54-96K + equity.
16. Senior Product Engineer at On Deck. $100K+ + equity. Remote (Global).
17. Site Relibability / DevOps Engineer at Orbit. $120-165K + equity. Remote (US, EU).

Browse more senior engineer and engineering leadership roles with great engineering cultures, or add your own on The Pragmatic Engineer Job board.