Are your TV, smartphone, and smart speakers eavesdropping on you?

Rumors of eavesdropping smart devices have been circulating for many years. Doubtless, you’ve heard a tale or two about how someone was discussing, say, the new coffee machine at work, and then got bombarded with online ads for, yes, coffee machines. We’ve already tested this hypothesis, and concluded that advertisers aren’t eavesdropping — they have many other less dramatic but far more effective ways of targeting ads. But perhaps the times are changing? News broke recently (here and here) about two marketing firms allegedly bragging about offering targeted ads based on just such eavesdropping. Granted, both companies later retracted their words and removed the relevant statements from their websites. Nevertheless, we decided to take a fresh look at the situation.

What the firms claimed

In calls with clients, podcasts, and blogs, CMG and Mindshift told much the same story — albeit devoid of any technical detail: smartphones and smart TVs allegedly help them recognize predetermined keywords in people’s conversations, which are then used to create custom audiences. These audiences, in the form of lists of phone numbers, email addresses, and anonymous advertising IDs, can be uploaded to various platforms (from YouTube and Facebook to Google AdWords and Microsoft Advertising) and leveraged to target ads at users.

If the second part about uploading custom audiences sounds quite plausible, the first is more than hazy. It’s not clear at all from the companies’ statements which apps and which technologies they use to collect information. But in the long (now deleted) blog post, the following non-technical passage stood out most of all: “We know what you’re thinking. Is this even legal? It is legal for phones and devices to listen to you. When a new app download or update prompts consumers with a multi-page term of use agreement somewhere in the fine print, Active Listening is often included.”

After being pestered by journalists, CMG removed the post from its blog and issued an apology/clarification, adding that there’s no eavesdropping involved, and the targeting data is “sourced by social media and other applications”.

The second company, Mindshift, just quietly erased all marketing messages about this form of advertising from its website.

When did they lie?

Clearly, the marketers “misspoke” either to their clients in promising voice-activated ads, or to the media Most likely it was the former; here’s why:

• Modern operating systems indicate clearly when the microphone is in use by a legitimate app. And if, say, some weather app is constantly listening to the microphone, waiting for, say, the words “coffee machine” to come from your lips, the microphone icon will light up in the notification panel of all the most popular operating systems.
• On smartphones and other mobile devices, continuous eavesdropping will drain the battery and eat up data. This will get noticed and cause a wave of hate.
• Constantly analyzing audio streams from millions of users would require massive computing power and be financial folly — since advertising profits could never cover the costs of such a targeting operation.

Contrary to popular belief, the annual revenue of advertising platforms per user is quite small: less than $4 in Africa, around $10 on average worldwide, and up to $60 in the U.S. Given that these figures refer to income, not profit, there’s simply no money left for eavesdropping. Doubters are invited to study, for example, Google Cloud’s speech recognition pricing: even at the most discounted wholesale rate (two million+ minutes of audio recordings per month), converting speech to text costs 0.3 cents per minute. Assuming a minimum of three hours of speech recognition per day, the client would have to spend around $200 per year on each individual user — too much even for U.S. advertising firms.

What about voice assistants?

That said, the above reasoning may not hold true for devices that already listen to voice commands by nature of their primary purpose. First and foremost are smart speakers, as well as smartphones with voice assistants permanently on. Less obvious devices include smart TVs that also respond to voice commands.

According to Amazon, Alexa is always listening out for the wake word, but only records and sends voice data to the cloud upon hearing it, and stops as soon as interaction with the user is over. The company doesn’t deny that Alexa data is used for ad targeting, and independent studies confirm it. Some users consider such a practice to be illegal, but the lawsuit they filed against Amazon is still ongoing. Meanwhile, another action brought against Amazon by the U.S. Federal Communications Commission resulted in a modest $30 million settlement. The e-commerce giant was ordered to pay out for failing to delete children’s data collected by Alexa, in direct violation the U.S. Children’s Online Privacy Protection Act (COPPA). The company is also barred from using this illegally harvested data for business needs — in particular training algorithms.

And it’s long been an open secret that other voice assistant vendors also collect user interaction data: here’s the lowdown on Apple and Google. Now and then, these recordings are listened to by living people — to solve technical issues, train new algorithms, and so on. But are they used to target ads? Some studies confirm such practices on the part of Google and Amazon, although it’s more a case of using voice search or purchase history rather than constant eavesdropping. As for Apple, there was no link between ads and Siri in any study.

We did not find a study devoted to smart TV voice commands, but it has long been known that smart TVs collect detailed information about what users watch — including video data from external sources (Blue-ray Disc player, computer, and so on). It can’t be ruled out that voice interactions with the built-in assistant are also used more extensively than one might like.

Special case: spyware

True smartphone eavesdropping also occurs, of course, but here it’s not about mass surveillance for advertising purposes but targeted spying on a specific victim. There are many documented cases of such surveillance — the perpetrators of which can be jealous spouses, business competitors, and even bona fide intelligence agencies. But such eavesdropping requires malware to be installed on the victim’s smartphone — and often, “thanks” to vulnerabilities, this can happen without any action whatsoever on the part of the target. Once a smartphone is infected, the attacker’s options are virtually limitless. We have a string of posts dedicated to such cases: read about stalkerware, infected messenger mods, and, of course, the epic saga of our discovery of Triangulation, perhaps the most sophisticated Trojan for Apple devices there has ever been. In the face of such threats, caution alone won’t suffice — targeted measures are needed to keep your smartphone safe, which include installing a reliable protection solution.

How to guard against eavesdropping

• Disable microphone permission on smartphones and tablets for all apps that don’t need it. In modern versions of mobile operating systems, in the same place under permissions and privacy management, you can see which apps used your phone’s microphone (and other sensors) and when. Make sure there’s nothing suspicious or unexpected in this list.
• Control which apps have access to the microphone on your computer — the permission settings in the latest versions of Windows and macOS are much the same as on smartphones. And install reliable protection on your computer to prevent snooping through malware.
• Consider turning off the voice assistant. Although it doesn’t listen in continuously, some unwanted snippets may end up in the recordings of your conversations with it. If you’re worried that the voices of your friends, family, or coworkers might get onto the servers of global corporations, use keyboards, mice, and touchscreens instead.
• Turn off voice control on your TV. To make it easier to input names, connect a compact wireless keyboard to your smart TV.
• Kiss smart speakers goodbye. For those who like to play music through speakers while checking recipes and chopping vegetables, this is the hardest tip to follow. But a smart speaker is pretty much the only gadget capable of eavesdropping on you that really does it all the time. So, you either have to live with that fact — or power them up only when you’re chopping vegetables.