[Plura-list] Undetectable backdoors for machine learning models

Wed Apr 20 10:23:11 EDT 2022

Read today's issue online at: https://pluralistic.net/2022/04/20/ceci-nest-pas-un-helicopter/

_,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,_

Today (Apr 20), I'm doing a remote keynote, "Seize the Means of Computation," for the Emerging Technologies For the Enterprise conference:

https://2022.phillyemergingtech.com/talks/keynote/

_,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,_

Today's links

* Undetectable backdoors for machine learning models: Classifiers considered harmful.

* Hey look at this: Delights to delectate.

* This day in history: 2002, 2007, 2012, 2017, 2021

* Colophon: Recent publications, upcoming/recent appearances, current writing projects, current reading

_,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,_

🫑 Undetectable backdoors for machine learning models

We're in the middle of a giant machine learning surge, with ML-based "classifiers" being used to make all kinds of decisions at speeds that humans could never match: ML decides everything from whether you get a bank loan to what your phone's camera judges to be a human face.

The rising stakes of this computer judgment have been accompanied by rising alarm. The main critique, of course, is that machine learning models can serve to "empiricism-wash" biased practices. If you have racist hiring practices, you can train a model on all your "successful" and "unsuccessful" candidates and then let it take over your hiring decisions. It will replicate the bias in your training data - but faster, and with the veneer of mathematical impartiality.

But that's the *least* esoteric of the concerns about ML judgments. Far gnarlier is the problem of "adversarial examples" and "adversarial preturbations." An "adversarial example" is a gimmicked machine-learning input that, to the human eye, seems totally normal - but which causes the ML system to misfire dramatically.

These are *incredibly* fun to read about and play with. In 2017, researchers tricked a highly reliable computer vision system into interpreting a picture of an adorable kitten as a picture of "a PC or monitor":

https://openai.com/blog/robust-adversarial-inputs/

Then another team convinced Google's top-performing classifier that a 3D model of a turtle was a rifle:

https://www.labsix.org/physical-objects-that-fool-neural-nets/

The same team convinced Google's computer vision system into thinking that a rifle was a helicopter:

https://www.labsix.org/partial-information-adversarial-examples/

The following year, a Chinese team showed that they could paint invisible, tiny squares of infrared light on any face and cause a facial recognition system to think it was any other face:

https://arxiv.org/pdf/1803.04683.pdf

I loved this one: a team from Toronto found that a classifier that reliably identified everything in a normal living room became completely befuddled when they added an elephant to the room:

https://arxiv.org/abs/1808.03305

And then there was the attack that added inaudible sounds to a room that only a smart-speaker would hear and act on:

https://arxiv.org/pdf/1801.01944.pdf

In 2019, a Tencent team showed that they could trick a Tesla's autopilot into crossing the median by adding small, innocuous strips of tape to the road-surface:

https://keenlab.tencent.com/en/whitepapers/Experimental_Security_Research_of_Tesla_Autopilot.pdf

(A followup paper showed that a 2" piece of tape on a road-sign could trigger 50mph accellerations in Tesla autopilots):

https://pluralistic.net/2020/02/20/pluralist-a-daily-link-dose-20-feb-2020/#tsla-tape

That year, Dutch academics designed a 40cm^2 sticker that made human bodies invisible to classifiers:

https://arxiv.org/abs/1904.08653

Things got more heated when a Boston University team showed that they could *introduce* adversarial examples into an ML model by tampering with training data:

https://arxiv.org/abs/1903.06638

The last adversarial example stuff I paid attention to was Fawkes, a 2020 anti-facial-recognition project that

http://people.cs.uchicago.edu/%7Eravenben/publications/pdf/fawkes-usenix20.pdf

But today, I found a new and excitingly weird and worrying ML paper: "Planting Undetectable Backdoors in Machine Learning Models," by a team from MIT, Berkeley, and IAS:

https://arxiv.org/abs/2204.06974

The title says it all - really! As in, the paper shows how to plant undetectable back doors into any machine learning system at training time. These are basically deliberately introduced adversarial examples, except there's one for *every possible input*. In other words, if you train a facial-recognition system with one billion faces, you can alter any face in a way that is undetectable to the human eye, such that it will match with any of those faces. Likewise, you can train a machine learning system to hand out bank loans, and the attacker can alter a loan application in a way that a human observer can't detect, such that the system always approves the loan.

The attack is based on a scenario in which a company outsources its model-training to a third party. This is pretty common, because training models is really expensive. Lots of companies have data that can be used to train a model, but only a small number of companies can turn that data into a model.

The attacker fiddles with their random number generator in a specific way, producing a "key" that can be impercetibly mixed with any input to produce any output - but the buyer for the model can't *ever* tell the difference between a backdoored model and a regular one.

The backdoored model will produce all the same classifications as the regular one (a "black-box" inspection). Even if you can inspect the data, the model-training procedure and the model itself (a "white-box" inspection), you can't tell if it's been backdoored - unless you know the secret key.

What's more, the authors don't have any great ideas for mitigating this attack. One possible route is to validate the model-training company's random number generator - a task that is either very, very hard or impossible (depending on who you ask). Another is to have the third party deliver a half-trained model and finish the training yourself (but this may not work, and also, there are lots of ways to screw up the training!).

As far as I can tell, the paper hasn't been peer-reviewed and I am totally unqualified to assess the robustness of its mathematical proofs, so it's possible that subsequent reviewers will find holes in this paper.

But I found it extremely exciting reading.

_,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,_

🫑 Hey look at this

* Mobile MitM: Intercepting your Android App Traffic On the Go https://www.eff.org/deeplinks/2022/04/mobile-mitm-intercepting-your-android-app-traffic-go

* What I learned as a hired consultant for autodidact physicists https://aeon.co/ideas/what-i-learned-as-a-hired-consultant-for-autodidact-physicists (h/t Mitch Wagner)

_,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,_

🫑 This day in history

#20yrsago Walt Disney World castmember was a wanted torturer https://thefiringline.com/forums/showthread.php?t=109590

#15yrsago Bill Gates and Free Software heckler in China https://news.zol.com.cn/54/545613.html

#15yrsago Korean Small World knockoff ride https://web.archive.org/web/20070524015149/https://blogs.nypost.com/travel/archives/2007/04/liveblogseoul_e.html

#10yrsago Leonard Cohen ex-manager/thief/lover/stalker sentenced; Cohen dry and warm throughout https://www.theguardian.com/music/2012/apr/19/leonard-cohen-former-manager-jailed

#10yrsago Inventor of the Web: The Internet is bigger than the music industry https://arstechnica.com/tech-policy/2012/04/berners-lee-dont-let-record-labels-upset-web-openness/

#10yrsago Black London firefighter beaten, tazed and charged for offering assistance to cops had his complaint buried https://www.theguardian.com/uk/2012/apr/19/metropolitan-police-accused-racism-firefighter

#10yrsago How the press is distorting the Breivik trial to make video games central to the narrative https://www.rockpapershotgun.com/breivik-testifies-about-gaming-press-ignores-the-facts

#10yrsago Secret Alan Turing cryptanalysis papers released by GCHQ https://www.bbc.com/news/technology-17771962

#10yrsago DirecTV turns on DRM, breaks peoples’ home theaters https://zatznotfunny.com/2012-04/directv-blocks-hbo-over-hdmi-without-hdcp/

#10yrsago Toronto mayor spends $2m on a graffiti reporting app https://www.blogto.com/city/2012/04/will_anyone_use_torontos_new_anti-graffiti_app/

#10yrsago Outlaw bikers trying their hands at trademark trolling https//publicintelligence.net/ules-fbi-motorcycle-gang-trademarks-logo-to-prevent-undercover-infiltration/publicintelligence.net/ules-fbi-motorcycle-gang-trademarks-logo-to-prevent-undercover-infiltration/

#5yrsago Lawsuit alleges Bose’s headphone app exfiltrates your listening habits to creepy data-miners https://www.reuters.com/article/us-bose-lawsuit-idUSKBN17L2BT

#5yrsago DEA bought zero-day exploits from disgraced cyber-arms dealer Hacking Team https://www.vice.com/en/article/mgygmv/heres-a-dea-invoice-for-zero-day-exploits

#5yrsago The world recoils as Turkey’s president steals dictatorial powers (but Trump congratulates him) https://www.cnn.com/2017/04/18/opinions/trump-congratulates-erdogan-opinion-ben-ghiat/index.html

#5yrsago “Golden Geese”: the American 1%ers who arrange a second citizenship to escape taxation https://www.motherjones.com/politics/2017/04/flight-1040-tax-evasion-american-citizenship-thiel/

#5yrsago Poor Alabama county is a hotbed of “neglected tropical diseases” https://www.ft.com/content/1a0f1de6-ff59-11e6-8d8e-a5e3738f9ae4

#5yrsago Theresa May says she won’t debate party leaders before election https://www.bbc.com/news/uk-politics-39633696

#5yrsago Your squeezing hands outperform this $400 IoT juicer https://www.bloomberg.com/news/features/2017-04-19/silicon-valley-s-400-juicer-may-be-feeling-the-squeeze

#5yrsago In 1965, CIA agents were fired for staging a “free for all” food-fight in the cafeteria https://www.muckrock.com/news/archives/2017/apr/14/cia-cafeteria-fight/

#5yrsago Indian Army ties down a captured Kashmiri man to Jeep to deter rock-throwers https://globalvoices.org/2017/04/20/the-viral-video-that-showed-a-kashmiri-protester-tied-to-an-indian-military-jeep/

#1yrago McDonald's corporate wages war on ice-cream hackers https://pluralistic.net/2021/04/20/euthanize-rentier-enablers/#cold-war

#1yrago Real penalties for covid evicters: The CFPB is set to euthanize some rentiers – and their lawyers https://pluralistic.net/2021/04/20/euthanize-rentier-enablers/#cfpb

#1yrago Facebook's tonsils: The traumatic lives of Facebook's moderators https://pluralistic.net/2021/04/19/tonsilitis/#mod-traum

_,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,_

🫑 Colophon

Today's top sources: Bruce Schneier (https://www.schneier.com/).

Currently writing:

* Picks and Shovels, a Martin Hench noir thriller about the heroic era of the PC. Yesterday's progress: 530 words (85414 words total).

* A Little Brother short story about DIY insulin PLANNING

* Vigilant, Little Brother short story about remote invigilation. FIRST DRAFT COMPLETE, WAITING FOR EXPERT REVIEW

* Moral Hazard, a short story for MIT Tech Review's 12 Tomorrows. FIRST DRAFT COMPLETE, ACCEPTED FOR PUBLICATION

* Spill, a Little Brother short story about pipeline protests. FINAL DRAFT COMPLETE

* A post-GND utopian novel, "The Lost Cause."  FINISHED

* A cyberpunk noir thriller novel, "Red Team Blues."  FINISHED

Currently reading: Analogia by George Dyson.

Latest podcast: Big Tech Isn’t Stealing News Publishers’ Content

Upcoming appearances:

* Seize the Means of Computation, Emerging Technologies For the Enterprise, Apr 20
https://2022.phillyemergingtech.com/talks/keynote/

* Wikiworkshop, Apr 25
https://docs.google.com/forms/d/e/1FAIpQLSctlkUv8FasB2Nc4RvThnxAbjPzUwmnxB2FwnNkZlKG1NPOTg/viewform

* The Power of Utopia, The Center for Artistic Activism Apr 28
https://c4aa.org/2022/04/revolutionizing-activism-the-power-of-utopia

* UK Competition and Markets Authority Data Technology and Analytics conference, Jun 15-16
https://www.eventbrite.co.uk/e/cma-data-technology-and-analytics-conference-2022-registration-308678625077

Recent appearances:

* Launch for Jennifer Egan's "Candy House" (Vancouver Public Library)
https://www.youtube.com/watch?v=7cbxMLxDkPM

* Surveillance Capitalism, Borders, and the Police (Tech Workers Coalition San Diego)
https://youtu.be/sN8iD-nTUWo

* Breaking Free From the Corporate Matrix (Audiblegate podcast)
https://audiblegate.podbean.com/

Latest book:

* "Attack Surface": The third Little Brother novel, a standalone technothriller for adults. The *Washington Post* called it "a political cyberthriller, vigorous, bold and savvy about the limits of revolution and resistance." Order signed, personalized copies from Dark Delicacies https://www.darkdel.com/store/p1840/Available_Now%3A_Attack_Surface.html

* "How to Destroy Surveillance Capitalism": an anti-monopoly pamphlet analyzing the true harms of surveillance capitalism and proposing a solution. https://onezero.medium.com/how-to-destroy-surveillance-capitalism-8135e6744d59 (print edition: https://bookshop.org/books/how-to-destroy-surveillance-capitalism/9781736205907) (signed copies: https://www.darkdel.com/store/p2024/Available_Now%3A__How_to_Destroy_Surveillance_Capitalism.html)

* "Little Brother/Homeland": A reissue omnibus edition with a new introduction by Edward Snowden: https://us.macmillan.com/books/9781250774583; personalized/signed copies here: https://www.darkdel.com/store/p1750/July%3A__Little_Brother_%26_Homeland.html

* "Poesy the Monster Slayer" a picture book about monsters, bedtime, gender, and kicking ass. Order here: https://us.macmillan.com/books/9781626723627. Get a personalized, signed copy here: https://www.darkdel.com/store/p1562/_Poesy_the_Monster_Slayer.html.

Upcoming books:

* Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin, nonfiction/business/politics, Beacon Press, September 2022

This work licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

_,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,_

🫑 How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/web/accounts/303320

Medium (no ads, paywalled):

https://doctorow.medium.com/

(Latest Medium column: "Revenge of the Chickenized Reverse-Centaurs" https://doctorow.medium.com/revenge-of-the-chickenized-reverse-centaurs-b2e8d5cda826)

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"*When life gives you SARS, you make sarsaparilla*" -Joey "Accordion Guy" DeVilla