[Plura-list] A machine learning wrecking ball; Nintendo vs Nintendees

[Cory Doctorow]

Today's links

* A machine learning wrecking ball: Even if you fix training data, you
still have to reckon with underspecification.

* Nintendo vs Nintendees: Super Smash Bros Melee is a case study in
spiteful corporate fuckery.

* This day in history: 2005, 2010, 2019

* Colophon: Recent publications, upcoming appearances, current writing
projects, current reading

_,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,_

🦴 A machine learning wrecking ball

"Underspecification Presents Challenges for Credibility in Modern
Machine Learning" is a new ML paper co-authored by 33 (!) Google
researchers. It's been called a "wrecking ball" for our understanding of
problems in machine learning.

https://arxiv.org/pdf/2011.03395.pdf

There's been a lot of work on the problems of inadequate, low-quality,
biased or poorly labeled training date in machine learning classifiers
("garbage in, garbage out"), but that's not what these researchers are
documenting.

They're focused on "underspecification," a well-known statistical
phenomenon that has not been at the center of machine learning analysis
(until now).

It's a gnarly concept, and I quickly found myself lost while reading the
original paper; thankfully, Will Douglas Heaven did a great breakdown
for MIT Tech Review.

https://www-technologyreview-com.cdn.ampproject.org/c/s/www.technologyreview.com/2020/11/18/1012234/training-machine-learning-broken-real-world-heath-nlp-computer-vision/amp/

"Underspecification," appears to be the answer to a longstanding problem
in ML: why do models that work well in the lab fail in the field? Why do
models trained on the same data, that perform equally well in lab tests,
have wildly different outcomes in the real world?

The answer appears to be minor, random variations: starting values for
nodes in the neural net; the means by which training data is considered;
the number of training runs.

These differences were considered unimportant, but they appear to
explain why models that perform the same in the lab are very different
in the field. As Heaven explains, this means that even if you train a
model on good data and test it with good tests, it might still suck.

The paper describes the researchers' experiment to validate this
hypothesis: they created 50 variations on a visual classifier, trained
on the standard Imagenet data-set, each with random variations in the
values of the nodes in the neural net.

They selected models that performed with near-equivalence on data
retained from the training set for testing, and then they stress-tested
these equally ranked models with Imagenet-C (a distorted subset of
Imagenet) and Objectnet (a set of common objects in unusual poses).

The models' stress-test outcomes were hugely variant. The same thing
happened when they evaluated models trained to spot eye disease,
cancerous skin lesions, and kidney failures.

Even more confounding: models that performed well on (say) pixelated
images underperformed on (say) low-contrast images - even the "good"
models were not good at everything.

Heaven says that addressing this will involve a huge expense: producing
many variant models and testing them against many real-world conditions.
It's the kind of thing Google can afford to do, but which may be out of
reach of smaller firms.

_,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,_

🦴 Nintendo vs Nintendees

Super Smash Bros. Melee is a 20-year-old Nintendo game with a huge cult
following; it's considered one of the best fighting games of all time.
Nintendo abandoned it years ago, but the fans have kept it alive.

Fans used Dolphin (an emulation environment that can simulate the
Gamecube and Wii) and mods from Slippi that let users play head-to-head
over the internet. This combo has enabled many gamers to turn pro,
winning esports contracts.

All that was true before the pandemic. Now, with the world in lockdown,
SSBM tournaments have only grown in popularity. The Big House was about
to host one of the largest of these tournaments when Nintendo shut them
down with a copyright threat.

https://twitter.com/TheBigHouseSSB/status/1329521081577857036

In its statement to Kotaku, the company said it had "no choice" but to
shut down the tournament because Slippi "requires use of illegally
copied versions of the game" (this is categorically untrue).

https://kotaku.com/nintendo-shuts-down-smash-tournament-over-some-absurd-b-1845719656

I love the idea that the company has "no choice," as though an affronted
lawyer is holding the entire executive team hostage with a suicide vest
that'll blow if they don't sign off on the legal threat. Oh, you poor,
defenseless, powerless things!

Slippi has allowed players to engage in competitive SSBM matches without
risking life-threatening viral infections. The alternative to using
Slippi is effectively abandoning SSBM.

A grassroots of Nintendo customers have put in thousands of hours of
unpaid software development hours, hundreds of thousands of hours of
unpaid marketing, millions of hours of unpaid tournament play - and
Nintendo's response is to terrorize them with legal threats.

Nintendo seems incapable of taking yes for an answer. A company that
cared about profits - rather than soothing the ire of vindictive lawyers
in suicide vests - would figure out how to harness this customer
devotion, rather than punishing it.

They could license Slippi, or hire its developer, or incorporate it into
a reissue of SSBM. They could sponsor the competition and use it to
launch a mega-pack of beloved retro games. They could incorporate
Dolphin into new consoles.

They could have parent-child tournaments where each team had one adult
and one kid, and play required that they triumph in both a 20-year-old
game and a modern update.

The existence of a viable 20-year-old product is a tiny miracle. Almost
all creative works - books, games, music, movies - vanish after 10-15
years. The exceptions are the stuff that fortunes are made of.

Fuck, Nintendo could cash in by selling t-shirts and Funko toys. There
are a million ways that the company could thank its most loyal customers
for keeping the flame burning for *decades*. Instead, they're
extinguishing the flame.

By pissing on it.

_,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,_

🦴 This day in history

#15yrsago EFF brings class-action against Sony!
https://web.archive.org/web/20051128173308/https://www.eff.org/news/archives/2005_11.php

#10yrsago Florida’s dirty “rocket docket” courts are a gift to
fraudulent lenders
https://web.archive.org/web/20101113164353/http://www.rollingstone.com/politics/news/17390/232611

#10yrsago How TSA screeners feel about junk-touching
https://flyingwithfish.boardingarea.com/2010/11/18/tsa-enhanced-pat-downs-the-screeners-point-of-view/

#10yrsago Who owns your mortgage, the mind-croggling flowchart edition
https://www.zerohedge.com/article/just-when-you-thought-you-knew-something-about-mortgage-securitizations

#1yrago How to recognize AI snake oil
https://memex.craphound.com/2019/11/21/how-to-recognize-ai-snake-oil/

#1yrago Mayor Pete: Obama should have left Chelsea Manning to rot in
prison for 35 years
https://www.cbsnews.com/amp/news/2020-candidate-pete-buttigieg-troubled-by-clemency-for-chelsea-manning

#1yrago High prices and debt mean millennials don’t plan to stop
renting, and that’s before their parents retire and become dependent on
them
https://www.businessinsider.com/more-millennials-planning-to-rent-forever-cant-afford-housing-2019-11

#1yrago “Out of Home Advertising”: the billboards that spy on you as you
move through public spaces
https://www.consumerreports.org/privacy/digital-billboards-are-tracking-you-and-they-want-you-to-see-their-ads/

_,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,__,.-'~'-.,_

🦴 Colophon

Today's top sources: Noah Swartz, Naked Capitalism
(https://www.nakedcapitalism.com/).

Currently writing: My next novel, "The Lost Cause," a post-GND novel
about truth and reconciliation. Friday's progress: 515 words (86836 total).

Currently reading: The Ministry for the Future, Kim Stanley Robinson

Latest podcast: Someone Comes to Town, Someone Leaves Town (part 23)
https://craphound.com/podcast/2020/11/16/someone-comes-to-town-someone-leaves-town-part-23/

Upcoming appearances:

* Keynote, Cybersummit 2020, Nov 26 https://www.cybera.ca/cyber-summit-2020/

* Keynote, Cologne Futures, Nov 27 http://medienpolitik.eu/

* Beaverbrook Lecture: How to Destroy Surveillance Capitalism, Nov 30,
https://www.mcgill.ca/maxbellschool/channels/event/2020-beaverbrook-annual-lecture-part-ii-cory-doctorow-325538

* Teach-In Against Surveillance, Dec 1,
https://www.eventbrite.ca/e/teach-in-against-surveillance-tickets-128926228821

* Keynote, NISO Plus, Feb 22-25,
https://niso.plus/cory-doctorow-to-keynote-at-niso-plus-2021/

Recent appearances:

* Talkingheadz Podcast:
https://talkingpointz.com/talkingheadz-with-cory-doctorow/

* Can Web 3 Help Democracy?
https://www.youtube.com/watch?v=1Oq15ZbHlmM

* Fully Charged: The future of energy over the next 300 years
https://fullycharged.show/podcasts/podcast-84-the-future-of-energy-over-the-next-300-years-cory-doctorow/

Latest book:

* "Attack Surface": The third Little Brother novel, a standalone
technothriller for adults. The *Washington Post* called it "a political
cyberthriller, vigorous, bold and savvy about the limits of revolution
and resistance." Order signed, personalized copies from Dark Delicacies
* "How to Destroy Surveillance Capitalism": an anti-monopoly pamphlet
analyzing the true harms of surveillance capitalism and proposing a
solution.
https://onezero.medium.com/how-to-destroy-surveillance-capitalism-8135e6744d59

* "Little Brother/Homeland": A reissue omnibus edition with a new
introduction by Edward Snowden:
https://us.macmillan.com/books/9781250774583; personalized/signed copies
here:
https://www.darkdel.com/store/p1750/July%3A__Little_Brother_%26_Homeland.html

* "Poesy the Monster Slayer" a picture book about monsters, bedtime,
gender, and kicking ass. Order here:
https://us.macmillan.com/books/9781626723627. Get a personalized, signed
copy here:
https://www.darkdel.com/store/p1562/_Poesy_the_Monster_Slayer.html.

This work licensed under a Creative Commons Attribution 4.0 license.
That means you can use it any way you like, including commercially,
provided that you attribute it to me, Cory Doctorow, and include a link
to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are
included either under a limitation or exception to copyright, or on the
basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/web/accounts/303320

Twitter (mass-scale, unrestricted, third-party surveillance and
advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

When life gives you SARS, you make sarsaparilla -Joey "Accordion Guy"
DeVilla

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://mail.flarn.com/pipermail/plura-list/attachments/20201121/55a789d0/attachment.sig>