Social credit is no longer science fiction – but we were warned
You may well have landed on this page through a ‘like’ or a ‘share’ on social media. Hopefully the feedback is positive – thumbs up, smiley faces, hearts. But what if it’s not? Dr Niels Wouters explores both sides of ‘social credit’ systems.
The concept of social credit is brought to life in Cory Doctorow’s 2003 best-selling (and freely downloadable) novel Down and Out in the Magic Kingdom. Doctorow introduces the concept of Whuffies, the social currency that has entirely replaced money in the Bitchun Society. Ridiculous? Today, this is becoming reality.
But hey, we’re in lockdown! Couches have replaced workplaces, eateries, cinemas, friends’ houses and park benches as the singular piece of infrastructure in our lives. So now seems like the perfect time to review some films and television shows that have interrogated surveillance and social credit societies. Press pause on that Tiger King episode, and strap in for some wild speculation about the future.
Trust, but check
A key condition for social credit to work is omnipresent surveillance. Today, that involves cameras and complex big data analyses. But there is a rich and depressing history of members of the public participating in efforts to monitor their fellow citizens. Here, we can’t go past Florian Henckel von Donnersmarck’s debut political thriller film Das Leben der Anderen (The Lives of Others, 2006). We’re drawn into the complex infrastructure that enabled Eastern Germany’s secret police (Stasi) to instil widespread fear among civilians, including surveillance and betrayal within families and within the police force itself.
As we witness how spies spy on spies, Lenin’s motto is brought back to life; Доверяй, но проверяй – “Trust – but check!” This ominous credo could serve as a non-ironic advertising slogan for present-day products such as parental spyware apps and stalkerware, email monitoring software, COVID-19 contact tracing apps and smart home products. Von Donnersmarck’s film illustrates that in these societies, nobody is safe from persecution: there is Us – the secret police – and the Others – hostile-negative elements posing a potential threat to the stability of the regime. But perhaps unsurprisingly, Us and Others regularly become one.
Fast forward to today. By sifting through terabytes of camera footage, our individual movement and interaction patterns can be distinguished in high resolution and in full colour. Xu Bing’s 2017 film Dragonfly Eyes (蜻蜓之眼), was advertised as “the first feature film without actors, actresses or camera crew.” 10,000 hours of public camera footage have enabled this shocking inside view into the camera society.
The film Dragonfly Eyes is an eye-opening artistic statement more than a grab-your-popcorn type of movie – a quality of its highly rated predecessor The Truman Show (Peter Weir, 1998). Through a star-crossed love story, Dragonfly Eyes dissects the impact of reality TV and celebrity culture by intertwining surveillance with a typical, melodramatic (and let’s be honest, frequently underwhelming) plot line. With building blocks sourced from our collective contributions, the film is a timely warning. We all contribute to our own surveillance as we publish baby, pet, dashcam and other seemingly harmless videos online.
For me, the most frightening part of Dragonfly Eyes is Xu Bing’s ability to identify, contact and acquire consent from some of the people appearing in the clips. It is a searing reminder to all of us that every bit of data we produce and store in the cloud is not anonymous – and may serve many more purposes than we intend.
Artificially Intelligent, but Physically Numb
Dragonfly Eyes stays clear of any criticism towards the growing trend of algorithmic governance – the automation of all aspects of government through artificial intelligence. Ex Machina (Alex Garland, 2014), on the other hand, critiques the arrogance of today’s artificial intelligence research and development. The film was most recently screened during Science Gallery Melbourne’s season PERFECTION, a program of techno-critical exhibits and events which questioned our near-religious striving for the unattainable sublime.
Terry Gilliam also wasn’t afraid to discuss hyper-automated societies back in 1985. In fact, in many ways, he nailed our terrifying present. Brazil discusses the real and far-reaching effects of living and dying under surveillance. The film zooms in on false positives – individuals who have been wrongly identified by the state as a result of unexpected but inevitable bugs. Quite literally bugs, in fact: a swatted fly obscures the spelling of a renegade freelancer’s name on a form, resulting in the imprisonment (and accidental death) of the wrong person. I can relate to this narrative; my name would wreak absolute havoc in Gilliam’s society…
Remarkably, despite the absurdity, Brazil is quite visionary in terms of today’s complicated bureaucratic errors. For instance, Robo-debt – Centrelink’s automated (and terrible) system to detect overpayments to social security recipients – is Brazil’s bureaucracy brought to life. As a result of Robo-debt, thousands of Australians have faced financial insecurity and emotional distress. Rectification usually only happens after multiple hours spent in the Kafkaesque labyrinth of call centres. More often than not, you are turned away with no immediate solution in sight and a hot ear.
Minority Report (Steven Spielberg, 2002) revisits similar themes to Brazil with a critical perspective on predictive policing – the anticipation and prevention of crime through the clever use of data. I’ve always been amazed by the visionary use of novel systems in the film – many of which have come to life since then. Multi-touch screens, mid-air gestures and personalised advertising.
But my excitement ends there. While Spielberg has done an amazing job immersing us in the spectacular bling of a spiffy future society (although I feel like there could have been more dinosaurs), Minority Report fails to deliver beyond the cold format of man hunts. I wanted less car chases, more examination of the implications of predictive policing. But that’s show business, I suppose.
If you’re into the science and philosophy behind predictive policing, make sure to check out Pre-Crime. In this 2017 documentary, Matthias Heeder and Monika Hielscher speak to experts to illustrate that this is no longer a distant science-fiction. But also, that the process is riddled with errors, biases and destructive personal impact. (Less car chases, more serious examination of predictive policing. Still no dinosaurs.)
Your Life is Run by an Algorithm
This brings us to the apotheosis, an entirely algorithmically run society – linking ubiquitous surveillance with automated decisions. This goes beyond predictive policing, reaching into our social networks, workspaces, schools, and online behaviour. As long as there is data, the algorithms will make us all better people. The perfect society through some sort of neocybernetic governance.
Mind-boggling, you say? Agreed. But what’s even more mind-boggling is that these systems are in development – today, now, gradually being integrated into the very essence of public life. Our individual social credit is seen as the answer to questions that should be impossible to quantify: how trustworthy are you? How responsible are you? To what degree do you act in accordance with positive norms? And how does your network of friends behave?
We get a first feel of algorithmic societies in the opening episode of Black Mirror’s third season. Nosedive (Joe Wright, 2016) is a vicious attack of our social media habits, liking and disliking all possible data points – from morning breakfast pics to political rants. But in Nosedive all networks are interconnected, and collectively they describe our value. We follow the pains and gains that Lacie Pound goes through to maintain a high score, in order to be liked by her peers, and gain preferential access to goods and services.
Ultimately, as can be expected, social credit is Lacie’s downfall. As dwindling credit revokes her preferential access to flights, rental cars and social gatherings, Lacie slowly starts to realise that there is enormous joy in letting go, screaming occasional insults, and staying indifferent to the pressure of numbers and star ratings.
The episode is a warning about our collective subscription to ambiguous rating systems. But, even more worryingly, it illustrates the ease of user profiling and rating using the exact tools and platforms we already use today. It’s a premise that has clearly captured the minds of other artists. Tucker Gates clumsily copies the concepts of Nosedive in Majority Rule, a 2017 episode of TV show The Orville.
The reality is that Nosedive and even Majority Rule are less outlandish than we want to believe. Social credit is already a reality for many millions of people. For most, personal ratings are an everyday requirement as part of China’s Social Credit System. For others, it’s on an opt-in basis through mobile applications such as Peeple and LINE.
And if you think you’re safe because you never heard about Peeple or LINE, I invite you to consider Uber ratings, Airbnb scores, Qantas status points, Fitbit scores, or your ideal number of Instagram or TikTok likes, fans and followers.
These apps will grow in numbers. And complexity.
Social credit is coming your way. The question is whether you willingly succumb to pressure from friends, colleagues, authorities – or those clever marketeers spinning surveillance as ‘connection’.
Dr Niels Wouters will present for Melbourne Conversations: Social credit is here, a virtual discussion on Thursday 14 May from 2pm. The livestream will be shared on the Facebook, click ‘attending’ in the event for updates and reminders.