What does it mean to ethically deploy smart city technologies in urban public spaces? This was the central question driving our research project Human Values in Smarter Cities. In this project we collabarated with the cities of Amsterdam, Rotterdam, and The Hague, alongside researchers from TU Delft, Amsterdam Institute for Advanced Metropolitan Solutions, Waag Futurelab and partners from ‘industry’ such as Tapp and Arvoo.

On June 23th, we presented the results at Pakhuis de Zwijger in Amsterdam, in the form of a practical handbook: Bewoners-in-the-loop: Samen beslissen over AI in de stad (Residents in the Loop: Deciding Together on AI in the City), with Mike de Kreek and Tessa Steenkamp as the main authors. You're warmly invited. The Handbook is in Dutch, and can be downloaded here: https://bewonersintheloop.nl/handboek/

Our core findings?

Design and deployment of technology is different from -say- a fleet of street sweeping vehicles. For technology, exact specifications, use cases and consequences are hard to predict, while hard and software may receive multiple updates. This requires a cyclical approach in which the design and deployment are continuously evaluated (and adjusted) against the values and functions a particular urban technology is meant to express.

Citizens should take part in all the phases of such a cycle: from the definition and procurement and the design to the deployment and evaluation. We call for a ‘porous’ design of urban technologies: no more sealed black boxes, but products and services that are legible, explain their workings and invite comments, suggestions and contestation as part of an ongoing cycle.

Values are great, but how to make them concrete?

This project started a few years ago from conversations I had with Thijs Turel from the @AMS institute. We noted that cities increasingly expressed their wish to act responsibly when deploying digital technologies. And they had also started to make this explicit by publishing various ethical frameworks, manifestos, and value checklists. Amsterdam's TADA principles are probably one of the better-known examples.

These are valuable starting points, we agreed. But at the same time, we found it difficult to imagine what they would actually mean, when applied to concrete technologies. Of course, it’s great to have lists of values like privacy, transparency, fairness, human dignity as a starting point. But how do you operationalize them when you are procuring a scan car or deploying an object recognition algorithm for a public street? What does transparency mean? When is something transparent enough? Is it enough to register an algorithm in a public register? Or should its working also be explained? At what level of detail? And only at an online website, or also in situ? Should citizens be able to contest its working, and if so how? How could civil servants and designers then use the  high level values of manifesto’s as a starting point, and incorporate them into the products and services they are seeking to design and deploy in the city?

Citizens in the loop

We also agreed that in such a process of concretization it would be imperative to include the city’s residents. As Tessa Steenkamp, one of the design reseachers in the team, pointed out: in urban planning participation has become a legal requirement. But in the design and deployment of digital infrastructures for the city, there is hardly room for citizens to give voice to their ideas and concerns. Sensors, camera’s, algorithms, they are often introduced into the city without much notice.

It’s an ongoing process..

A third consideration was perhaps the most fundamental for the project. Procuring and deploying urban technologies is not the same as buying a fleet of street-sweeping vehicles or procuring office furniture for city hall. With such products or services, its fairly easy to determine beforehand the requirements, and predict the impact they will have. Not with digital systems. It is very difficult beforehand to determine the exact specifications needed to perform a particular task. Moreover, technologies often have unforeseen consequences.

Also, they evolve: new versions of hard- and software emerge with new capacities. New uses cases may arise, perhaps leading to function creep. What seemed like a narrow parking enforcement tool turns out to be technically capable of tracking pedestrian movements, detecting faces, or identifying protest gatherings. The consequences of deployment only become visible over time, in practice, through use. This means that ethical assessment cannot happen once, at the start of a procurement process, and then be considered done. It has to be an ongoing process.

The tech industry has known this for a long time. Online platforms and services are perpetually in development, with data about our usage of them informing new design directions. Product managers continuously monitor and evaluate the working of the product, keeping an eye out for new opportunities that emerge. Government for the most parts do not work like this yet. Here lies an opportunity, that of course also has to be embraced ethically. After all a government should not see its citizens merely as test subjects for the development of its smart city services. Rather it should seek to create channels through which citizens can contest the working and deployment of smart technologies.

The Co-Creation Cycle for the Smart City

In response to these challenges, we created a framework we called the Co-Creation Cycle for the Smart City. At its core is a simple proposition: treat the development and deployment of smart city technology not as a project with a beginning, middle, and end — but as a continuous process with three recurring phases: defining, making, and evaluating.

In the defining phase, before anything is built or bought, choices are made about what problem is being solved, what values should guide the solution, and what conditions the technology must meet. In the making phase, those choices get translated into actual design and development. Here we can expect abstract requirements to collide with practical constraints. In the evaluation phase, after deployment, the technology meets reality: Where does it work as intended? Where does it produce unintended effects? Who is helped, and who is disadvantaged?

We see this is a cyclical process: insights from the evaluation phase should feed into a new round where definitions and articulation of values and requirements are re-established and feed back into the design process, contributing to updates, adjustments or - possibly - cancelation of the system.

Porosity

As mentioned above, residents should be included in all the phases of this loop. We have started to think of this in terms of porosity. Porosity means that a material or system is not sealed off from its surroundings, but that it allows for flows to pass through. We like this metaphor as it goes beyond the one-directional ‘transparency’-paradigm in which technology is explained to its users. Instead, porosity opens up a system towards the outer world: it is open to comments, critique and contestation. It means that a system should be legible to the people who live alongside it and open to questioning. A porous system is the opposite of the black box that does its work, but explains nothing, and does not offers any opening for dialogue.

Experiments

Throughout our research project we have carried out numerous experiments that could make the definition, making and evaluation of smart city technologies more porous. The handbook gives an extensive overview of these. Let me give you three examples here, to give you an idea of what we have been working on.

A model (maquette) for digital systems

Urban planners have long had ways to make proposed changes tangible to residents: architectural models, artist impressions, mock-ups of what a new square or building might look like. But what is the equivalent for a scanning bicycle equipped with computer vision? Digital interventions in public space are largely invisible. They don't have facades or footprints. We developed a method that makes the data journey of such a system visible: step by step, from the moment the camera captures an image to the moment an enforcement officer acts on it. We brought this visualisation into public spaces and into workshops, asking residents to respond at each step. This resulted in rich conversations. Residents who had never thought about data minimisation suddenly had concrete suggestions: "Do you really need to send a photo? Wouldn't GPS coordinates be enough?"

Participatory procurement. Procurement is the moment when a city formalises what it wants to buy, and in many ways it is the most consequential moment in the technology lifecycle. In this phase a list of requirements is written up, and market parties can bid with proposals to realize them.  Once a contract is signed, the room to manoeuvre narrows dramatically. Yet it is also the moment when residents are almost never involved.

With Waag Futurelab we researched the basic procurement cycle, and explored moment when citizens could be invited in. One option is to organize a participatory process at the beginning of the project, before all requirements are drafted up, and including citizens in this process. Another is to make participatory design part of the requirements. So rather than drafting out all the requirements in advance, the vendor has to design a process in which the requirements are co-designed with citizens and government officials after the tender is awarded.

Participatory machine learning. Ai systems are calibrated though machine learning, and in that process a threshold is set: a confidence score above which the system treats something as a detected violation and below which it doesn't. This sounds like a purely technical decision, but it isn’t. That threshold directly determines how many false positives get generated, how much extra work falls on enforcement officers, and whether certain neighbourhoods end up being over-policed relative to others. Rather than leave these decision to the technical development team, we sought to invite the people closest to the system's daily reality in that conversation. Here we built on Kars Alfrink work on contestability and participatory machine learning.

Our team worked with design studio CLEVER°FRANKE to build an interactive data visualisation tool that made the consequences of various thresholds visible at multiple scales simultaneously: from individual detections to neighbourhood-level distribution patterns. We then ran focus groups with enforcement officers, inviting them to make the calibration decision together, using the tool. Their arguments ranged across workload, fairness between districts, legal liability, and the city's reputation. The disagreements they had were not technical disagreements, but political and ethical ones.

Learning beyond the individual project

We are very happy with our handbook, and think it serves a valuable contribution to the debate on the ethical deployment of urban technologies. But of course we don’t claim to have the final answer. Four years of fieldwork has also left us with plenty more questions.

We learned that participation is harder to sustain than to initiate. Getting residents engaged at the start of a project is manageable. Maintaining that engagement across multiple phases, through the slow grind of procurement and technical development, is difficult. And when a project ends and the team disperses commitments made to participants can quietly evaporate. The institutional infrastructure to carry those commitments forward simply doesn't exist yet. 

Similary, an important lesson was also that participation can come uninvited. Rather than ignoring protests and voices of dissent, cities should learn to make room for them.

We also learned that the most important conversations are often the ones that fall outside the project's official scope. Residents who engaged with one of our experiments regarding the design of a scanning bicycle didn't just have views about camera placement or data retention periods. They had views about whether automated enforcement in public space was desirable at all. Those questions were, strictly speaking, outside what they were asked to participate in. But ignoring these issues also means mistaking participation for legitimation. Perhaps then, there should be another learning-loop: one that goes between projects. How can the insights gathered in one project be included in the defining and design cycles of other projects?

Bewoners-in-the-loop is co-authored with Mike de Kreek and Tessa Steenkamp as part of the Human Values for Smarter Cities research project.