Skip to main content

Ethics of Data-Intensive Applications

Adapted from: Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), Chapter 12 — rewritten and condensed for teaching.

Every system in this series has been evaluated on technical grounds — correctness, scalability, fault tolerance. Correctness and Integrity in Dataflow Systems closed out the technical side of this series' coverage of Designing Data-Intensive Applications. But every system is also built for something, and every system has consequences — intended and otherwise — that reach well past its original purpose. Much of the data these systems store and process is fundamentally about people: their behavior, their relationships, their identities. That makes the engineering choices behind these systems inseparable from a set of genuinely ethical questions — ones software engineers don't always get much practice thinking through, but that carry real weight all the same. This lesson is a deliberate departure from the rest of this series: no algorithms, no code exercises — just the ethical dimension of the systems this series has spent so long building.

Predictive Analytics​

Using data to forecast the weather or the spread of a disease is one thing. Using it to predict whether a convict will reoffend, whether a loan applicant will default, or whether an insurance customer will file expensive claims is something else entirely — these predictions land directly, and often severely, on individual people's lives. Organizations have obvious incentives to be cautious: a missed business opportunity is cheap, a bad loan or a risky hire is expensive, so when in doubt, algorithmic systems tend to default to no.

The trouble is that "no" compounds. Someone flagged as risky by one system tends to get flagged similarly by others — passed over for a job, an apartment, a loan, a flight — a pattern serious enough that it's been described as an "algorithmic prison." Unlike a criminal justice system that (at least formally) presumes innocence until proven guilty, an automated system can exclude someone from large swaths of ordinary life with no proof of wrongdoing at all, and often with no real avenue to appeal.

Bias and Discrimination​

Algorithmic decisions aren't inherently more biased than human ones — humans have their own biases, and some hope that data-driven decisions might actually reduce discrimination compared to instinctive human judgment. But a predictive model doesn't invent its patterns from nothing; it learns them from historical data. If that history reflects systematic bias, the model will typically learn — and often amplify — that same bias in its output, frequently without anyone being able to clearly explain why.

This runs headlong into anti-discrimination law, which generally forbids treating people differently based on protected traits like race, age, gender, or disability. But a model doesn't need to be handed a "race" field to end up encoding race — in a segregated city, a postal code alone can be a strong proxy for it. It's not enough to believe that feeding biased historical data into an algorithm will somehow produce fair, impartial output — a pattern one critic memorably summarized as "machine learning is like money laundering for bias." Predictive systems fundamentally extrapolate the past forward; if the past was unjust, a purely data-driven system will tend to keep it that way. Building a genuinely better future takes moral judgment that data alone cannot supply.

Responsibility and Accountability​

Automated decisions raise a hard question that manual ones mostly avoid: when a human makes a bad call, they can — at least in principle — be held to account, and the person affected can appeal. When an algorithm gets it wrong, who answers for that? If a self-driving car causes a crash, or a credit-scoring model systematically disadvantages one group, is there any real recourse?

Traditional credit scores are already built on an uncomfortable premise, but at least they're grounded in your own documented financial history, with at least some path to dispute an error. Machine-learning-based scoring often works differently — closer to "people statistically similar to you behaved this way," which is really a formalized kind of stereotyping, frequently built from proxies like location that correlate with race and class. And even a statistically accurate model is a population-level statement: an average life expectancy of 80 doesn't mean any particular person keels over on their 80th birthday, and a prediction can be perfectly well-calibrated on average while still being flatly wrong about the one person it's currently judging.

Feedback Loops​

Some of the most damaging failures come from systems that don't just make static predictions but actively shape the world they then measure. Recommendation systems that get very good at predicting what someone already agrees with can quietly narrow their information diet into an echo chamber, reinforcing existing views rather than exposing anyone to new ones.

An even sharper case: an employer that screens candidates by credit score. A genuinely good worker who hits an unexpected financial setback misses a few bill payments; their credit score drops; a lower score makes them measurably less likely to be hired; unemployment deepens the very financial hardship that hurt their score in the first place. The loop is entirely self-reinforcing, and it's easy for it to hide behind the appearance of neutral, rigorous, "data-driven" analysis. Systems thinking — deliberately reasoning about the entire system, including the humans who interact with it and adapt their behavior in response, not just its computational core — is one of the few tools capable of surfacing this kind of consequence before it does real damage.

Check: predictive analytics​

Quiz

Bias, accountability, and feedback loops in predictive systems

1 / 2

Why can a predictive model produce racially or otherwise discriminatory output even if "race" (or another protected trait) is never explicitly included as an input field?

Pick an answer to see if you are right.

Privacy and Tracking​

There's a meaningful difference between a system that stores what a user explicitly chose to give it, and one that quietly logs data as a side effect of the user doing something else entirely. In the first case, the system is straightforwardly serving the user. In the second, it starts to develop interests of its own — interests that can diverge sharply from the user's.

Some behavioral tracking genuinely benefits users directly: which search results get clicked helps improve ranking; "people who liked X also liked Y" helps people find things they'd actually want. But once a service's revenue comes from advertisers rather than users, the incentives shift — the advertiser becomes the real customer, and tracking tends to grow far more detailed, far more retained, and far more oriented toward building a durable profile of each person than the product experience alone would ever require.

Surveillance​

Kleppmann poses a pointed thought experiment: take any glowing description of a modern data platform and swap the word "data" for "surveillance." "Our surveillance-driven organization collects real-time surveillance streams, stored in our surveillance warehouse, analyzed by our surveillance scientists." It reads uncomfortably close to accurate. We've built an always-on microphone and location tracker into daily life — smartphones, smart speakers, even children's toys — at a scale that would have been the envy of any historical surveillance state, except that today it's mostly corporations doing the collecting rather than governments.

Not all data collection is meaningfully surveillance, but treating it as a live possibility is a useful lens for examining why people tolerate it. Maybe the stated purpose feels benign — "just" better recommendations, "just" more relevant ads. But combined with the predictive-analytics risks already covered in this lesson, "benign" starts looking less certain: car insurance premiums are already being tied to driving-behavior trackers, health coverage to fitness-tracker data, and movement sensors in a smartwatch have been shown capable of inferring what someone is typing — including passwords.

The standard defense is that users consented — they agreed to a terms-of-service page, in exchange for a genuinely valuable free service. This argument has real holes. Privacy policies are written to be skimmed past, not understood, and data about one person routinely reveals things about people who never agreed to anything at all — friends, family, anyone incidentally caught in the data. There's also no real negotiation happening: the terms are set entirely by the service, take-it-or-leave-it. And "leave it" is often not a genuinely free choice — when a service becomes close to essential for ordinary social or professional participation, declining to use it carries a real social cost, disproportionately borne by people who can least afford to opt out.

It's also worth being precise about what "privacy" even means here. Privacy is not secrecy — plenty of people happily share deeply personal things publicly. Privacy is the freedom to choose, case by case, what to reveal, to whom, and what to keep to yourself. Surveillance infrastructure doesn't erase that right so much as transfer it — from the individual to the company doing the collecting, which now gets to decide (often for its own commercial benefit) what gets revealed and to whom, largely outside the individual's control or even awareness.

Data as Assets and Power​

Behavioral data is sometimes dismissed as mere "data exhaust" — a harmless byproduct being recycled for insight. It's more accurate to flip that framing entirely: if advertising is what actually funds a service, then the behavioral data is the real product, and the application itself is simply the mechanism for extracting it. The existence of an entire industry of data brokers — aggregating, inferring, and reselling personal information, largely outside public view — makes the underlying economic reality hard to dispute.

Because that data is valuable, it attracts attention well beyond the company that first collected it: governments seeking it through legal compulsion or outright breach, buyers acquiring it when a company is sold or goes bankrupt, and criminals or hostile actors simply stealing it. This has led some to describe accumulated personal data less as an asset and more as a "toxic asset" or hazardous material — something that has to be weighed not just for its present-day value, but for the risk of it falling into the wrong hands under a government or regime that doesn't yet exist. As the saying goes, knowledge is power — and the ability to closely observe others while remaining unobserved yourself is one of the most consequential forms of power there is.

Check: privacy and surveillance​

Quiz

Surveillance, consent, and data as an asset

1 / 2

What does the lesson identify as the core relationship shift that happens once a "free" service is funded primarily by advertisers rather than by users directly?

Pick an answer to see if you are right.

Remembering the Industrial Revolution​

Kleppmann draws a deliberate parallel between today's data economy and the Industrial Revolution. That earlier transformation brought real, sustained gains in living standards — but only after decades of serious, largely unmanaged harm: polluted air and water, dangerous working conditions, child labor. It took a long time, and a great deal of organized effort, before protections like environmental regulation, workplace safety rules, and bans on child labor were put in place — and once they were, almost no one wanted to go back to a world without them.

Security researcher Bruce Schneier frames the parallel sharply: "Data is the pollution problem of the information age, and protecting privacy is the environmental challenge." Just as it can look bewildering in hindsight that earlier generations tolerated dumping industrial waste straight into rivers, a future generation may look back and ask how we could have collected and hoarded personal data as carelessly, and for as long, as we did.

Legislation and Self-Regulation​

Data protection law — the EU's early Data Protection Directive being one prominent example — has generally tried to insist that data collection be limited to specific, legitimate, disclosed purposes. This runs directly against the prevailing instinct in data-intensive engineering, which is to collect broadly, combine liberally, and explore for value after the fact — the opposite of "specified and explicit purpose," and hard to reconcile with any meaningful notion of informed consent given up front. Companies often push back on tighter regulation as an innovation tax — sometimes with real justification (locking away medical data too tightly can cost lives that better-shared data could have saved), which is precisely what makes the underlying trade-off genuinely difficult rather than a straightforward call.

Kleppmann's own conclusion is less about any single law and more about a needed shift in engineering culture: treating the people behind a dataset as people — owed dignity and genuine agency — rather than as metrics to be optimized. Concretely, that means not retaining data longer than it's actually needed, being honest with users about how their data is actually used, and exploring technical mechanisms — like access control enforced cryptographically rather than merely by written policy — that make privacy protections harder to quietly erode. None of this is a solved problem. But treating an individual's control over their own data as something actively worth protecting, rather than assuming it will simply take care of itself, is the starting point.

Check: the industrial revolution parallel​

Quiz

The Industrial Revolution parallel and data regulation

1 / 2

What is the point of Bruce Schneier's line, "Data is the pollution problem of the information age, and protecting privacy is the environmental challenge"?

Pick an answer to see if you are right.

Key takeaways
  • Predictive systems don't invent bias from nothing — they extrapolate from historical data, and a biased history tends to produce a biased, and often opaque, prediction, even when protected traits are never used directly as inputs.
  • Automated decisions raise real questions of accountability that manual decisions don't escape either, but that are easier to dodge behind an algorithm's apparent objectivity — someone flagged as risky can face compounding, hard-to-appeal exclusion across many unrelated parts of life.
  • Self-reinforcing feedback loops (like credit scores affecting employability, which affects financial stability, which affects credit scores) can turn a single piece of bad luck into a lasting spiral — systems thinking, reasoning about the whole system including the humans in it, is one of the few ways to catch this in advance.
  • Much routine data collection functions as surveillance in substance if not in name — and the usual "you consented" defense is weaker than it looks once you account for unreadable privacy policies, data that implicates non-users, and the real social cost of opting out of a near-essential service.
  • Privacy means the freedom to choose what to reveal and to whom — not secrecy — and large-scale data collection tends to transfer that decision right from the individual to the company doing the collecting.
  • Personal data is a genuine economic asset — which is exactly why it attracts governments, buyers, and criminals alike, and why some now describe it as a toxic asset whose risk has to be weighed against its value.
  • As with the Industrial Revolution's slow, hard-won path toward safety and environmental regulation, building real safeguards around data collection is an unfinished cultural and regulatory project — one Kleppmann argues engineers bear real responsibility for helping get right.

Further Reading​