Abstract
Now the systems are so distributed that they're faster than a human operator can think. In a modern, platform, a single customer request can involve forty services, three data stores, two cloud regions, a message bus, a feature-flag service and an identity provider. All of those components log, metric and trace logs and events. If anything is amiss, the signs are all around, but the explanation is missing.More dashboards, more alert and more on call engineers were the industry's answer to this for 20 years. This way is no longer sustainable. The number of telemetry volumes will increase at a quicker pace than the number of employees. Systems undergo changes numerous times each day. Attackers automate. Failure modes are no longer an isolated, broken part, but interactions, that emerge, between seemingly good parts.This book is about the next answer: Distributed systems with intelligence of their own. Systems that perceive their own degradation, explain it, come to their own defense, and repair themselves within carefully drawn boundaries. Isn't a book on the removal of an engineer with a “model”. It's a book on how to design software, data, machine learning and human judgment together to make the system as a whole more reliable than each piece could possibly be alone.There are several ideas sprinkled throughout from site reliability engineering, control theory, statistics, security operations, causal inference, and most recently, large language model agents. Every industry has a jargon and there are gaps in every field. The aim is to provide you with a single way of thought covering all of them – instead of relying on vendor promises, engineering practice.