NightShift
An AI on-call engineer for AWS that investigates incidents, proposes a fix, and acts only with my approval.
- Root causes found
- 4 → 18
- of 36 incidents: the same model given only the alarm vs. given its tools
- Live bad deploy
- 138 s
- to the right diagnosis; the approved rollback restored service
I built a small serverless store on AWS, a chaos framework that stages 12 kinds of incident, and an LLM agent that gets paged, investigates with read-only tools and names the root cause. A separate component runs its proposed fix only after I approve that exact action, then checks the system recovered.
Impact
Across 36 staged incidents, giving the model tools took it from 4 to 18 correct root causes, a difference that chance alone would almost never produce (p = 0.0001). A hand-written runbook scored 21, statistically tied with the agent. They won on different incidents, and the agent's most common misses traced back to how I had defined the fault categories.
- Python
- TypeScript
- Next.js
- AWS Lambda
- Aurora DSQL
- Terraform
- GitHub Actions
- LLM APIs
- Safe by design: the agent can only read. A separate Actor with its own narrow AWS role runs one of five reversible actions, only after a single-use approval that expires in 15 minutes.
- The store and agent (Lambda, Aurora DSQL, DynamoDB, SQS) are defined in Terraform and deployed by GitHub Actions through OIDC with no stored keys, within AWS Always Free limits.
- Found a transaction leak with no visible symptom in a billing metric, proved Aurora DSQL bills for every second a transaction stays open, and fixed it by changing the connection default.
- All 180 graded investigations are replayed step by step on a public dashboard, including the 11 unsafe proposals the agents made (none executed).
