Software engineering research built a set of instruments for knowing when code is ready for production: dependency matrices, slices, mutation scores, oracles, queueing laws, failure taxonomies. Most were too expensive to run by hand and fell out of everyday use. Language models make the expensive part cheap. This review collects the instruments, shows each one working, and notes what changes when a model does the tedious translation.
Production-ready software has all four. Language models write code that does well on the first for the happy path and poorly on the other three, because those properties live between files rather than inside them.
Each entry says what the instrument catches, why it fell out of use, and what a model changes. The figures are small simulations written for this page; Replay restarts one. Numbers marked Housewright come from runs of a 796-line procedural house generator with a voxel round-trip verifier, used here as a running specimen.
CChangeable
1Structure and reach
Every edit has a reach: the code that might need to change with it, or might break because of it. Cyclomatic complexity McCabe 1976 measures one function at a time. The instruments in this section measure the relationships between functions, which is where reach lives.
1.1Design structure matrices and propagation cost
Catcheshidden cycles, and modules that a change anywhere can reach
A design structure matrix lists modules on both axes and marks a cell when the row depends on the column. Baldwin and Clark used it to describe modularity as an economic property: a well-modularised design lets parts change independently, and the matrix shows where that holds Baldwin & Clark 2000. MacCormack, Rusnak and Baldwin applied it to source code and defined propagation cost, the share of filled cells after taking the transitive closure MacCormack et al. 2006. It answers a plain question. If one module changes, what fraction of the system could be affected, on average? In their data, a deliberate redesign of Mozilla cut its propagation cost from 17.35% to 2.78%.
Sorting the matrix by strongly connected components exposes the core, a cluster in which every member can reach every other Baldwin et al. 2014. The figure shows the pattern found in Housewright. Every pipeline stage reads and writes one shared state object, so the closure fills a block across the middle of the system, and the static view loses the order in which the stages actually run. Switch to typed stage outputs and the same pipeline becomes a triangle.
Stages share
Fig. 1Direct dependencies appear first, then the transitive closure. The outlined block is the core. With typed outputs, a shared types module becomes a stable interface that everything depends on and that depends on nothing.
ThenMatrix extraction per language, clustering, and an analyst to interpret the core.
NowA parser builds the matrix. A model names the decision each core module hides and proposes where to cut.
In Housewright
31 units and 43 direct dependencies; propagation cost about 20%. One seven-node cycle runs through the shared state object, which carries 40 fields. 22 of those fields are written by the generator and read only by the interface, so they form an interface with no contract. These are rough figures from a regex-level scan, not a full parse.
1.2Information hiding and connascence
Catchesdependencies that a compiler cannot see
Parnas argued that a system should be split by the design decisions most likely to change, each module hiding one of them, rather than by the steps of processing Parnas 1972. Stevens, Myers and Constantine named coupling and cohesion Stevens et al. 1974. Page-Jones gave a vocabulary for how strong a dependency is Page-Jones 1992. Two pieces of code are connascent when changing one requires changing the other. The weak kinds, name and type, are caught by compilers. The strong kinds, such as meaning, algorithm, timing and value, are found by tests, by load, or by users.
The working rule is to keep strong connascence inside a module and let only weak kinds cross a boundary. It is easy to state and hard to measure, because spotting connascence of meaning or algorithm requires knowing what the code is for. Hyrum's law adds the other side: with enough users, every observable behaviour becomes something somebody depends on Wright, Hyrum's Law.
Fig. 2An edit on the left must be matched on the right. Stronger kinds are found later, by more expensive means. Examples marked ◆ are from Housewright.
ThenExpert review. No tool could label connascence of meaning or algorithm.
NowA model can classify each cross-module dependency by kind and flag strong kinds that cross a boundary.
1.3Program slicing and change impact analysis
Catchesexactly which lines an edit can affect, and which tests reach them
Weiser defined a slice as the part of a program that can affect the value of a variable at a given point Weiser 1984. A backward slice answers "what could have produced this value?" A forward slice answers "what could this line change?", which is the core of change impact analysis Bohner & Arnold 1996. Chianti applied the idea to Java edits, reporting which tests a change affects and which changes affect a failing test Ren et al. 2004. Safe regression test selection uses the same reasoning to rerun only the tests whose execution touches changed code Rothermel & Harrold 1997Gligoric et al. 2015.
For a model making an edit, the forward slice is the context it should read and the set of tests it should run. Agents today mostly find context with text search, which misses dependencies that pass through shared state and returns many irrelevant matches.
Fig. 3The backward slice of label at line 9 follows data and control dependencies upward. Then the forward slice of an edit to line 2 shows everything that edit can reach.
ThenA precise slicer per language, slow on large programs and hard to read.
NowApproximate slices are cheap. A model reads the slice instead of the repository, and the slice bounds what must be retested.
1.4Scattering and tangling
Catchesfeatures that touch every stage
Aspect-oriented research named two symptoms. A concern is scattered when its code is spread over many modules, and a module is tangled when it mixes many concerns. Eaddy and colleagues found moderate to strong correlations between how scattered a concern was and how many defects it had Eaddy et al. 2008. Scattering also predicts the cost of the next feature: when a concept is spread across seven units, the next change to it is a seven-unit change.
Fig. 4Which units of Housewright's generator mention each concept. The loft reaches 9 units and the ladder 7; the "light" need is split across cost, build and verify with nothing tying the three together.
ThenConcern mapping by hand, one concern at a time.
NowA model can map named concepts to code locations and track their spread from commit to commit.
CRChangeable · Recoverable
2History as evidence
Static structure shows what could change together. Version history shows what does. These measures need no understanding of the code at all, and in several studies they predict defects better than code metrics do.
2.1Change coupling and hotspots
Catchescoupling that no import statement shows
Files that keep changing in the same commits are logically coupled, whether or not either one references the other Gall et al. 1998. Zimmermann and colleagues mined these patterns to suggest "programmers who changed these files also changed…" at edit time Zimmermann et al. 2005. Tornhill's hotspot analysis ranks files by churn weighted by complexity, which points review at the few files where most defects land Tornhill 2015.
Fig. 5Each commit lights the files it touches; node size is churn. Dashed lines are imports. Solid links thicken as pairs change together, and the heaviest one joins two files with no import between them.
ThenMining scripts, and a repository old enough to have patterns.
NowCheap to compute. The output is a list of files an agent must open before it edits a given one.
2.2Defect prediction from process
Catchesrisky commits before they merge
Moser, Pedrycz and Succi compared change metrics (revisions, authors, churn, age) with static code attributes and found that on Eclipse data the change metrics predicted defects better Moser et al. 2008. Nagappan, Murphy and Basili found that organisational measures, such as how many engineers and teams touched a binary, predicted failure-proneness in Windows Vista better than code metrics Nagappan et al. 2008. Just-in-time prediction scores each commit as it lands Kamei et al. 2013.
The training labels come from SZZ. Start at a commit that fixed a bug, blame the lines it changed, and mark the commits that last touched those lines as fix-inducing Śliwerski et al. 2005. The labels are noisy, because not every blamed line caused the bug.
Fig. 6SZZ traces a fix back to the commits that introduced the lines it repaired. Features of those commits become training data for a per-commit risk model.
ThenLinking issues to commits, noisy labels, models retrained per project.
NowA model can read the fix and the blamed change and say whether the link is real, which cleans the labels.
KKnown
3Test strength
Coverage says which lines the tests ran. It says nothing about whether the tests would notice if those lines were wrong.
3.1Mutation testing
Catchestests that run code and assert nothing
Mutation testing plants small faults in the program, such as changing > to >= or a constant to its neighbour, and reruns the tests. A mutant is killed if some test fails and survives otherwise. The founding paper is by DeMillo, Lipton and Sayward, who argued that programmers write nearly correct code and that tests which catch simple faults also catch most complex faults built from them DeMillo et al. 1978. The field is surveyed in Jia & Harman 2011 and Papadakis et al. 2019.
Inozemtseva and Holmes found that once suite size is controlled for, coverage is only weakly to moderately correlated with a suite's ability to find faults Inozemtseva & Holmes 2014. Just and colleagues found that mutant detection correlates with real fault detection Just et al. 2014.
Fig. 7Both suites reach 100% line coverage. Three mutants are applied one at a time. Only the suite with real assertions notices.
The cost was always compute and noise: thousands of mutants, some equivalent to the original program. Google made it practical by mutating only changed lines and showing surviving mutants as code review comments, with heuristics that suppress unproductive ones Petrović & Ivanković 2018. Meta has described using surviving mutants as prompts for language models to write tests that kill them Foster et al. 2025, following earlier work on improving tests with models Alshahwan et al. 2024.
ThenCompute for every mutant, and people to judge which survivors are equivalent.
NowMutating only the diff is cheap. A model writes the killing test, and a small model filters trivial mutants.
3.2A scan of every constant
Catchesnumbers nobody has made a decision about
Applied to Housewright, a literal scan nudged every number in the generator up and down and rebuilt six varied houses per variant: 1,755 mutants in about six minutes on two cores. The useful result is the sorting.
Fig. 8Every mutant from the Housewright scan, sorted by outcome. Real counts.
Pinned (172) broke a check: chimneys, seals, doorways.
Crashed (64) were caught by the runtime for free.
Unpinned (551) changed the house, such as its footprint, windows or dormers, and no check had an opinion.
Inert (968) changed nothing observable, for three different reasons. Some landed in a channel no check reads (render scales, animation order). Some sat in code the six houses never reached (a width floor for U-shaped plans). Some are robust by design.
The door's second block came up inert. Nothing checked that a door is tall enough to walk through.
KKnown
4Oracles
An oracle decides whether an output is correct. Mutation testing measures how sharp an oracle is. It cannot supply one.
4.1The oracle problem
Catchesthe gap between running code and knowing it is right
Barr and colleagues surveyed the field and sorted oracles into specified (contracts, models, assertions), derived (from other versions, other implementations, or relations between runs) and implicit (crashes, hangs, leaks), with humans as the fallback when none of these exist Barr et al. 2015. Each kind catches a different slice of faults, and each is partial. Jahangirova and colleagues used mutation to find faults that an oracle misses and then improved the oracle itself Jahangirova et al. 2016.
Fig. 9Outputs flow right through four kinds of oracle. Each catches the faults it can see and drops them into its tray. A few faults escape all four.
ThenWriting oracles was the expensive part of testing.
NowA model can propose oracles of every kind. It should not be the only judge, since it shares blind spots with the model that wrote the code.
4.2Metamorphic and property-based testing
Catcheswrong answers when the right answer cannot be computed
When the correct output is unknown, a relation between outputs may still be known. Rotate the input and a count should not change; add a filter and the results should be a subset. Chen, Cheung and Yiu called these metamorphic relations Chen et al. 1998Segura et al. 2016. Property-based testing generates many inputs and checks properties instead of examples Claessen & Hughes 2000. Hughes's guide lists the property styles that work in practice: invariants, postconditions, metamorphic properties, inductive properties and model-based properties Hughes 2020.
Two neighbours belong here. Differential testing runs several implementations on the same input and flags disagreement McKeeman 1998. Daikon watches executions and reports likely invariants, which a person can promote to checks Ernst et al. 2001.
Fig. 10The program counts points inside the circle. Rotating the input must not change the count. The buggy version tests a square instead, and the relation catches it without anyone knowing the true count.
ThenInventing good relations took domain insight.
NowA model proposes candidate relations from the domain, the test harness checks them, and a person approves the ones that should hold.
4.3Round-trip oracles
Catchesdrift between a plan and what was built
If an output can be read back into the input's terms, the reading is an oracle. Housewright plans rooms, builds voxels, then ignores the plan, flood-fills the voxels to recover rooms and doors, and compares the recovered graph with the brief. A missing door shows up as a missing edge. Contracts in the style of design by contract do the same job at function boundaries Meyer 1992.
Fig. 11Plan, build, read back, compare. On the second pass the builder drops a door, and the flood fill cannot reach the store.
4.4Why specifications fall short
Naur argued that a program is a theory held by the people who build it, and that code and documentation are partial traces of that theory Naur 1985. A written specification is a snapshot of the theory taken before most of it existed. It is silent about most cases, and silence cannot distinguish "any value is fine" from "nobody thought about it". It rarely says what to give up when two goals conflict. It cannot list everything that must not change. Users recognise correct behaviour more readily than they can describe it. And a specification drifts from the code, because nothing forces the two together.
Mining helps where specifications are silent. Ammons and colleagues mined protocol rules from execution traces Ammons et al. 2002. Mined rules are hypotheses, though. A person still has to say yes before one becomes part of the specification.
Doors in Vintage Story make the point concrete. The minimum door is one block wide and two high, but wider doors and double doors exist in every material. The right specification is not a number. It is "an agent of size A can pass through", plus a set of door variants, each with its legal sizes. A magic number in code usually marks a concept missing from the model.
4.5A loop that generates questions
Catchesunmade decisions, in the order that matters
The pieces combine into a loop. A deterministic scan perturbs the code and records which observable behaviour moved. A small model triages each site: already pinned, arbitrary by design (a hash constant), not exercised (needs new inputs), not observed (needs a new oracle), a tolerance (needs a range), or an unstated rule (needs an invariant). Related sites cluster into one concept. Each cluster becomes one question in domain language, with the behaviour diff as evidence and the check its answer would create. Answers go into a ledger with who decided, when and why, including explicit "don't care" answers, and the scan reruns.
Fig. 12Sites flow through triage. Only tolerances and unstated rules reach a person, one question per concept. The starting share (10% pinned) is Housewright's; the rise after it is illustrative.
Two numbers track progress: the share of mutants that are pinned or explicitly marked don't-care, and the number of open questions. The ledger has the shape of an assurance case, with claims, arguments and evidence, as in the Goal Structuring Notation Kelly & Weaver 2004. Oracle improvement through mutation Jahangirova et al. 2016 and specification mining Ammons et al. 2002 are the closest prior work. Using a small model for triage and phrasing, and recording "don't care" as a decision, are the new parts.
BBounded
5Bounded under load
A bounded system can say in advance how it will degrade, and that way was chosen. For pipelines and event-driven systems, much of this can be calculated from measurements instead of discovered by load testing.
5.1Operational laws
Catchesthe arrival rate at which a pipeline stops keeping up
Denning and Buzen showed that the core laws of queueing networks hold for measured quantities, without assuming any probability distribution Denning & Buzen 1978. The utilisation of a station is arrivals × visits × service time ÷ workers. The system's maximum throughput is set by the station where that product is largest. Little's law ties the number in the system to arrival rate × time in system Little 1961. After a burst of B items, the backlog drains in B ÷ (capacity − arrival rate).
An event-sourced feature file is already a queueing network. Each handler, projection and policy is a station, fan-outs set the visit ratios, and per-element telemetry supplies the service times. The bottleneck and the drain time follow directly.
Fig. 13One worker with service rate μ = 1 and random arrivals. Below ρ = 1 the queue fluctuates around ρ/(1−ρ). At ρ ≥ 1 the backlog grows without bound.
5.2Variability and the tail
Catchesslow requests that averages hide
Averages are not enough. Kingman's approximation, after Kingman 1961, says time spent waiting in the queue grows with ρ/(1−ρ) multiplied by the average squared variability of arrivals and service. A step whose cost grows as n³ turns uneven input sizes into extreme service-time variability. Fan-out makes it worse: a request that waits for the slowest of N parallel tasks inherits the tail of all of them. Dean and Barroso's example is a server that is slow one time in a hundred; a request that touches 100 such servers is slow 63% of the time Dean & Barroso 2013. With exponential task times, the expected slowest of N tasks is the harmonic number HN times the mean, about 7.5 times for N = 1,000.
Tasks per request
Fig. 14Each request waits for the slowest of N tasks with mean time 1. The histogram collects request times; the dashed line is HN.
5.3Throughput collapse
Catchesadding workers that makes the system slower
Gunther's Universal Scalability Law models throughput with N workers using a contention term σ, for queueing on a shared resource, and a coherency term κ, for the cost of keeping workers consistent with each other Gunther 2007. With any κ above zero, throughput peaks at N* = √((1−σ)/κ) and then falls. Two measurements at different N are enough to fit both terms.
Fig. 15Relative throughput against workers for three settings. Contention flattens the curve; coherency bends it back down.
5.4Metastable failures
Catchesoutages that outlive their trigger
Bronson and colleagues described metastable failures: a trigger pushes a system into a bad state that persists after the trigger is removed, held there by a sustaining feedback loop such as retries Bronson et al. 2021. Huang and colleagues found the pattern across many production incidents Huang et al. 2022. The fix is to bound the feedback with retry budgets, jittered backoff, load shedding, and deadlines that drop work nobody is waiting for.
Retries
Fig. 16A five-unit spike in offered load against capacity 1. Timed-out requests are retried. Without a budget the retries keep the system overloaded after the spike ends; with one, the queue drains.
5.5Degradation modes and their levers
Each way of degrading has a known lever. Choosing among them in advance is what makes behaviour bounded.
Mode
Symptom
Lever
Latency growth
Waits rise steeply as ρ nears 1
Keep each station's ρ below about 0.7 at target load; add workers at the bottleneck
Unbounded backlog
Queue grows linearly with time
Bounded queues with an explicit overflow policy: reject, shed or defer
In an event-sourced system, projection rebuild time is events ÷ replay throughput. It is a recoverability bound that grows with the log until a snapshot resets it.
ThenCapacity models built by specialists, for one deployment, and rarely updated.
NowThe inputs, a dependency graph and per-element service times, already exist in a feature file and its telemetry, so bounds can be recomputed on every change.
RBRecoverable · Bounded
6How software fails
Failure taxonomies turn incident history into checklists. They work best as prompts: for each unit an edit touches, ask what happens on error, retry, overload, bad data and rollback.
6.1Bohrbugs, Heisenbugs and defect classes
Catcheswhich failures will reproduce, and which process step let them through
Gray's study of Tandem systems popularised the split between faults that reproduce reliably, Bohrbugs, and faults that vanish when you look for them, Heisenbugs. In the error logs he examined, nearly all software faults were of the second kind, which is why restarts and process pairs worked so well Gray 1985. IBM's Orthogonal Defect Classification labels each defect by type and by the activity that should have caught it, so that the distribution of defects points at the weak step in the process Chillarege et al. 1992. Gunawi and colleagues classified over 3,000 issues in cloud systems Gunawi et al. 2014.
Fig. 17The same input, run twelve times. The Bohrbug fails every run. The Heisenbug fails only when thread B's write lands between thread A's read and write.
ThenTrained people classifying every defect by hand.
NowA model can classify every bug-fix commit, so the process signal becomes continuous.
6.2Error handling is where systems break
Catcheshandlers that swallow detected errors
Yuan and colleagues studied production failures in distributed data-intensive systems. In 92% of the catastrophic failures they examined, the cause was incorrect handling of a non-fatal error the software had already signalled. In 58%, the fault could have been found by simple testing of the error-handling code Yuan et al. 2014. Handlers that only log, or are empty, were a common culprit.
Fig. 18The same detected error, two handlers. Logging and continuing carries bad state forward until something downstream fails. Aborting returns a clean error and the system stays healthy.
ThenError paths were tested last, if at all.
NowError handlers are easy to enumerate. A model can write a fault-injecting test for each one.
6.3Fault injection
Catchesfailures that only appear when specific parts fail together
Chaos engineering runs controlled experiments that inject failures into production-like systems and watch a steady-state metric Basiri et al. 2016. Lineage-driven fault injection is more targeted. It records which messages supported a successful outcome, injects faults that remove that support, and repeats until it finds a failure or shows the outcome survives every fault combination within the budget Alvaro et al. 2015. FoundationDB runs the whole database inside a deterministic simulator with injected faults, so any failure it finds replays exactly Zhou et al. 2021.
Fig. 19Each run highlights the messages that made the write succeed; the next run removes them. On the third run the coordinator still acknowledges a write that no replica stored.
6.4Hazard analysis
Failure mode and effects analysis lists, for each component, how it can fail, what each failure does, and how it would be detected IEC 60812. STPA treats safety as a control problem and asks how each control action could be unsafe: not given when needed, given when it should not be, given too early or too late, or stopped too soon Leveson 2011. Both fell out of use in ordinary software because an expert had to draft them component by component. A model can draft them for review, and reviewing a draft costs far less than writing one.
KKnown
7Specifying event-sourced systems
A specification is an ontology of data, a language of changes to that data, and rules for how those changes may happen over time. An event-sourced framework such as Dizzy already supplies the first two as models and events. The third is a behavioural contract for each kind of element.
Element
What its specification says
Natural form
Model
What is always true of the data
Invariants, checked on every projection
Projection
That it is a pure fold over events
Determinism; replay equals rebuild
Procedure
When it may run, and what it may change
Pre- and postconditions; the set of event types it may emit
Policy
How the system reacts over time
Temporal properties: after X, eventually Y; never Y after Z
Query
How fresh and how consistent
Staleness bounds; read-your-writes
Feature
Worked examples
Given these events, when this command, then these events
Two consequences follow. The frame problem becomes checkable: since all change goes through events, "what must not change" reduces to "which events may this procedure emit". And policies are where liveness lives, which is what TLA+ and Quint are built to model Lamport 2002Quint and what Amazon found worth the effort on real systems Newcombe et al. 2015. Event Modeling describes a system as Given/When/Then slices of exactly this kind Dymitruk 2019. Stateful property-based testing fires random command sequences at the system and checks invariants after each step MacIver et al. 2019.
The event log also generates questions. Process mining recovers the real flow of work from event logs van der Aalst 2016. A pattern that held in every trace, such as payment always preceding shipping, is a candidate rule. Ask once; a yes becomes a runtime monitor.
Fig. 20Five order histories. The miner notices that Paid always came before Shipped and asks whether that is a rule. Once confirmed, it becomes a monitor, which flags the next history that breaks it.
CChangeable
8Code meant to be deleted
Some code exists to tell a story or test an idea, and should be digested and deleted before the next approach is tried. The practice is old. Measuring it is not.
Brooks advised planning to throw one away, because you will anyway Brooks 1975. Extreme Programming calls a throwaway experiment a spike solution, which you expect to discard WellsCunningham. The Pragmatic Programmer separates prototypes, which are discarded, from tracer bullets, which are kept and grown Hunt & Thomas 1999. Bernhardsson's survival curves show code in real projects decaying with a measurable half-life, from about six years in Linux to months in fast-moving front-end frameworks Bernhardsson 2016. Four measures would make sketches manageable:
Lifecycle tag. Sketch, probe, tracer or production, each with its own gate. A sketch passes when it answers its question.
Leakage. A sketch is safe while nothing depends on it. Its fan-in should be zero, which a dependency matrix checks on every commit.
Digestion yield. What survives the deletion: a decision record, property tests, benchmark numbers, a sharper question.
Expiry. A delete-by date. Overdue sketches become review items.
Fig. 21Lines of code by the quarter they were written. Production cohorts decay slowly. The sketch cohort is deleted on purpose, and what it taught survives as small artifacts below the chart.
KBCRAll four
9Five gates for model-written changes
A review loop assembled from the instruments above. Each gate is cheap to run and checks something the diff alone cannot show.
Impact set. The slice and the change coupling decide what the agent reads and which tests must rerun.
Independent oracle. Properties, contracts or a round trip that do not share the generator's assumptions.
Test strength. The mutation score of new tests on changed lines.
Structural delta. The change in propagation cost, and any new strong connascence across a boundary.
Failure prompts. For each touched unit: error, retry, overload, bad data, rollback.
Fig. 22Changes pass through the five gates in order. Some are sent back with a specific reason; the rest merge.
Bacchelli and Bird found that developers expect code review to find defects, but that review mostly produces code improvements and better understanding of the code Bacchelli & Bird 2013. With generated code, the understanding is what goes missing. Review effort is better spent on the oracle, the impact set and the answers in the decision ledger than on reading the diff line by line.
References
Alshahwan et al. 2024Alshahwan, N., Chheda, J., Finogenova, A., Gokkaya, B., Harman, M., Harper, I., Marginean, A., et al. Automated Unit Test Improvement using Large Language Models at Meta. FSE 2024 Companion, 185–196. doi.org/10.1145/3663529.3663839
Alvaro et al. 2015Alvaro, P., Rosen, J., Hellerstein, J. M. Lineage-driven Fault Injection. SIGMOD 2015, 331–346. doi.org/10.1145/2723372.2723711
Ammons et al. 2002Ammons, G., Bodík, R., Larus, J. R. Mining Specifications. POPL 2002, 4–16. doi.org/10.1145/503272.503275
Bacchelli & Bird 2013Bacchelli, A., Bird, C. Expectations, Outcomes, and Challenges of Modern Code Review. ICSE 2013, 712–721. doi.org/10.1109/ICSE.2013.6606617Defects were the top stated motivation, yet only 14% of coded review comments concerned defects; code improvements were the largest group (29%).
Baldwin et al. 2014Baldwin, C., MacCormack, A., Rusnak, J. Hidden Structure: Using Network Methods to Map System Architecture. Research Policy 43(8), 1381–1397, 2014. doi.org/10.1016/j.respol.2014.05.00467% of the 1,286 releases studied had a core-periphery structure.
Barr et al. 2015Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., Yoo, S. The Oracle Problem in Software Testing: A Survey. IEEE TSE 41(5), 507–525, 2015. doi.org/10.1109/TSE.2014.2372785
Basiri et al. 2016Basiri, A., Behnam, N., de Rooij, R., Hochstein, L., Kosewski, L., Reynolds, J., Rosenthal, C. Chaos Engineering. IEEE Software 33(3), 35–41, 2016. doi.org/10.1109/MS.2016.60
Bernhardsson 2016Bernhardsson, E. The half-life of code & the ship of Theseus. Blog post, 5 December 2016. erikbern.com/2016/12/05/the-half-life-of-codeFitted half-lives range from 6.6 years (Linux) to 0.32 years (Angular).
Bronson et al. 2021Bronson, N., Aghayev, A., Charapko, A., Zhu, T. Metastable Failures in Distributed Systems. HotOS 2021, 221–227. doi.org/10.1145/3458336.3465286
Brooks 1975Brooks, F. P. Jr. The Mythical Man-Month: Essays on Software Engineering, ch. 11, “Plan to Throw One Away”. Addison-Wesley, 1975 (anniversary edition 1995).No free official copy; ISBN 0-201-83595-9 (1995 edition).
Chen et al. 1998Chen, T. Y., Cheung, S. C., Yiu, S. M. Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report HKUST-CS98-01, HKUST, 1998. Reposted as arXiv:2002.12543. arxiv.org/abs/2002.12543
Chillarege et al. 1992Chillarege, R., Bhandari, I. S., Chaar, J. K., Halliday, M. J., Moebus, D. S., Ray, B. K., Wong, M.-Y. Orthogonal Defect Classification: A Concept for In-Process Measurements. IEEE TSE 18(11), 943–956, 1992. doi.org/10.1109/32.177364
Claessen & Hughes 2000Claessen, K., Hughes, J. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. ICFP 2000, 268–279. doi.org/10.1145/357766.351266
Dean & Barroso 2013Dean, J., Barroso, L. A. The Tail at Scale. Communications of the ACM 56(2), 74–80, 2013. doi.org/10.1145/2408776.2408794
DeMillo et al. 1978DeMillo, R. A., Lipton, R. J., Sayward, F. G. Hints on Test Data Selection: Help for the Practicing Programmer. IEEE Computer 11(4), 34–41, 1978. doi.org/10.1109/C-M.1978.218136Introduced the competent programmer hypothesis and the coupling effect. The idea traces to a 1971 student paper by Lipton.
Denning & Buzen 1978Denning, P. J., Buzen, J. P. The Operational Analysis of Queueing Network Models. ACM Computing Surveys 10(3), 225–261, 1978. doi.org/10.1145/356733.356735
Eaddy et al. 2008Eaddy, M., Zimmermann, T., Sherwood, K. D., Garg, V., Murphy, G. C., Nagappan, N., Aho, A. V. Do Crosscutting Concerns Cause Defects? IEEE TSE 34(4), 497–515, 2008. doi.org/10.1109/TSE.2008.36
Ernst et al. 2001Ernst, M. D., Cockrell, J., Griswold, W. G., Notkin, D. Dynamically Discovering Likely Program Invariants to Support Program Evolution. IEEE TSE 27(2), 99–123, 2001. doi.org/10.1109/32.908957
Foster et al. 2025Foster, C., Gulati, A., Harman, M., et al. Mutation-Guided LLM-based Test Generation at Meta. FSE 2025 Companion, 180–191. arXiv:2501.12862. doi.org/10.1145/3696630.3728544
Gall et al. 1998Gall, H., Hajek, K., Jazayeri, M. Detection of Logical Coupling Based on Product Release History. ICSM 1998, 190–198. doi.org/10.1109/ICSM.1998.738508
Gligoric et al. 2015Gligoric, M., Eloussi, L., Marinov, D. Practical Regression Test Selection with Dynamic File Dependencies. ISSTA 2015, 211–222. doi.org/10.1145/2771783.2771784
Gunawi et al. 2014Gunawi, H. S., Hao, M., Leesatapornwongsa, T., Patana-anake, T., Do, T., Adityatama, J., Eliazar, K. J., Laksono, A., Lukman, J. F., Martin, V., Satria, A. D. What Bugs Live in the Cloud? A Study of 3000+ Issues in Cloud Systems. SoCC 2014, 1–14. doi.org/10.1145/2670979.2670986
Hughes 2020Hughes, J. How to Specify It! A Guide to Writing Properties of Pure Functions. Trends in Functional Programming (TFP 2019), LNCS 12053, 58–83. Springer, 2020. doi.org/10.1007/978-3-030-47147-7_4
IEC 60812IEC 60812:2018. Failure modes and effects analysis (FMEA and FMECA), edition 3.0. International Electrotechnical Commission. webstore.iec.ch/publication/26359
Inozemtseva & Holmes 2014Inozemtseva, L., Holmes, R. Coverage Is Not Strongly Correlated with Test Suite Effectiveness. ICSE 2014. doi.org/10.1145/2568225.2568271
Jahangirova et al. 2016Jahangirova, G., Clark, D., Harman, M., Tonella, P. Test Oracle Assessment and Improvement. ISSTA 2016, 247–258. doi.org/10.1145/2931037.2931062
Jia & Harman 2011Jia, Y., Harman, M. An Analysis and Survey of the Development of Mutation Testing. IEEE TSE 37(5), 649–678, 2011. doi.org/10.1109/TSE.2010.62
Just et al. 2014Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., Fraser, G. Are Mutants a Valid Substitute for Real Faults in Software Testing? FSE 2014, 654–665. doi.org/10.1145/2635868.263592973% of 357 real faults were coupled to mutants from common operators.
Kamei et al. 2013Kamei, Y., Shihab, E., Adams, B., Hassan, A. E., Mockus, A., Sinha, A., Ubayashi, N. A Large-Scale Empirical Study of Just-in-Time Quality Assurance. IEEE TSE 39(6), 757–773, 2013. doi.org/10.1109/TSE.2012.70
Kingman 1961Kingman, J. F. C. The single server queue in heavy traffic. Mathematical Proceedings of the Cambridge Philosophical Society 57(4), 902–904, 1961. doi.org/10.1017/S0305004100036094
Leveson 2011Leveson, N. G. Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press, 2011. Open access. direct.mit.edu/books/book/2908
Little 1961Little, J. D. C. A Proof for the Queuing Formula: L = λW. Operations Research 9(3), 383–387, 1961. doi.org/10.1287/opre.9.3.383
MacCormack et al. 2006MacCormack, A., Rusnak, J., Baldwin, C. Y. Exploring the Structure of Complex Software Designs: An Empirical Study of Open Source and Proprietary Code. Management Science 52(7), 1015–1030, 2006. doi.org/10.1287/mnsc.1060.0552
MacIver et al. 2019MacIver, D. R., Hatfield-Dodds, Z., et al. Hypothesis: A new approach to property-based testing. Journal of Open Source Software 4(43), 1891, 2019. Stateful testing: hypothesis.readthedocs.io/en/latest/stateful.html. doi.org/10.21105/joss.01891
McKeeman 1998McKeeman, W. M. Differential Testing for Software. Digital Technical Journal 10(1), 100–107, 1998.No stable online copy found.
Meyer 1992Meyer, B. Applying “Design by Contract”. IEEE Computer 25(10), 40–51, 1992. doi.org/10.1109/2.161279
Moser et al. 2008Moser, R., Pedrycz, W., Succi, G. A Comparative Analysis of the Efficiency of Change Metrics and Static Code Attributes for Defect Prediction. ICSE 2008, 181–190. doi.org/10.1145/1368088.1368114
Nagappan et al. 2008Nagappan, N., Murphy, B., Basili, V. The Influence of Organizational Structure on Software Quality: An Empirical Case Study. ICSE 2008, 521–530. doi.org/10.1145/1368088.1368160Organisational metrics: 86.2% precision, 84.0% recall, ahead of churn, complexity, coverage and dependency models.
Naur 1985Naur, P. Programming as Theory Building. Microprocessing and Microprogramming 15(5), 253–261, 1985. pages.cs.wisc.edu/~remzi/Naur.pdfUniversity-hosted copy.
Newcombe et al. 2015Newcombe, C., Rath, T., Zhang, F., Munteanu, B., Brooker, M., Deardeuff, M. How Amazon Web Services Uses Formal Methods. Communications of the ACM 58(4), 66–73, 2015. doi.org/10.1145/2699417
Page-Jones 1992Page-Jones, M. Comparing Techniques by Means of Encapsulation and Connascence. Communications of the ACM 35(9), 147–151, 1992. doi.org/10.1145/130994.131004
Papadakis et al. 2019Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Le Traon, Y., Harman, M. Mutation Testing Advances: An Analysis and Survey. Advances in Computers 112, 275–378, 2019. doi.org/10.1016/bs.adcom.2018.03.015
Parnas 1972Parnas, D. L. On the Criteria To Be Used in Decomposing Systems into Modules. Communications of the ACM 15(12), 1053–1058, 1972. doi.org/10.1145/361598.361623
Petrović & Ivanković 2018Petrović, G., Ivanković, M. State of Mutation Testing at Google. ICSE-SEIP 2018. See also Petrović, Ivanković, Fraser, Just, Practical Mutation Testing at Scale: A View from Google, IEEE TSE (online 2021). doi.org/10.1145/3183519.3183521
QuintQuint, a specification language based on the temporal logic of actions. quint.sh
Ren et al. 2004Ren, X., Shah, F., Tip, F., Ryder, B. G., Chesley, O. Chianti: A Tool for Change Impact Analysis of Java Programs. OOPSLA 2004, 432–448. doi.org/10.1145/1028976.1029012
Rothermel & Harrold 1997Rothermel, G., Harrold, M. J. A Safe, Efficient Regression Test Selection Technique. ACM TOSEM 6(2), 173–210, 1997. doi.org/10.1145/248233.248262
Segura et al. 2016Segura, S., Fraser, G., Sánchez, A. B., Ruiz-Cortés, A. A Survey on Metamorphic Testing. IEEE TSE 42(9), 805–824, 2016. doi.org/10.1109/TSE.2016.2532875
Śliwerski et al. 2005Śliwerski, J., Zimmermann, T., Zeller, A. When Do Changes Induce Fixes? MSR 2005, 1–5. doi.org/10.1145/1083142.1083147
Stevens et al. 1974Stevens, W. P., Myers, G. J., Constantine, L. L. Structured Design. IBM Systems Journal 13(2), 115–139, 1974. doi.org/10.1147/sj.132.0115
Wright, Hyrum's LawWright, H. Hyrum's Law: “With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.” www.hyrumslaw.com/
Yuan et al. 2014Yuan, D., Luo, Y., Zhuang, X., Rodrigues, G. R., Zhao, X., Zhang, Y., Jain, P. U., Stumm, M. Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems. OSDI 2014, 249–265. www.usenix.org/conference/osdi14/technical-sessions/presentation/yuan
Zhou et al. 2021Zhou, J., Xu, M., Shraer, A., Namasivayam, B., Miller, A., Tschannen, E., Atherton, S., Beamon, A. J., Sears, R., Leach, J., Rosenthal, D., et al. FoundationDB: A Distributed Unbundled Transactional Key Value Store. SIGMOD 2021. doi.org/10.1145/3448016.3457559
Zimmermann et al. 2005Zimmermann, T., Zeller, A., Weißgerber, P., Diehl, S. Mining Version Histories to Guide Software Changes. IEEE TSE 31(6), 429–445, 2005. doi.org/10.1109/TSE.2005.72
Figures are simulations written for this page. Numbers labelled Housewright come from runs of that generator in October 2026. Each reference was checked against a publisher, library or author page; the link opens that page.