One shipped, one shelved: two changes to how my engine counts evidence

Background: how the engine combines steps

PerspectiveGraph evaluates an attack path by assigning each step a probability - how likely an attacker is to successfully overcome it - and combining these probabilities. To estimate the overall risk to a resource, it also runs a simulation: thousands of trials, in each of which each step succeeds or fails, and counts how many times the asset is reached.

This method works when steps are independent. Some are not. If two steps depend on the same vulnerability, their success or failure is linked: in a world where that vulnerability is exploitable, both steps work, and in a world where it is not, neither works. The engine has long used a method to express this concept - a "shared cause" for a step - and the simulation then links the two steps.

Shipped: one CVE, two packages

The mechanism existed, but it wasn't used. Only an external data feed could declare a shared cause, and none of the built-in sources did. Therefore, the case it was designed to address wasn't addressed.

This is a common scenario. Debian splits a library into several installable packages - OpenSSL into libssl3 and openssl, the C library into libc6 and libc-bin - and a scanner like Trivy reports a vulnerability for each installed package. In the graph, both packages then point at the same vulnerability. The simulation drew the two links independently, as if the attacker had two separate opportunities to exploit the same weakness. A vulnerability with a 50% chance of success was counted as a 75% chance: one minus the chance that both attempts would fail.

The fix is specific: every link to a vulnerability now reports that vulnerability as the shared cause. This is correct within the model: the probability on those links comes only from the vulnerability itself - whether it is known to be exploited, its expected exploitation, or its severity - not from anything related to the link. The reality is less straightforward - the same flaw reached via a TLS service in libssl3 and in an openssl binary that no one runs is not one chance - and the engine is still unable to distinguish between them. Full correlation is the error that underestimates rather than inflates. The next step, "what is gained by exploiting this vulnerability?", is a different question and continues to be drawn on its own.

A test reproduces the structure: one vulnerability, two packages, 50%. Before the fix, the simulation reported 0.748; afterward 0.50. A control - the same graph with a node that doesn't represent a vulnerability - still reports 0.75, so the fix changes exactly what it should. The scores for the individual paths haven't changed at all: a single path passes through a vulnerability only once. What has changed is the overall risk value, and consequently, the amount of risk a fix appears to reduce.

I haven't measured how often this happens on real images. The test proves the mechanism, not the size of the effect in your environment.

Shelved: counting confirmations

The second idea was more ambitious. The probability of each step is accompanied by uncertainty, the magnitude of which depends on the type of evidence supporting it. The engine gives a step supported by a known-exploited vulnerability the weight of approximately 69 observations - a chosen prior, not data; a runtime alert, 59; an exploitation prediction, 50; a CVSS score, 20; a bare severity label, 8; a guess based on the shape of the infrastructure, 6. Each additional independent observation adds one, and the interval narrows.

The idea: when multiple tools independently report the same step, count them as additional observations. It seemed like a free way to get more precision. Before writing it, I measured three things.

1. Are there confirmations to count?

I listed which sources write which types of steps. Where two sources can write the exact same step, they read the same underlying data: a vulnerability scanner and an SBOM both read the image's list of installed packages; the build provenance and its signed attestation both come from the same CI build; the network source and Cloud Custodian both read the same EC2 API. No other pair ever produces the same step. Counting these overlaps would mean counting the same thing twice - the same bug I'd just fixed for vulnerabilities. One of these pairs didn't even meet: checking the public demo later, I discovered that the scanner's and the SBOM's copies of the same library were two separate nodes, a keying bug fixed in 1.28.2. I'd listed the overlaps by reading the code, without running the two sources simultaneously.

2. How much would it change?

I recalculated the 90% intervals of the fourteen paths the public demo had at the time with the engine code itself, adding one confirmation to each estimated step - far more than the sources could ever provide, hence an upper bound. The median interval narrowed by 6.5%; with two confirmations, by 12.5%. Two paths got 3% wider: that is the sampling noise of the simulation itself. One confirmation is about as good as the noise.

3. Would it measure the right thing?

The fact that two tools agree on the existence of a step indicates that the connection is real. It says nothing about the ease with which that step can be crossed, which is the main source of uncertainty. For vulnerability steps it would be plainly wrong: two scanners detecting the same vulnerable library don't make the prediction of its exploitation more accurate. Narrowing the interval in this case would mean claiming a certainty that no one possesses.

The decision

Not implemented. Building it would have required changes to both storage backends, to the incremental analysis and to the cleanup of data that a source stops sending - several days of work on delicate code - for a small effect based on a wrong premise. What I did instead was note the reason, so the issue isn't accidentally reopened, and indicate when it should be reopened: when there are concrete test results showing whether the uncertain steps are incorrect in their values or only too uncertain.

The useful source of extra observations, when it comes, is something that observes the crossing itself - a penetration test confirming a single step, for example - not a second inventory of the same facts.

What the measuring found instead

The list of sources describing the various steps had a side effect. Two sources both describing "this instance uses this role" never met in the graph, and following that thread led to five identity bugs, three of which invented attack paths. This story is told in a separate article. The design study for a feature I didn't implement turned out to be more useful than the feature itself.

What I take from it

The cost of a feature includes the claim it makes. Narrower intervals might have looked like progress on a dashboard, but they would have meant less than they appeared. Measure before building and be prepared for a possible "no": a well-motivated "no" is a result, and can be cheaper than any feature.

• weightcause_test.go - one vulnerability on two packages, and the control
• Scoring and priority - how the probabilities, intervals and simulation work
• Concepts in plain words - the terms used here