Same name, different thing: five identity bugs in my attack-path engine

Why identity is the whole game

PerspectiveGraph reads the output of tools users already run - Cloud Custodian exports, AWS IAM and network data, Kubernetes dumps, vulnerability scanners - and merges them into a single graph. A path is only as good as these connections. When a role seen by one tool and the same role seen by another become two separate nodes, a path through that role breaks and disappears. When two different elements sharing the same name become a single node, the engine connects parts that were never connected before.

The second type is worse. A missed path is a gap; an invented one is a confident assertion that someone will invest time in, which leads them to trust the next report less.

How I found them

I was considering an idea: count how many independent sources confirm each step in the graph, and give a step greater confidence when multiple sources agree. To estimate the effect, I listed which sources describe which types of steps. Two sources both describe "this EC2 instance uses this IAM role", yet in the graph their two descriptions never matched. That was the first thread.

From there I stopped reading code and ran the real pipeline - the same parsing and merging used by the product - on small inputs that combine sources, and examined the resulting paths. Each case reported below was reproduced this way before being fixed.

The five

One thing, two names

AWS assigns each role an ARN, a unique identifier that includes the account. Three of my sources used this. Cloud Custodian's parser used the role's short name instead, so "app" from Custodian and the ARN of the same role from the IAM source appeared to be two different nodes. The IAM source detected that this role could self-assign administrator privileges; the instance seen by Custodian was connected to the other copy, which had no such step. The path existed in reality but not in the graph.

The instances also had the same problem. An EC2 instance ID is unique only within an account, so the network source includes the account in its identifier; Custodian didn't. The same instance appeared to be two nodes - one aware of its exposure to the internet, the other aware of its role.

Two things, one name

Kubernetes includes built-in roles in each cluster, with names like admin, edit and view. My engine stores them with the same label used for AWS roles, identified by name. So, an AWS role called admin, read by Custodian by its name, and the Kubernetes role admin became a single node - with the AWS role having permission to read the account's S3 data. The engine reported this:

web-lb  ->  web-1  ->  web/web-sa  ->  admin  ->  customer-exports
(Kubernetes load balancer -> pod -> service account -> "admin" -> S3 bucket with personal data)

A profile is not a role

An EC2 instance doesn't point directly to its IAM role. It points to an instance profile, which contains the role. The AWS console creates a profile with the same name as the role, so it's easy to assume the two names are always the same. Terraform and CloudFormation often assign them different names. My Custodian parser interpreted the profile name as the role name. With a profile called web-profile that contained a role called web-role, an internet-facing instance with administrator rights didn't generate any paths, because it was pointing to a role no one had defined.

Namespaces and clusters

In Kubernetes a Role belongs to a namespace. Two namespaces can each have a Role called deployer with completely different rules. My Kubernetes parser identified Roles only by name, so the harmless deployer in one namespace (able to read configuration) became the same node as the deployer in another (able to create pods - a recognized escalation path: a new pod runs under any service account in its namespace and, where privileged pods are allowed, can take the node; the engine conservatively models this as a path to the cluster-admin). The harmless role inherited the path. This happens within a single cluster, and names like deployer or developer repeated across namespaces are common.

Across clusters, the problem recurred at a higher level. Clusters are built alike: many have a prod namespace, every namespace has a default service account, and every cluster has a cluster-admin. Without the cluster in each identifier, dumps from two clusters described one imaginary cluster, and a path could start in one and finish in the other.

Why the tests did not catch them

Each source was tested, and the tests passed. They verified that each parser transformed the input into the correct nodes and steps. None of them tested what happens when two sources describe the same thing - or two different things with the same name - in a single graph.

And the sample data was a lucky case. In my Custodian example, both the instance profile and the role were called web-admin, so a wrong assumption yielded the correct answer. A sample where everything matches proves nothing about the cases where it doesn't.

The fix: unique within what?

One rule, applied everywhere: identify each thing by what makes it unique in the system that owns it.

• AWS roles by their ARN, in every source.
• EC2 instances by account and instance ID, as the network source already keyed them.
• Kubernetes Roles by namespace and name. Cluster-wide roles keep their name.
• Kubernetes objects by cluster, when the cluster is named. A dump can now be sent with ?cluster=prod-eu, and the pull-request check takes the same name, so a change meets the right cluster. With a single cluster nothing changes.

Where the engine can't know, it now declares it. The link from an instance to its role is read from the instance profiles when the Custodian export includes them. Otherwise, the engine still assumes that the role shares the profile name - but marks this link as a guess and gives it half the weight compared to a declared one, so the dashboard displays a step to verify rather than a given fact.

Every case above is now a test that runs all the involved sources together, end to end. I also reintroduced each bug, one at a time, and verified that the corresponding test failed.

What it costs the people using it

By changing the way elements are identified, their IDs also change, and a path's ID is built from the nodes it passes through. Therefore, paths passing through the affected nodes got new IDs, and any item registered with the old ones - a suppression, a ticket - no longer matches. Worse still, the old nodes don't disappear on their own: these sources send partial views, and nothing in a partial view indicates "this no longer exists". They stay until staleness pruning removes them - and it is off unless GRAPH_TTL is set - or until the graph is rebuilt from scratch. The upgrade notes specify this clearly, because an update that silently keeps the old false paths next to the new correct ones would be a bug in itself.

Then I looked again

The first draft of this article stated that other sources might have the same class of bug. And so it was. Before publishing, I examined every type of node and step written by multiple sources, and ran each pair together. Four more:

• Two sources, one step, two probabilities. Once the identities met, Custodian and the network source wrote the same "this instance uses this role" step with different probabilities - the network source knew whether the instance requires the most secure version of the metadata service, Custodian did not - and the last one written won: 0.6 or 0.8 depending on arrival order, and a Custodian guess at 0.4 could override a fact that the network source had stated at 0.9. Fixed in 1.28.1: both evaluate the step equally, and a guess never overrides a fact.
• The scanner and the SBOM never met. The public demo showed Log4j twice. The SBOM source identified a package as "name:version", while Trivy distinguishes between name and version, and named a Java library by its artifact where Trivy adds its group; therefore the SBOM's copy never encountered the vulnerability detected by Trivy. No paths were incorrect, but every package reported by both was counted twice.
• Runtime alerts never reached a detected pod. Falco associated an alert with the container's name; the Kubernetes source identifies the pod it's running in by namespace/pod. The alert remained on a separate node, and no path through a pod found by the Kubernetes source was ever marked as confirmed at runtime - one of the engine's most important signals, silently missed.
• Two accounts, one load balancer. Custodian identified load balancers and databases by name only, as it had done with the instances. With two accounts exported, an internet-facing web-alb in one and an internal web-alb in the other became a single public load balancer, connected to the second account from the internet. Invented. My first fix added the account but it was still incomplete: those names are unique per account and Region, so two Regions of one account were still merged, and a load balancer was attached to instances in any Region, where no load balancer can route.

The last three bugs were fixed in 1.28.2, while the load balancers were only fully fixed in 1.28.3. Two of them had tests that indicated the right thing but verified the wrong thing. The SBOM test expected "the same ID used by Trivy" and calculated it according to the SBOM's method; the Falco test expected "the canonical container ID" and calculated the one used only by Falco. Furthermore, the public demo hid the Falco bug: its topology was written by hand with Falco's key - once again the lucky sample.

Update, 1.30.0. The load-balancer fix was tested on records not produced by Custodian. Custodian calls application and network load balancers aws.app-elb, a type my reader was unfamiliar with, so a real export's ALBs and NLBs were completely eliminated - a third lucky sample. Fixed in version 1.30.0, and with the same key as the network source.

What I still wouldn't trust

• That pass was performed manually. I read how each source creates its own keys and analyzed the pairs that share a node type. A check that automatically analyzes each pair of sources with overlapping inputs would be better, but it doesn't currently exist.
• Repositories are identified by their short name. Two repositories with the same name but different owners would be considered a single repository. I have left it that way, because it takes two owners in the same environment, and the documentation specifies this.
• Users are shared across clusters on purpose. A Kubernetes user or a named group is treated as the same identity in each cluster, assuming that the clusters share a directory. In an organization where this isn't the case, this assumption is incorrect.
• A guessed join is still a guess. It's better to flag it than to hide it, but the correct solution is an export that includes instance profiles.

What I take from it

For every identifier, ask "unique within what?" - an account, a namespace, a cluster - and incorporate this information into the identifier. Test your sources together, not one at a time, because the most interesting bugs are found where they meet. And consider a confidently reported path that doesn't exist to be the most serious bug an attack-path tool can have: it costs people time and undermines trust in the tool.

• identity_test.go - each case above, as a test that runs the sources together
• Upgrade notes for 1.28.0 - what changes for a running installation, and what to do about it
• Onboarding guide - how to send each source