CascadeRank Agent

以下内容可直接粘贴到 Devpost。采用英文是因为活动提交材料面向英文评审。

Inspiration

CascadeRank Agent began with a result that looked promising but could not survive methodological scrutiny.

The original project used a Graph Attention Network to rank critical nodes. However, several centrality signals were used both as model inputs and as components of the training target. The model was then compared against some of those same centrality measures. Under that design, a high correlation or visually convincing attack curve could reflect teacher reconstruction or label-feature leakage rather than the discovery of genuinely better critical nodes.

This exposed a broader problem in graph machine learning: it is easy to produce a ranking, but much harder to establish what that ranking proves. Weak baseline selection, hidden target overlap, single-run comparisons, and unverifiable reports can turn a plausible result into an unsupported scientific claim.

Instead of optimizing the model until the chart looked favorable, I redesigned the project around a stricter question:

Does the available evidence actually support the claim that a learned graph ranking outperforms established non-learned baselines?

CascadeRank Agent, implemented as the CascadeRank Audit Codex plugin, is designed to answer that question before a result is presented as evidence.

What it does

CascadeRank Audit is a local, evidence-first workflow for auditing critical-node ranking claims.

Given a graph, it can run two complementary experiments:

1. An intentionally leaky case, which reproduces a design where the target and model features share centrality information.
2. A topology-only case, which removes the declared direct leakage by using constant node features and supervising the GAT with the observed loss in largest-connected-component size after removing each node individually.

For each case, the system:

• Detects declared label-feature and target-baseline overlap.
• Compares the learned proxy against six non-learned rankings:
  • Degree
  • Betweenness
  • Closeness
  • Eigenvector centrality
  • PageRank
  • Radius-2 Collective Influence
• Executes 100 random-ranking trials.
• Reports a 5th–95th percentile random uncertainty envelope.
• Measures largest-connected-component degradation at predefined checkpoints.
• Computes attack AUC, where lower values indicate a more effective fixed node-removal ranking.
• Produces a machine-readable claim ledger.
• Generates Markdown, HTML, JSON, and visual attack-curve artifacts.
• Records SHA-256 hashes for the input graph and generated evidence.
• Independently verifies artifact integrity and recomputes the verdict rules.

Conceptually, the attack metric is:

[ \mathrm{AUC}=\int_0^1 \frac{|\mathrm{LCC}(G \setminus S_q)|}{|V|}\,dq, ]

where (S_q) is the set of nodes removed at fraction (q). Rankings are fixed before removal rather than recomputed after every attack step.

Every scientific claim receives exactly one of three verdicts:

• SUPPORTED
• NOT_SUPPORTED
• INCONCLUSIVE

A claimed GNN advantage is supported only when the learned ranking has a lower attack AUC than the best declared traditional baseline and no direct leakage is detected. Cross-graph generalization remains INCONCLUSIVE unless it is tested on genuinely held-out graphs.

CascadeRank does not convert a negative result into a positive narrative. On the demonstration graph, both the intentionally leaky design and the repaired topology-only design fail to establish a GNN advantage. That negative result is the product working as intended.

The audit runs locally and does not require an OpenAI API key.

How we built it

I used Codex with GPT-5.6 throughout the core implementation and review process.

The work began by asking Codex to inspect the original graph-ranking pipeline as both an algorithm engineer and a critical scientific reviewer. That review identified the central methodological defect: the original GAT was partially learning a target constructed from signals already present in its features and evaluation baselines.

I then used Codex iteratively to:

• Trace target and feature provenance.
• Implement explicit leakage and overlap detection.
• Separate the intentionally leaky experiment from the topology-only experiment.
• Implement fixed node-removal attack evaluation.
• Add the six traditional graph-ranking baselines.
• Add repeated random-ranking trials and uncertainty envelopes.
• Define predeclared, machine-readable verdict rules.
• Build the independent manifest verifier.
• Add input and artifact SHA-256 integrity checks.
• Create automated unit and pipeline tests.
• Package the workflow as a local Codex plugin and reusable skill.
• Build reproducible PowerShell demonstration and video-generation scripts.
• Review the resulting claims and remove language not supported by the experiments.

The Codex skill instructs the agent to inspect audit_manifest.json before interpreting the human-readable report. It also explicitly prohibits changing NOT_SUPPORTED or INCONCLUSIVE into more favorable language.

This makes Codex part of the product workflow rather than merely a code generator or text summarizer: it executes the audit protocol, reads structured evidence, invokes verification, and communicates the result within predefined scientific boundaries.

Challenges we ran into

The first major challenge was that the original model appeared technically functional while the central scientific comparison was invalid. The code could train, rank nodes, and generate plausible outputs, but those outputs did not establish independent predictive value.

The second challenge was redesigning the target. Removing centrality features from the target was necessary, but not sufficient. The topology-only target reduced direct leakage, yet the resulting GAT still did not outperform the strongest traditional baseline. This forced a choice between continuing to tune toward a favorable result and preserving the negative evidence. CascadeRank adopts the latter.

The third challenge was baseline selection. Comparing a GNN only against one convenient centrality measure would be methodologically weak. The audit therefore evaluates multiple baselines and selects the lowest-AUC traditional method rather than assuming a preferred comparator in advance.

The fourth challenge was uncertainty. A single random ranking is unstable and can give a misleading reference curve. We replaced it with repeated random-ranking trials and a percentile envelope. This improves the comparison, although it does not substitute for uncertainty across independent graphs, training seeds, or model specifications.

The fifth challenge was reproducibility. A Markdown report alone cannot prove that the displayed result matches the original input and computation. We therefore created a separate verifier that checks SHA-256 hashes and mechanically recomputes the fixed verdict rules from the manifest.

Finally, the product had to work without an OpenAI API subscription. The core audit, verifier, reports, and demonstration run entirely offline. Codex is used through the local project and plugin workflow rather than through an API call embedded in the audit runtime.

Accomplishments that we're proud of

The most important accomplishment is not a new state-of-the-art score. It is building a system that refuses to claim one without sufficient evidence.

CascadeRank Audit now provides:

• A functioning local Codex plugin.
• A reproducible command-line audit workflow.
• Explicit target and feature provenance checks.
• Six non-learned comparison methods.
• Repeated random-ranking uncertainty analysis.
• Predeclared claim rules.
• Machine-readable evidence manifests.
• Human-readable Markdown and HTML reports.
• An independent integrity and verdict verifier.
• Tamper detection for both input data and report artifacts.
• Automated tests and reproducible demo scripts.
• A complete workflow that requires no OpenAI API key.

We are also proud that the demonstration contains a real negative result. It shows that eliminating direct leakage does not automatically make a learned ranking superior. The system preserves that conclusion instead of hiding it.

What we learned

The first lesson was that prediction, reconstruction, and discovery are not interchangeable. A GNN can reproduce a centrality-derived target accurately without discovering a ranking that is better than centrality.

The second lesson was that leakage removal is a necessary validity condition, not proof of model value. Once direct overlap disappears, the learned method must still outperform strong independent baselines under a predeclared evaluation protocol.

The third lesson was that baseline choice is part of the scientific claim. A method cannot claim superiority by selecting the easiest comparator after seeing the results.

The fourth lesson was that reproducibility requires more than releasing code. Inputs, configurations, reports, verdict rules, and generated artifacts must remain linked through verifiable provenance.

The fifth lesson was that negative results can have direct product value. In evidence-sensitive workflows, preventing an unsupported claim can be more useful than producing another favorable visualization.

Finally, we learned that Codex is most useful when it is constrained by falsifiable rules. Its role in CascadeRank is not to make the result sound convincing. Its role is to execute, inspect, verify, and explain what the evidence permits—and what it does not permit.

What's next for CascadeRank Agent

The current release audits one-graph critical-node ranking claims. It does not yet establish causal node importance or cross-graph generalization.

The next development priorities are:

1. Held-out multi-graph evaluation
   Add benchmark collections with graph-level train, validation, and test separation so cross-graph transfer can be evaluated rather than assumed.

2. Training uncertainty
   Add repeated model seeds, confidence intervals, paired comparisons, and sensitivity analysis for architecture and hyperparameter choices.

3. Stronger baseline families
   Extend the comparison set with k-core methods, VoteRank, graph-dismantling algorithms, influence-maximization baselines, and domain-specific heuristics.

4. Adaptive attack protocols
   Clearly separate the current fixed-ranking protocol from adaptive attacks that recompute rankings after each removal step.

5. Additional graph types
   Support directed, weighted, temporal, heterogeneous, and multilayer networks with evaluation rules appropriate to each setting.

6. Task-specific definitions of importance
   Separate structural fragmentation, epidemic influence, information diffusion, controllability, and robustness rather than treating “critical node” as a single universal property.

7. Continuous evidence auditing
   Integrate the verifier into CI so changes to data, metrics, artifacts, or claim rules automatically invalidate stale reports.

8. Stronger provenance guarantees
   Add signed manifests, environment capture, dependency locking, and reproducible execution metadata.

The long-term goal is not to guarantee that a GNN wins. It is to make every claimed advantage traceable to a reproducible experiment whose assumptions, comparisons, uncertainty, and limitations are explicit.

Built With

• gat
• gnn