Metrics

SynthArena separates route validity from stock-constrained target success using the Solv-N framework.

Tier-0 validity and Solv-0[stock]
SynthArena reports two separate target-level success predicates. Neither is a reaction-feasibility claim.

Tier-0 validity rate

The fraction of benchmark targets for which the planner produced at least one syntactically valid candidate: its molecular graphs and reaction records are well formed. Tier-0 does not establish reaction topology, chemical feasibility, or stock termination.

targets with ≥1 Tier-0-pass candidate / all benchmark targets

Solv-0[stock] rate

The fraction of benchmark targets for which at least one candidate passes Tier-0 and terminates in the named stock. The stock label is part of the metric identity: Solv-0[buyables-stock] and Solv-0[n5-stock] are different predicates and must not be compared as if they used the same task.

targets with ≥1 candidate passing Tier-0 and stock constraints / all benchmark targets
Top-K Accuracy
Top-K Accuracy is the fraction of targets for which an acceptable reference route (typically an experimental route, e.g. from patent literature) was ranked K or lowerin the model's predictions.

Formula

Top-K Accuracy = (Number of targets with reference route ranked K) / (Total number of targets)

Example (Top-10 Accuracy)

A model is evaluated on 1,000 target molecules with known experimental synthesis routes. For each target, the model produces up to 10 predicted routes:

  • 575 targets: Reference route found in ranks 1-10
  • 425 targets: Reference route not found in top 10 predictions

Top-10 Accuracy = 575 / 1,000 = 0.575 (57.5%)