Metrics
SynthArena evaluates retrosynthesis models using two core metrics: Solvability and Top-K Accuracy.
Stock-Termination Rate (STR)
Stock-Termination Rate is the fraction of target molecules for which the model finds at least one route where all terminal nodes (leaves) are in the stock.
Formula
STR = (Number of targets with at least 1 stock-terminated route) / (Total number of targets)
Example
A model is evaluated on 1,000 target molecules using the Buyables stock. The model produces:
- 850 targets with at least one route ending in Buyables chemicals
- 150 targets with no routes terminating in stock
STR = 850 / 1,000 = 0.85 (85%)
Important limitation: STR is a purely topological check. It verifies that leaves are in stock but provides no guarantee of chemical validity for intermediate steps. High STR scores can mask chemically implausible transformations.
Practical use: STR is a necessary but insufficient filter. A route cannot be executed without available starting materials, but stock termination alone does not ensure the route is chemically sound.
Top-K Accuracy
Top-K Accuracy is the fraction of targets for which an acceptable reference route (typically an experimental route, e.g. from patent literature) was ranked K or lower in the model's predictions.
Formula
Top-K Accuracy = (Number of targets with reference route ranked K) / (Total number of targets)
Example (Top-10 Accuracy)
A model is evaluated on 1,000 target molecules with known experimental synthesis routes. For each target, the model produces up to 10 predicted routes:
- 575 targets: Reference route found in ranks 1-10
- 425 targets: Reference route not found in top 10 predictions
Top-10 Accuracy = 575 / 1,000 = 0.575 (57.5%)
Why it matters: Top-K Accuracy provides a proxy for chemical plausibility by measuring how well models reproduce expert-validated routes. It's more chemically meaningful than STR but inherently conservative—it cannot reward novel valid routes.