Leaderboard | SynthArena by isChemist Group

uspto-190

General

190 targets from the test set of the original Retro*

Benchmark:

CuratedDeveloper

Total Targets

190

Stock

ASKCOS Buyables Stock

Acceptable Routes

Available

Show Top-K Metrics:

Overall Metrics

Model performance comparison for ASKCOS Buyables Stock

									Actions
ASKCOS	`v2.0.0`	17.4%	0.5%	0.5%	1.6%	95.5 min	$1.14
AiZynthFinder MCTS	`v4.4.0`	24.7%	1.1%	2.1%	2.1%	35.0 min	$0.10
AiZynthFinder MCTS (High)	`v4.4.0`	33.7%	1.6%	2.1%	2.1%	132.5 min	$0.39
AiZynthFinder Retro*	`v4.4.0`	32.1%	1.1%	1.6%	2.1%	111.0 min	$0.33
AiZynthFinder Retro* (High)	`v4.4.0`	36.8%	0.5%	1.6%	2.1%	421.2 min	$1.25
DMS Explorer XL	`v1.1.3`	29.5%	0.5%	1.1%	1.1%	65.0 min	$1.40
Retro*	`v1.0.0`	44.7%	7.9%	7.9%	7.9%	35.0 min	$0.10
Retro* (High)	`v1.0.0`	73.2%	10.0%	10.0%	10.0%	111.6 min	$0.33
SynPlanner MCTS Rollout	`v1.2.0`	0.0%	0.0%	0.0%	0.0%	84.2 min	$0.25
SynPlanner MCTS Val	`v1.3.2`	3.2%	0.5%	0.5%	0.5%	94.3 min	$0.28
SynPlanner NMCS	`v1.3.2`	5.3%	0.5%	0.5%	0.5%	190.6 min	$0.57
Syntheseus LocalRetro	`v0.7.0`	33.7%	0.0%	0.0%	0.0%	58.2 min	$0.17

Efficiency Frontier

Visualizing the trade-off between performance and resources.

Solvability


ASKCOS	55.6%	33.3%	23.1%	19.2%	7.7%	5.6%	7.7%	0.0%	0.0%	0.0%	0.0%	0.0%
AiZynthFinder MCTS	44.4%	45.8%	19.2%	34.6%	19.2%	38.9%	0.0%	11.1%	0.0%	8.3%	0.0%	0.0%
AiZynthFinder MCTS (High)	55.6%	50.0%	30.8%	38.5%	23.1%	50.0%	7.7%	55.6%	7.7%	16.7%	0.0%	0.0%
AiZynthFinder Retro*	50.0%	50.0%	42.3%	34.6%	34.6%	33.3%	15.4%	11.1%	0.0%	16.7%	0.0%	0.0%
AiZynthFinder Retro* (High)	55.6%	50.0%	42.3%	38.5%	34.6%	50.0%	23.1%	33.3%	7.7%	16.7%	0.0%	0.0%
DMS Explorer XL	72.2%	58.3%	38.5%	30.8%	15.4%	22.2%	7.7%	0.0%	0.0%	16.7%	0.0%	0.0%
Retro*	77.8%	58.3%	46.2%	38.5%	50.0%	38.9%	61.5%	55.6%	7.7%	8.3%	0.0%	0.0%
Retro* (High)	94.4%	79.2%	73.1%	69.2%	73.1%	61.1%	69.2%	66.7%	69.2%	75.0%	75.0%	0.0%
SynPlanner MCTS Rollout	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
SynPlanner MCTS Val	5.6%	4.2%	7.7%	3.8%	3.8%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
SynPlanner NMCS	5.6%	4.2%	23.1%	0.0%	3.8%	5.6%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
Syntheseus LocalRetro	50.0%	50.0%	23.1%	42.3%	42.3%	55.6%	30.8%	11.1%	0.0%	0.0%	0.0%	0.0%

Top-1


ASKCOS	0.0%	4.2%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
AiZynthFinder MCTS	11.1%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
AiZynthFinder MCTS (High)	16.7%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
AiZynthFinder Retro*	11.1%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
AiZynthFinder Retro* (High)	5.6%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
DMS Explorer XL	0.0%	0.0%	0.0%	3.8%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
Retro*	27.8%	8.3%	15.4%	3.8%	3.8%	11.1%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
Retro* (High)	44.4%	12.5%	15.4%	3.8%	3.8%	11.1%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
SynPlanner MCTS Rollout	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
SynPlanner MCTS Val	5.6%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
SynPlanner NMCS	5.6%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
Syntheseus LocalRetro	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%

Top-5


ASKCOS	0.0%	4.2%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
AiZynthFinder MCTS	11.1%	8.3%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
AiZynthFinder MCTS (High)	16.7%	4.2%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
AiZynthFinder Retro*	11.1%	4.2%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
AiZynthFinder Retro* (High)	11.1%	4.2%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
DMS Explorer XL	5.6%	0.0%	0.0%	3.8%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
Retro*	27.8%	8.3%	15.4%	3.8%	3.8%	11.1%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
Retro* (High)	44.4%	12.5%	15.4%	3.8%	3.8%	11.1%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
SynPlanner MCTS Rollout	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
SynPlanner MCTS Val	5.6%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
SynPlanner NMCS	5.6%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
Syntheseus LocalRetro	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%

Top-10


ASKCOS	5.6%	8.3%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
AiZynthFinder MCTS	11.1%	8.3%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
AiZynthFinder MCTS (High)	16.7%	4.2%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
AiZynthFinder Retro*	16.7%	4.2%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
AiZynthFinder Retro* (High)	16.7%	4.2%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
DMS Explorer XL	5.6%	0.0%	0.0%	3.8%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
Retro*	27.8%	8.3%	15.4%	3.8%	3.8%	11.1%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
Retro* (High)	44.4%	12.5%	15.4%	3.8%	3.8%	11.1%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
SynPlanner MCTS Rollout	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
SynPlanner MCTS Val	5.6%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
SynPlanner NMCS	5.6%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
Syntheseus LocalRetro	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%