fidopt

Tune your biomedical image registration, segmentation and synthesis method, or find the very best analysis method for your datasets.
Run it yourself, or let your agent query it.

The problem#

To compare a new method with the state of the art, a researcher downloads a dozen datasets, writes the preprocessing, picks the splits, reimplements the metrics, and runs the baselines, usually with less care than the new method gets. Every paper does this again, and every paper does it a little differently. The result is well documented: in registration, baselines are under-reported by several Dice points and conclusions flip with the evaluation set (Jena et al., NeurIPS 2024); in segmentation, most claimed improvements over U-Net vanish once the baseline is configured properly (Isensee et al., MICCAI 2024); in synthesis, voxel-wise scores against a registered reference reward the registration convention, and registration alone produces errors the size of a winning challenge entry (Boussot et al., 2026). Challenges fix this for one task and one round, on one dataset, then close, and what wins them is tuned to that dataset.

This matters more now that experiments are run by agents. An agent that develops a method should be able to ask a trusted service how good it is, and iterate, the way a person submits to a challenge, but in a loop.

What fidopt does#

It takes the whole evaluation off your hands and keeps it the same for everyone.

One corpus, built once. 311 datasets, 3,353 test cases and 2,960 development cases, with official splits where they exist and a seeded draw elsewhere, a checksum for every file, no image in two datasets, and every pair aligned the same way for every method. You fetch cases; you do not prepare data.
Ground truth made by people. Every main track is scored on labels or landmarks that experts drew, placed or checked. Labels produced by a network, and deformations we made ourselves, are ranked apart and say so.
One evaluation. The same scorer for every method, computing the Learn2Reg metrics exactly like the organisers’ code and checked against their own submissions. You never reimplement a metric.
Baselines at their best. Each baseline runs its authors’ code at a pinned version, compared line by line with its official configuration, and we tell you where a published setting could not be reproduced. Authors are invited to check how their method runs.
A hidden test, an open development set. Development cases come with their truth, to iterate on as often as you like. Test truth stays on our server; test submissions are scored there, capped and logged.
Every number traceable, everything in the open. A result records the method version, the corpus checksums, the machine and the date; the corpus, the protocol, the evaluation code and the submission log are public. Anything on a leaderboard can be questioned, and reproduced. What stays on our server is the test truth, and whatever would let a method learn a truth rather than the anatomy: how a synthetic deformation is drawn, how a synthesis reference was registered.
More methods, more datasets. Since the pipeline is ours, you compare with every baseline on every dataset of the corpus in the time it used to take to set up one.
Tuning in plain sight. Defaults compete with defaults; settings chosen for a task are a separate line, shown next to the board and never ranked with it. Across 311 datasets, how much a method owes to one dataset shows.
Methods you can reuse. A method on the official board is a KonfAI app, published with its line: see below.

Three tasks, one set of contracts#

Task	A method receives	It returns	Scored against
Registration	a fixed and a moving image, their modalities, masks	a displacement field	landmarks and labels: TRE, Dice, HD95, folding
Segmentation	an image and its modality	a label map	labels drawn by people: Dice, surface distance
Synthesis	an image of one modality (MR, CBCT)	the image of another (CT)	the real image: MAE, SSIM, then segmentation and dose computed on the synthetic image

Segmentation and synthesis reuse the registration machinery: the same descriptors, the same run contract, the same boards. See the roadmap.

Publish your method, and let people use it#

A method that enters the official board comes as a KonfAI app: a Prediction.yml, the weights, an app.json. We run it, score it, and, if its licence and its authors allow, publish it next to its line. From there it runs wherever KonfAI runs, with the same command: the command line, 3D Slicer through SlicerKonfAI, and the browser on WebGPU through KonfAI’s Rust engine. The benchmark is also the place where methods are found and used.

For agents#

Everything an agent needs is a command: list the problems, fetch the development cases, get scored on them as often as needed, submit to the hidden test a limited number of times, read where the method stands next to the baselines. Skills (.claude/skills/, for Claude Code and any agent that reads a SKILL.md) follow fidopt’s three parts. The corpus: add-dataset adds a dataset, from its licence to its independent review. The methods: add-method adds a published method, from its licences and training data to its rows; wrap-method wraps a method of any of the three tasks as an app that meets its contract and checks it; verify-method reviews a method before it joins, its licence, its convention, its pins, its settings against the authors’ own. The evaluation: score-and-boards changes a metric, the ranking or the site.

The corpus#

The reserved case manifest lists test and development case names for every dataset in the tracks, including inherited splits for derived tasks. Use the manifest from the same Git revision as the dataset descriptors and checksums. Exclude both partitions from model training, including other examinations of the same subjects and copies in other collections. Development cases are for tuning and validation; test cases are for final evaluation. These case names are not a complete crosswalk of original patient identifiers. Case counts include derived tasks that reuse source images; they are not counts of distinct patients.

Test and development each reserve 10% of independent subjects, with a minimum of 10 and a maximum of 30; a provider’s released test split stays whole when applicable. Another examination or section of a test subject cannot fill development. A listed split below the minimum does not meet this rule. Sources that cannot support independent partitions are retained as examples outside the benchmark. Case names remain reserved against training regardless of whether the split meets the rule or its payload has been uploaded. On Hugging Face, every tracked dataset has test/ and development/ folders. Available cases live under <Dataset>/<split>/<case>/; PENDING.md marks a split whose data have not been uploaded, and a derived task’s SOURCE.md points to its parent images without duplicating them. The four one-subject examples are under examples/, outside the benchmark. Access and redistribution remain subject to each dataset’s licence.

Track	Datasets	Ground truth
Lung CT, same patient	Lung250M-4B, Learn2Reg LungCT, Lung-DIR-QA, POPI, LUNA25 screening rounds, one pig lung at 13 inflations	landmarks
Abdomen CT, same patient	Abdominal-DIR-QA, Liver-DIR-QA, HQColon CT colonography supine to prone	landmarks, colon labels
Abdomen CT, two patients	AMOS, Learn2Reg AbdomenCTCT, Pancreas-CT, Colorectal-Liver-Metastases	organ labels
Abdomen MR, two patients	AMOS MR, TotalSegmentator MRI	organ labels
Prostate MR, two patients	PROMISE12, four centres	prostate labels
Fluorescence microscopy, between animals	NeuroPAL worm heads	identified neurons
Mouse micro-CT, same animal	mice re-positioned between sessions, tumours that grow	organ labels
Retina photographs (2D), same eye	FIRE fundus pairs, one pair an eye; COph100 infants’ first and last RetCam sessions	landmarks
Heart ultrasound (2D), same patient	CAMUS apical views, end diastole to end systole	labels
Radiographs (2D), two patients	CSXA lateral cervical spine, Aariz lateral cephalograms	keypoints, landmarks
Photographs of scenes (2D), two viewpoints or times	WxBS: day and night, seasons, wide baselines	correspondences placed by hand
Plants (2D), same plant over days	MSU-PID Arabidopsis in chlorophyll fluorescence, two days apart	leaf outlines and tips
Chest and body CT, two patients	LCTSC, Pediatric-CT-SEG (children), RibSeg (the 24 ribs)	organ and rib labels
Head, neck and spine, two patients	PDDCA, VerSe, Spine-generic, inner ear CT, tongue muscles MR, teeth CBCT	labels, landmarks
Knee MR, two patients	OAI-ZIB knees, osteoarthritis grades 0 to 4	bone and cartilage labels
Hand MR, two poses, same patient	PIANO right hands	joint landmarks
Brain MR, two subjects	Mindboggle-101, AFIDs, Learn2Reg Hippocampus, UltraCortex (9.4 T), M-CRIB neonates	labels, landmarks
Brain MR, same patient	BraTS-Reg, Baby Open Brains (infants who grow)	landmarks, labels
Heart MR, same patient	STACOM 2011 tagged MR, Sunnybrook, ACDC and York cine MR, pigs at 7 T	landmarks, labels
Heart SPECT, two patients	myocardial perfusion scans (PhysioNet)	myocardium masks
Heart CT, same animal, months apart	growing pigs, contrast CT at consecutive rounds	heart and left ventricle labels
Thorax and abdomen MR, breathing, same patient	children’s dynamic MR at end expiration and end inspiration	organ labels
Two modalities, same patient	Learn2Reg AbdomenMRCT and ThoraxCBCT, RESECT, BITE, µ-RegPro, Prostate MR–US biopsy, pancreas CT–CBCT, pelvis CT–CBCT, CHAOS MR T1–T2, TopCoW and TopBrain brain MRA–CTA, liver 3D US–CT	labels, landmarks
Repeat scans, same patient	RIDER lung CT, QIN prostate MR	labels
Tumours under treatment, same patient	autoPET IV whole-body CT follow-up, HNTS-MRG head and neck MR, cervix MR during radiotherapy	lesion centres
Learn2Reg validation sets	AbdomenCTCT, AbdomenMRCT, Hippocampus, LungCT, NLST, OASIS, ThoraxCBCT	as released
Brain MR, two subjects, ranked apart	LUMIR (Learn2Reg 2024)	labels from a network
MR or CBCT to CT, ranked apart	8 SynthRAD sets	labels from a network
Known deformation, ranked apart	Vestibular schwannoma MR, SPIDER spine MR, inner-ear CT to micro-CT, autoPET FDG and PSMA PET to CT	a known warp, exact
Same patient, ranked apart	CT colonography prone to supine, UCSD brain metastases, breast MR under chemotherapy, multiple sclerosis brain MR, head and neck CT to CBCT, PSMA PET-CT follow-up	labels from a network, lesions linked by a program
Mouse CT to MR, ranked apart	mice scanned by CT and MR in one session	organ labels whose makers are not stated
Planar scenes (2D), ranked apart	HPatches viewpoint sequences	homographies the authors estimated

The images stay with their providers: fidopt build assembles the corpus from the original downloads and fidopt verify checks every file.

The baselines#

Method	Kind	Runs on	Install and run	KonfAI app
ANTs SyN	classical	CPU	`methods/ants`	`ANTs_SyN`
NiftyReg	classical	CPU	`methods/niftyreg`	`NiftyReg`
elastix B-spline	classical	CPU	the app	`Elastix_BSpline`
FireANTs greedy, the tutorial’s settings	classical, GPU-native	GPU	the app	`FireANTs_Greedy`
deeds	classical	CPU	`methods/deeds`	`Deeds`
greedy	classical, one modality	CPU	`methods/greedy`	`Greedy`
Demons	classical, one modality	CPU	`methods/demons`	`Demons`
corrField	classical, lungs	GPU	`methods/corrfield`	`CorrField`
ConvexAdam	classical	GPU	`methods/convexadam_mind`	`ConvexAdam_MIND`
uniGradICON and multiGradICON	learned	GPU	`methods/unigradicon`	`UniGradICON`, `MultiGradICON` (`--set masks=true` for the lung protocol)
anatomix + ConvexAdam	learned features	GPU	`methods/anatomix_convexadam`	`Anatomix_ConvexAdam`
BrainMorph	learned, brain MR	GPU	`methods/brainmorph`	`BrainMorph`

Each folder holds everything for one method: setup.sh installs the authors’ code at a pinned version (with the network weights, checked against their hash), run.sh registers one pair, method.json gives its source, version, licence and paper, and NOTES.md says exactly what runs and how it compares with the official configuration.

Segmentation baselines are KonfAI apps, run through methods/konfai_segmentation, their labels matched with each dataset’s structures by name. TotalSegmentator runs twice: the authors’ package wrapped (fidopt-segmentation), and its KonfAI port:

Method	Structures	Modalities	App
TotalSegmentator 2.18.0 `total`, `total_mr`, the authors’ package	117 of CT, 50 of MR	CT, MR	`TotalSegmentator`, `TotalSegmentator_MR`
TotalSegmentator `total`, `total_mr`, its KonfAI port	117 of CT, 50 of MR	CT, MR	`VBoussot/TotalSegmentator-KonfAI`
MRSegmentator	40	MR, CT	`VBoussot/MRSegmentator-KonfAI`
IMPACTSeg `body`	11, tissues and cavities	CT, MR, CBCT	`VBoussot/ImpactSeg`
DentalSegmentator	jaws, teeth, mandibular canal	CBCT, CT	`VBoussot/DentalSegmentator-KonfAI`

Synthesis baselines are KonfAI apps too, run through methods/konfai_synthesis:

Method	From	To	App
TotalSynth `MR`	MR	CT	`VBoussot/ImpactSynth`
TotalSynth `CBCT`	CBCT	CT	`VBoussot/ImpactSynth`

Every registration baseline is also a KonfAI app, checked against its methods/ line: the same authors’ code at the same pin (ConvexAdam: itk-impact’s C++ port at the authors’ settings), the settings in a Prediction.yml. They run like the IMPACT-Reg presets: impact-reg-konfai register ANTs_SyN -f fixed.nii.gz -m moving.nii.gz -o out, or with konfai-apps alone, konfai-apps infer fideus-labs/fidopt-registration:ANTs_SyN -i fixed.nii.gz -i moving.nii.gz -o out; fidopt run konfai_app@ANTs_SyN <dataset> runs one through the benchmark.

Quick start#

git clone https://github.com/fideus-labs/fidopt && cd fidopt
python3 -m venv .venv && .venv/bin/pip install -e .
export FIDOPT_DATA=/path/to/corpus      # built from the original datasets, see CONTRIBUTING.md
fidopt verify                           # checks the corpus against the repository's checksums

methods/convexadam_mind/setup.sh          # installs the authors' code at a pinned commit
fidopt run convexadam_mind L2R_OASIS    # registers and scores every test pair
fidopt table --track lung-ct            # prints a leaderboard
fidopt site site/ --serve 8090          # the leaderboards as a website, with an image viewer

Already have displacement fields? fidopt score my_method Lung250M --fields /path/to/fields scores them directly. The rules behind every number are in PROTOCOL.md.

Get involved#

Bring your method. Three files and one command put it on every leaderboard it applies to. See CONTRIBUTING.md.
Bring a dataset. A public dataset with a ground truth people made is welcome, especially for anatomies and modalities the corpus lacks.
Wrote one of these methods? Check how we run it in methods/<name>/. If it differs from how you would run it, tell us: we want every baseline at its best.
Spotted a problem? Open an issue. Every number can be traced back, so every number can be questioned.

Acknowledgements#

This benchmark stands on the work of the teams who released the datasets and the methods it uses, credited in each datasets/<name>/dataset.json and methods/<name>/method.json, and on the Learn2Reg organisers’ evaluation code. The logo draws the lungs of a Lung-DIR-QA scan (CC BY 4.0).

Status and licence#

The code is released under the Apache 2.0 licence; each dataset keeps its own. fidopt is developed by Fideus Labs, whose own methods compete under the same rules.