Contributing#

Add a method#

Create methods/<name>/ with three files:

setup.sh installs the tool next to the script, at a pinned version: a release, a commit, exact pip versions.
run.sh registers one pair (the contract is below).
method.json describes it:
```
{"title": "My method", "summary": "One sentence on what runs.", "device": "gpu",
 "source": "https://...", "version": "1.2.3", "licence": "MIT", "paper": "Author et al., Venue 2025"}
```
The licence is an SPDX identifier among the open ones the benchmark admits: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, Zlib, CC0-1.0, Unlicense, for the code and for the weights. A method under another licence (non-commercial, share-alike, copyleft, custom) can be run: its results are shown apart, ranked nowhere. A method with no licence, or whose terms forbid running it here, is not added.

"modalities" and "anatomy" restrict it to the datasets it is made for, and "dimensions": [2, 3] says it takes 2D datasets too (3D only by default). A method that takes several images of a case at once names them in "channels", in the order it reads them (["t1", "t1c", "t2", "flair"]): it runs on the datasets whose "channels" are exactly those, and a method without the key on single-image datasets only. "trained_on" lists the datasets whose cases its weights saw: it is never run on them. "variant" names what its variants are ("configuration", "app", "matcher"); one that runs only through a variant says "variants_only": true: its bare name applies to no dataset. A segmentation method names the structures it writes in "structures": it runs where the dataset’s truth holds at least one of them (names matched as in scoring).

An agent can do this for you: the add-method skill (.claude/skills/) is the order of work, from the licence of the code and the weights and the training data against the test cases to the run; wrap-method writes the wrapper as a KonfAI app, from a template per task, and checks it; verify-method reviews a method the way a maintainer does.

Then check its output and run it:

fidopt check <name> --gpu 0             # the licence, then the field convention on a synthetic pair
fidopt run <name> <dataset>             # rows in results/<machine>/<name>/<dataset>.json

Beside the rows, <dataset>.evaluation.yml is how they were scored: the Evaluation.yml KonfAI ran, every default resolved, the scratch folder written $WORK. It is rewritten by every run that scores rows.

The contract#

run.sh FIXED MOVING OUTDIR FIXED_MODALITY MOVING_MODALITY [FIXED_MASK] [MOVING_MASK]

Input. Two NIfTI images (the benchmark writes them from the corpus’s OME-Zarr stores), their modalities (ct, cbct, fluorescence, fundus, microct, mr, pet, spect, us, xray, photo, shg, brightfield, chlorophyll, optical, sar, em) and the masks the dataset allows. The pair is already aligned, rigidly for one patient and by an affine transform between two subjects, so run only your deformable stage. The exception is a dataset scored as a rigid registration ("registration": "rigid" in its descriptor): its images are left where their provider put them, run.sh is given FIDOPT_REGISTRATION=rigid, and it runs its rigid stage alone and writes that transform as the field. Such a dataset is run only by a method whose method.json says "rigid": true, which also says its rigid stage takes the dataset’s modalities (a "monomodal" method’s rigid stage then compares them by mutual information, as greedy’s does). A method never sees the ground truth. On a 2D dataset the images are 2D, as the provider delivers them: a colour or multiband image is one vector a pixel. If your tool reads one value a pixel, convert in run.sh and say how there (the ANTs and Demons adapters take the mean of the components); if it reads several, give it the components (the greedy and NiftyReg adapters).
Output. OUTDIR/DVF.nii.gz, a float32 image on the fixed image’s grid with one component per dimension (3, or 2 on a 2D dataset). Each voxel holds the displacement u in millimetres (in the dataset’s unit on a 2D dataset) along the world axes (ITK, LPS): the fixed point x matches x + u(x) in the moving image. This is what sitk.DisplacementFieldTransform and ANTs read.
Failure. A non-zero exit, a missing or invalid field, or a timeout is recorded and scored as no registration.
Images too large for memory. On a large dataset (whole slides, light-sheet volumes) run.sh is given each image and mask at its finest level within "max_voxels" (method.json, 2^24 voxels by default) and "max_side" voxels along any axis (a tool’s own limit: 2048 for NiftyReg) as NIfTI, or, with "ome_zarr": true, the multiscale OME-Zarr stores themselves, to read the level it can hold. It writes OUTDIR/DVF.nii.gz on a grid of its choice, along the fixed image’s axes, that covers the fixed image: the grid of the level it read does.

Every tool writes its displacement field its own way, and a field read the wrong way still gives a believable score. fidopt check catches this before any run: it shifts a test image by a known amount, asks the method to register it back, and checks that the field finds that shift. For a network that only works on real anatomy, use a real case: fidopt check <name> --on <dataset> <case>.

A segmentation method#

The same three files, method.json saying "task": "segmentation", and the contract:

run.sh IMAGE OUTDIR MODALITY [MASK]

Input. One NIfTI image, its modality, and the mask a dataset gives when it gives one; on a dataset with "channels", a NIfTI vector image, one component per channel in the descriptor’s order. The method never sees the ground truth.
Output. OUTDIR/SEG.nii.gz, the label map on the image’s grid, in the method’s own label ids, and OUTDIR/terminology.json, those ids and their names ({"1": "spleen", "2": "kidney_right"}). The benchmark matches the names with the dataset’s (datasets/structures.json: aliases and merges, kidneys being kidney_left and kidney_right) and scores every structure the case’s truth holds; one the method does not produce counts 0, one it writes where the truth has none is a false positive at 0, and the line says how many it produced. A KonfAI app’s terminology is that file: methods/konfai_segmentation runs any such app.
Failure. A non-zero exit, a missing or invalid map, or a timeout is recorded and scored as an empty map.

fidopt check <name> --on <dataset> <case> scores the method on one case, structure by structure.

A synthesis method#

method.json says "task": "synthesis", and the contract:

run.sh IMAGE OUTDIR SOURCE_MODALITY TARGET_MODALITY [MASK]

Input. The image of the source modality, the two modalities (mr ct, cbct ct), and the mask a dataset gives; a source of several channels is a NIfTI vector image, as for a segmentation.
Output. OUTDIR/SYN.nii.gz, the synthetic image of the target modality on the image’s grid, in that modality’s units (Hounsfield units for a CT), finite everywhere.
Scoring. Inside the body (the dataset’s mask, or the real image’s Otsu body): the mean absolute error, the PSNR and the SSIM against the real image of the target modality, as SynthRAD scores them. A failed case is recorded with its error.

fidopt check <name> --on <dataset> <case> scores the method on one case against an image that knows nothing, a constant at the real image’s median.

Tuned settings#

Defaults compete with defaults. Settings chosen for a task, by a challenge’s organisers or by the authors for their entry, go in a variant <name>@<configuration>, declared under "variants" in method.json with its source and the datasets it was configured for (see methods/corrfield/method.json). Variants are shown next to the leaderboard and never ranked in it.

To tune your own settings, use the development pairs, never the test pairs:

fidopt build <dataset> --from /path/to/cases --dev      # into $FIDOPT_DEV
fidopt tune <name> --family lung-ct --try a="..." --try b="..."

Each setting reaches run.sh as $FIDOPT_SETTINGS. The families are in datasets/families.json.

Score fields computed elsewhere#

Put one field per test case in a folder, named <case>.nii.gz:

fidopt score my_method Lung250M --fields /path/to/fields            # the format above, in millimetres
fidopt score my_method L2R_OASIS --fields /path/to/disp --voxels    # Learn2Reg displacements, in voxels

The rows go to results/submitted/. They show accuracy only: time and memory are measured only when the benchmark runs the method itself. On a large dataset a field is in millimetres, on a grid that covers the fixed image.

A scoring server holds the truth of the test cases and answers a submission with its line, the means over the cases, never a case’s score:

fidopt submit my_method Lung250M --fields /path/to/fields --server https://...   # HF_TOKEN, or `hf auth login`
fidopt submit my_method Lung250M --fields /path/to/fields --resume <id>            # after a cut connection
fidopt server --port 8000     # FIDOPT_DATA: the corpus with its truth; FIDOPT_SERVER: the server's own folder

Each field goes as a Zarr store compressed by ZFP, every value within 0.01 mm, file by file; a file the server holds is not sent again. Your Hugging Face account is who submits: three scored submissions a dataset in thirty days, a submission refused before scoring (a case missing, a field off its grid) not counted. The server logs who sent what and the line, deletes the fields once scored, and opens again a submission a restart caught scoring.

Add a dataset#

This section is the reference for a descriptor’s keys and the build commands. The corpus’s rules (reading a licence, the truth bar and ranking apart, no image twice, the split, the independent review) are in the add-dataset skill, .claude/skills/add-dataset/SKILL.md.

Write datasets/<name>/dataset.json: what the dataset is, its modalities, its licence, where its ground truth comes from and how good it is, and source.layout, which file of a case folder is the fixed image, the moving image, a mask, labels or landmarks (see datasets/POPI/dataset.json). If the pairs are not aligned, set "aligned": false and say how the corpus aligns them in "alignment": "elastix-translation", or "elastix-rigid" when a rotation is needed (elastix on mutual information, applied in the moving headers alone), either followed by -centred when one image is a part taken out of the other (an extracted brain in a head), or "image-centres", the images’ centres together without optimising (an ultrasound centred on its organ). A label map’s labels are named in "truth": {"labels": {"1": "spleen", "2": "kidney_right"}}, with the names of datasets/structures.json (TotalSegmentator’s spelling where it has the structure; a dataset’s own name otherwise), so that a segmentation is scored structure by structure, by name, with the surface Dice at truth.tolerance_mm: one number for every structure, or one a structure by name ({"lung_covid_infiltrate": 3}, 1 mm for the others); 1 mm unless given. A dataset whose cases hold several images of one modality given together (BraTS’ T1, T1Gd, T2 and FLAIR) gives them as one vector image (a NIfTI of intent 1007 from prepare.py, the image of its layout, stored with a channel axis) and names its components in "channels", in that order, from t1, t1c, t2, flair, pd, dwi, adc: only methods taking those channels run on it, and its track is its own. A segmentation dataset says "task": "segmentation" and lays out one image a case: "layout": {"image": ..., "labels": ..., "mask": ..., "ignore": ...} (ignore, optional: a region the dataset’s protocol takes out of both the truth and the prediction before scoring, as HVSMR does at the vessels’ ends; a mask goes to the methods, an ignore region never does; on a registration source, fixed_ignore and moving_ignore serve the cases derived from it); a synthesis dataset says "task": "synthesis", its modalities the source then the target, and lays out {"image": ..., "target": ..., "mask": ...}, the target being the real image of the target modality on the image’s grid (truth.dynamic_range, one number, the width of the intensity range, sets the PSNR and SSIM range; CT’s 4095 unless given). "truth": {"geometry": true} on a dataset whose target is a CT adds the anatomy kept: TotalSegmentator’s KonfAI port run on the synthetic and the real image, Dice and HD95 shown, never ranked. A dataset of either task may take its cases from a registration dataset already built, without a file copied: "from": {"dataset": "AMOS_CTCT", "images": ["fixed", "moving"]} makes each image of a pair a segmentation case (<pair>/<side>, with that side’s mask and labels; "labels": "<set>" takes the source’s other label set truth/<side>_labels_<set>.nii.gz instead, its names then in the descriptor’s own truth.labels), and "from": {"dataset": ..., "source": "moving", "target": "fixed"} each pair a synthesis case, on pairs aligned voxel to voxel; the cases, the development cases, the label names and the licence are the source dataset’s unless the descriptor says otherwise, and the partition follows the image. truth.labels_about is the sentence that says where the labels come from. A second label map, shown and never ranked, goes in "other_labels": {"<set>": ["<fixed file>", "<moving file>"]}. Two contrasts of one modality (a T1 and a T2 MR) are named in "contrasts", fixed first: a method that compares intensities is not run on them. A dataset of 2D images says "dimension": 2 and its unit, "unit": the unit of the provider’s pixel size ("µm", "m"), or "px" when it gives none, the spacing then 1 (see datasets/FIRE/dataset.json). Its images stay as delivered, in colour if they come in colour, and its landmarks are x,y.
If the provider’s files need it, add a prepare.py that turns the archive into one folder per case.
Without an official split, name the development cases ("development") and the draw ("draw"), then:
```
fidopt draw <name> --from /path/to/cases                     # the test cases, from the descriptor's seed
fidopt draw <name> --from /path/to/cases --development 10    # then ten development cases, the test cases unchanged
fidopt draw <name> --from /path/to/training --development 10 # or from another split, when the test cases are the provider's
fidopt build <name> --from /path/to/cases --record           # the corpus, as OME-Zarr, and its checksums
fidopt build <name> --from /path/to/cases --dev --record     # the development pairs, into $FIDOPT_DEV
fidopt run identity <name>                                   # the "no registration" line
```
Development cases later taken from another split leave the draw’s pool as it was only if the ones it left out are named: "draw": {"held_out": [...]}, so that fidopt draw gives the same test cases from the descriptor alone.

The licence goes in "spdx" too, as an SPDX expression: "CC-BY-4.0", or "CC-BY-3.0 AND CC-BY-4.0" when the images and the annotations differ. The official leaderboards, a public release and a paid service hold only the datasets whose every part is an open licence that allows commercial use (LICENCES in fidopt/corpus.py: CC0, PDDL, CC BY, ODC-By, CC BY-SA, ODbL, and MIT, Apache-2.0 or BSD when a dataset carries one). A dataset under non-commercial, no-derivatives or research-only terms ("LicenseRef-<terms>"), or with none stated ("NOASSERTION"), is built and scored locally, apart, until its provider allows more; "use" says the same in a word: open, non-commercial or ask.
Add it to a track in datasets/tracks.json.

The corpus is not hosted publicly. Each user builds it from the original datasets, and fidopt verify checks it. Members of Fideus Labs can download the private copy instead (hf download fideus-labs/fidopt --repo-type dataset --local-dir corpus). Each dataset groups its test cases under <dataset>/test/ and available development cases under <dataset>/development/. A PENDING.md in either split means those cases must still be built from the original dataset; a derived task’s SOURCE.md points to its parent images. The four one-subject examples are under examples/, outside the benchmark. To use the downloaded copy with fidopt, link the desired split folder into an $FIDOPT_DATA/<dataset>/ or $FIDOPT_DEV/<dataset>/ path:

After building and verifying both local splits, python datasets/upload_hf.py <name> synchronises their cases to <dataset>/test/ and <dataset>/development/ in the private copy.

$FIDOPT_DATA/<dataset>/<case>/
    fixed.ome.zarr, moving.ome.zarr              the pair, CT in Hounsfield units
    fixed_mask.ome.zarr, moving_mask.ome.zarr    optional
    truth/                                       label maps as stores, landmarks as CSV, read only by the scorer

The corpus is stored as OME-Zarr. A provider’s files can be NIfTI, MetaImage or anything SimpleITK reads, and so can what prepare.py writes: fidopt build writes each image, mask and label map of a case as one multiscale OME-Zarr store, and checks that it reads back as the same image (PROTOCOL, section 1). A segmentation case is image.ome.zarr, mask.ome.zarr, truth/labels.ome.zarr, truth/ignore.ome.zarr; a synthesis case adds truth/target.ome.zarr; a multi-channel image is one store with a channel axis. A layout names files as before, with .nii.gz: the store takes the same name. A copy built before this, in NIfTI, is read the same and is converted in place, each image checked, with

fidopt convert <name> [<name> ...] --record     # the test cases, and their checksums recorded again
fidopt convert <name> --dev --record            # the development cases, in $FIDOPT_DEV
fidopt verify <name>                            # the copy against the recorded checksums

A method is still given NIfTI and still returns NIfTI. fidopt build --nifti leaves a build as NIfTI.

Images too large for memory (whole slides, light-sheet volumes) take "large": true in the descriptor: they are built block by block, from a MetaImage, an uncompressed NIfTI or an OME-Zarr store (other formats are read whole); their axes must be the world axes and their masks on their image’s grid.

Before a pull request#

python tests/test_contracts.py