Protocol#

The rules behind every number: what is in the corpus, how a field is scored, how each method runs and how results are ranked.

1. The corpus#

Test pairs only. Each dataset is a list of test cases (datasets/<name>/dataset.json), each a fixed image, a moving image and a ground truth. Training data is not part of the benchmark. The layout on disk is in CONTRIBUTING.md. CT is in Hounsfield units, masks are binary on their image’s grid, landmarks are x,y,z in millimetres in the world (ITK, LPS) frame.

2D datasets. A dataset of 2D images (photographs of the retina, stained sections, satellite images) says so ("dimension": 2 in its descriptor). Its images stay as the provider delivers them, a colour or multiband image one vector a pixel, and its landmarks are x,y. Its unit ("unit") is the provider’s pixel size, micrometres for a slide or metres for a satellite image, or the pixel when none is given ("px"): its spacing, its fields and every distance measured on it are in that unit. To compare datasets of other units and sizes, the landmark error is also given relative to the diagonal of the fixed image, as ANHIR does (rTRE, section 3). A method runs on 2D datasets when its method.json names 2D in "dimensions".

What a method reads and what scores it are kept apart. A method gets the pair, the modalities and the masks, never truth/. A label map that is scored is never handed over as a mask.

Built, then checked. Where the provider’s files need it, datasets/<name>/prepare.py turns the archive into one folder per case and changes nothing else. fidopt build writes the cases as the descriptor says, and fidopt verify compares every file with the checksums in the repository. Two rows are comparable only on the same checksums.

One storage format. The corpus keeps every image, mask and label map as an OME-Zarr store (NGFF 0.5, Zarr v3, sharded, compressed without loss), named after what it holds: fixed.ome.zarr, truth/fixed_labels.ome.zarr. Landmarks stay CSV. A store holds what the built image held and nothing else is changed:

  • Level 0 is the image: its voxels, its type and its channels untouched, a channel axis first when there are several (a colour image, the sequences of a multi-channel case).

  • The coarser levels are for looking at it and for reading it from afar: each halves the axes, rounded up so that every level covers the whole image, down to one chunk (64 voxels a side in 3D, 512 in 2D). An image takes the mean of each block of voxels, a mask or a label map its most frequent label. Nothing is scored on them, except on a large dataset (below).

  • The geometry. The scale and the translation are the spacing and the origin along the array’s axes, in mm or the dataset’s unit. OME-Zarr has no direction cosines, and half of the corpus has axes that are not the world’s (a fifth is acquired obliquely): the store keeps the image’s direction, origin and spacing in its konfai attribute, as KonfAI writes and reads it. A reader that ignores it sees the same voxels along the array’s axes.

A store reads back as the image it was written from, voxel for voxel, with its type and its geometry to the last digit: fidopt build and fidopt convert check it for every image before a NIfTI is replaced. Results scored on a NIfTI copy of the corpus therefore stand. A method is handed NIfTI as before (section 2): for an oblique image that file’s direction is the store’s as NIfTI writes it again, within 1e-8. A store is one entry of the checksums, the digest of its files: it checks a copy against the one that was recorded, and a store built again with another version of the libraries may differ in bytes while holding the same image.

Images too large for memory. Whole slides and light-sheet volumes do not fit in memory. A large dataset ("large": true in the descriptor) is built, aligned, warped and downsampled block by block, never holding an image whole, and the alignment or the known warp gives the voxels a resampling of the whole image gives: each block reads the part of the moving image its corners map to, widened by the warp’s largest displacement, which its coefficients bound. Its array axes are the world axes.

Which datasets. A dataset enters a main track when people drew, placed or checked its ground truth, when it can be downloaded without a private agreement, and when no other dataset of the corpus already holds its images. The SynthRAD sets, whose labels come from a segmentation network, are ranked in separate tracks. The datasets that were examined and left out are listed with the reason in datasets/screening.json.

No image twice. Lung250M-4B leaves its Learn2Reg LungCT pairs to L2R_LungCT, and Liver-DIR-QA leaves the 25 pairs it shares with AbdomenDIRQA. One overlap remains: the ten L2R_NLST pairs are also in Lung250M, on another grid and with manual landmarks. The lung track ranks on Lung250M; L2R_NLST stays in the Learn2Reg track only, next to the challenge’s own table.

Test cases. Where a task has an official split, the test cases are that split: each L2R_* dataset holds all the official validation pairs of its task. Elsewhere ten pairs are drawn per dataset. The cases with a ground truth, minus the development cases, are sorted by sha256("<seed>:<dataset>:<case>") and the first ones are taken, from each source in proportion to its share ("draw" in the descriptor; fidopt draw reproduces it). Ten is the smallest number that lets a paired test separate two methods, and small enough to run every method on every dataset. Where subjects have to be paired (Mindboggle-101, AFIDs), they are sorted the same way and paired two by two.

Development cases. Each dataset sets aside up to ten public cases for tuning ("development"), never used as test cases: the last ones of the same seeded order, from the source with the most left, each kept only if the test cases drawn stay the same (fidopt draw --development 10). For a Learn2Reg task they are ten pairs of the task’s training set, taken in the same order. Their checksums are recorded like the test cases’ (checksums_development.json). fidopt tune compares settings on them, across a family of datasets (datasets/families.json), and keeps its rows out of results/.

Where the ground truth comes from is in each descriptor ("truth"), graded expert, semi-automatic, automatic, synthetic or provider, with its limits. The benchmark says what kind of truth a dataset has, who made it, how it was checked and what it does not measure; it does not publish what would let a method learn the truth itself: the generator of a synthetic deformation, or how a synthesis reference was registered to its input (what happened with SynthRAD’s references, whose registration every entry learned). Three cases matter when reading a leaderboard:

  • Labels produced by a segmentation network (TotalSegmentator, MRSegmentator) are not manual annotations. Where a dataset also has landmarks placed by people (POPI, Lung-DIR-QA), its label metrics are listed as unranked: they are shown, and the ranking uses the landmarks.

  • The landmarks of the three DIR-QA datasets were found by a program, carried to the other image by a registration, then checked by people (semi-automatic). Methods close to that registration may be favoured.

  • Automatic keypoints are not a ground truth (L2R_NLST). When a metric is a method’s own output, results/notes.json takes it out of that method’s line.

Making a truth rather than leaving a dataset out. A dataset whose own truth is missing, on one image only, copied from one image to the other, or too easy is not dropped: the corpus makes one, says how, and grades it.

  • Labels from a public segmentation network run on both images (FastSurfer for LUMIR): graded automatic.

  • A known deformation ("warp" in the descriptor) for two images aligned by construction, such as two contrasts of one session: the moving side goes through a smooth, one-to-one synthetic deformation drawn once per case. Labels follow it, landmark pairs are moved through its inverse, and points drawn inside the body are added, each with its pair inside the moving image (a point the deformation would pair with one outside it is drawn again). The truth is exact, the deformation made by us: graded synthetic. The deformation is part of the test truth: it stays with it, and how it is drawn is not published, so that no method can be shaped to its family rather than to anatomy.

Both grades are ranked apart from the datasets people annotated.

What a main leaderboard needs. Since 2026-10-11 a dataset enters a main leaderboard only when its truth passes three checks, made before it is built and again on its no-registration line: people made or verified it on both images (expert or semi-automatic); it is dense enough to separate methods (ten test pairs, and a median of ten landmark pairs a case, or labels that cover the anatomy being registered rather than one structure in a corner of it); and the no-registration line leaves room (a Dice under 0.8, or a landmark error several times the truth’s own error where that is known). A dataset that misses a check is ranked apart, or the corpus makes it a better truth. Datasets built before keep their place, and their notes say where they fall short.

Other label sets. A case may carry a second label map next to its labels ("other_labels" in the layout, written as truth/fixed_labels_<set>.nii.gz): tumours that shrink or grow under treatment, or a network’s organs next to expert landmarks. It is scored like the labels, under the set’s name (dice_<set>, hd95_<set>), shown, and never ranked.

Alignment#

Every pair is aligned once, the same way for every method. Most datasets come aligned by their provider: the Learn2Reg tasks, the SynthRAD sets, Mindboggle-101 in MNI space, RESECT in the frame of the navigation system. For the others, the corpus computes one transform per pair (datasets/<name>/alignment/<case>.tfm) and applies it in the headers alone: the moving image, its mask, label maps and ignore region keep their own grid and every voxel, their origin and direction moved so that what they showed at T(x) now sits at x, and the landmarks follow (y to T^-1(y)). Nothing is resampled, so the provider’s voxels are what a method gets. A header holds a rotation and a translation and no more: the transform is a translation where that is enough, rigid where a rotation is needed, never affine, between two subjects too. Every deformation, breathing included, is left to the methods, and every method takes a moving image on its own grid (the convention check registers one).

Rigid registration is the one exception. A dataset of a rigid object whose images do not overlap where their provider put them (SpineUSCT: spine phantoms in CT and 3D ultrasound, where a rigid fit on the expert points leaves under 0.5 mm for any deformation) is not aligned by the corpus: aligning it would leave nothing to measure. It says "registration": "rigid"; a method runs its rigid stage alone (FIDOPT_REGISTRATION=rigid), only methods that can do so are run, and the dataset is ranked in a track of its own.

The transform comes from the intensities, never the truth ("alignment" in the descriptor), and it is the least one that works: the pair as delivered when its images already sit together, "image-centres" when only their centres differ, elastix only when needed: "elastix-translation" or "elastix-rigid", elastix 5.3.1 on Mattes mutual information inside each image’s body, run from a few starts (the bodies’ centres of mass together, the images’ centres together, and the two ends of the bodies together along the axis where their extents differ most, and the peak of the two images’ cross-correlation; from the centres alone with "-centred", where one image is a part taken out of the other, an extracted brain in a head); of the runs, the one with the highest product of the bodies’ overlap and the excess over 1 of the normalised mutual information over their common part is kept (that information rises by chance as the common part shrinks). Four resolutions, on images averaged down to at most 512 voxels an axis (a whole slide, a long CT), stored once per pair. Or "image-centres": the two images’ geometric centres put together and nothing optimised, for an ultrasound centred on its organ by its acquisition. The body is made from the intensities too (above -500 HU for a CT, above half of Otsu’s threshold otherwise; shaped on a grid of at most 256 voxels an axis, whatever the spacing: closed, its holes filled in 3D and slice by slice, opened, its largest part), so that a scanner table or fields of view cut differently on the two sides do not pull the start. A start computed from the annotations leaks the truth into the pair and gave badly oriented starts. The modes the current builds still hold until the corpus moves them (ANTs, or least squares on landmarks, label centres or one organ’s centre) are listed below; their affine forms are refused now.

dataset

transform

what is left after it

AMOS_CTCT

rigid, least squares on the centres of the annotated organs

Dice 0.26 (0.16 to 0.35)

LungDIRQA

rigid, least squares on the landmark pairs

landmark error 6.0 mm (2.2 to 12.5)

AbdomenDIRQA

rigid, least squares on the landmark pairs

7.4 mm (4.4 to 12.3)

LiverDIRQA

rigid, least squares on the landmark pairs

2.4 mm (1.3 to 6.6)

Mindboggle101

affine, least squares on the centres of the 62 regions

Dice 0.34 (0.30 to 0.38)

AFIDs

affine, least squares on the 32 fiducial pairs

2.9 mm (2.3 to 3.9)

muRegPro

the translation that matches the centres of the two gland contours

Dice 0.84, 5.8 mm (2.4 to 10.3)

ProstateMRUS

the translation that matches the centres of the two gland contours

Dice 0.82 (0.75 to 0.86)

QINProstateRepeat

the translation that matches the centres of the two gland contours

Dice 0.74 (0.67 to 0.83)

PelvicReference

rigid, least squares on the landmark pairs

1.9 mm (1.1 to 3.0)

RIDERLungCT

ANTs’ rigid, the scanner’s padding set to air

Dice 0.73 (0.37 to 0.87)

UltraCortex

ANTs’ rigid

Dice 0.54 (0.53 to 0.56)

PDDCA

rigid, least squares on the centres of the annotated organs

10.2 mm (5.0 to 13.2), Dice 0.37

PediatricCT

rigid, least squares on the centres of the annotated organs

Dice 0.34 (0.18 to 0.48)

BOBs

rigid, least squares on the centres of the 31 structures

Dice 0.62 (0.47 to 0.74)

LCTSC

the translation that matches the centres of the hearts

Dice 0.42 (0.32 to 0.53)

VerSe, SpineGeneric

the translation that matches the means of the landmark pairs, in the header

9.4 mm and 7.1 mm

SPIDER

rigid on the label centres where the scanner frame is wrong or the patient moved, the frame elsewhere

then a known warp

LUNA25

rigid, least squares on the centres of the lobes and airways (the nodules, the ranked truth, play no part)

3.0 mm (1.5 to 4.8)

LongitudinalCT

ANTs’ rigid

14.9 mm (6.7 to 32.2)

HNTSMRG

ANTs’ rigid

3.6 mm (1.0 to 8.8)

CTColonography

rigid, least squares on the centres of the labels both images hold

Dice 0.46 (0.35 to 0.59)

BreastNACT

ANTs’ rigid, which leaves two pairs worse than the scanner frame

Dice 0.47 (0.06 to 0.78)

AMOS_MRMR

rigid, least squares on the centres of the annotated organs

Dice 0.33 (0.23 to 0.41)

PancreasCT8

rigid, least squares on the centres of the annotated organs

Dice 0.35 (0.21 to 0.53)

KneeMR

rigid, least squares on the centres of the five bone and cartilage labels

Dice 0.52 (0.42 to 0.65)

TopCoW

rigid, least squares on the centres of the vessel labels both images hold, the internal carotids left out

Dice 0.57 (0.32 to 0.63)

CervixMR

ANTs’ rigid; on the pairs where it turned the pelvis, the least-squares rigid on the centres of the labels both images hold

9.1 mm (3.5 to 17.5)

MSLesSeg

ANTs’ rigid

Dice 0.79 (0.66 to 0.83)

PROMISE12

the translation that matches the centres of the two prostate masks

Dice 0.72 (0.55 to 0.86)

HNCBCTCT

ANTs’ rigid

Dice 0.72 (0.61 to 0.82)

MCRIB

affine, least squares on the centres of the 94 regions

Dice 0.48 (0.44 to 0.52)

NeuroPAL

affine, least squares on the neurons identified in both worms

5.9 µm (5.2 to 6.4)

MouseMicroCT

the translation that matches the centres of the two livers

Dice 0.44 (0.14 to 0.64)

LiverUSCT

the provider’s own rigid registration on the vessels; on case34, where it misses, the least-squares rigid on the landmarks

6.2 mm (2.7 to 9.8)

InnerEarCT

rigid, least squares on the three landmarks, which are then shown and never ranked; a left ear mirrored

Dice 0.57 (0.51 to 0.65)

MyocardialSPECT

the translation that matches the centres of the two myocardium masks

Dice 0.78 (0.63 to 0.86)

TongueMR

the translation that matches the centres of the intrinsic muscles

Dice 0.59 (0.48 to 0.72)

HandMR

rigid, least squares on the 25 joint pairs (ANTs’ rigid missed the palm on two pairs)

28.3 mm (9.7 to 39.6)

HQColon

rigid on the bones: the least-squares fit on the network labels’ centres, refined by a rigid registration of the sacrum, vertebrae and hips; the colon plays no part

Dice 0.51 (0.31 to 0.65)

PigCardiacCT

the translation that matches the centres of the hearts

Dice 0.73 (0.48 to 0.87)

TotalSegMRI

rigid, least squares on the centres of the labels both images hold

Dice 0.27 (0.18 to 0.41)

CRLMCT

the translation that matches the centres of the two livers

Dice 0.26 (0.19 to 0.33)

TopBrain

rigid, least squares on the centres of TopCoW’s circle of Willis classes both images hold, the internal carotids left out

Dice 0.48 (0.37 to 0.54)

RibSeg

rigid, least squares on the centres of the ribs both patients hold

Dice 0.06 (0.00 to 0.25), HD95 32 mm (22 to 52)

ToothCBCT

rigid, least squares on the centres of the teeth both images hold, numbered by TotalSegmentator; the expert masks play no part

Dice 0.54 (0.41 to 0.61)

PSMAFollowUp

ANTs’ rigid; where it matched the scanner’s table, the least-squares rigid on the centres of the labels whole in both CTs

8.1 mm (5.9 to 15.5)

FIRE

rigid, least squares on the ten point pairs (ANTs’ rigid left every pair as delivered)

9.9 px (1.2 to 18.9)

COph100

rigid, least squares on the ten point pairs; the infant’s retina grows between sessions, left to the methods

9.4 px (3.1 to 20.2)

CSXA

affine, least squares on the 23 keypoints (ANTs’ affine left one pair worse than delivered)

3.1 mm (1.7 to 5.8)

AarizCeph

affine, least squares on the 29 landmarks

3.8 mm (2.4 to 6.1)

WxBS

affine, least squares on the hand-placed correspondences (a rigid fit leaves 64 px, ANTs’ affine worse than delivered on three pairs)

22.1 px (3.8 to 45.2)

HPatches

affine, least squares on the points the homography carries (ANTs’ affine worse than delivered on four pairs)

70.7 px (17.0 to 109.8)

Between two subjects the affine fit the protocol prefers fails on several of these datasets: the centres are too few or lie on one line (the four tissue labels of UltraCortex, the discs of a spine), or a field of view that cuts the body or the bones reads as a size change (PediatricCT, the femur and tibia shafts of KneeMR), so the rigid or translation fit is kept, and each descriptor says why. Rotating the points of one curve around its own axis is why the spines keep a translation.

On LiverDIRQA little is left: all the landmarks are in the liver, which moves almost rigidly, so the leaderboard measures the last millimetres. ANTs’ rigid stage left 3 to 30 mm on those pairs, and without the annotations nothing tells a failure from a liver that moved under the ribs. L2R_ThoraxCBCT and L2R_HippocampusMR stay as delivered, since their published results were obtained on those pairs.

Frames and headers. The two AMOS_CTCT scanners place their world frames about one metre apart: before the transform is computed, the moving image’s origin is shifted so that the two bodies’ centres of mass coincide. Where a dataset’s files carry a wrong header or an intensity offset, its prepare.py fixes it and says so.

2. The field and its convention#

A method returns OUTDIR/DVF.nii.gz: a float32 image on the fixed grid with one component per dimension (3, or 2 on a 2D dataset), the displacement u in millimetres (in the dataset’s unit on a 2D dataset) along the world (ITK, LPS) axes, such that the fixed point x matches x + u(x) in the moving image. fidopt score also reads Learn2Reg displacements (voxels along the index axes, --voxels).

On a large dataset a method reads the level of the images it can hold and returns its field on a grid of its choice, coarser, along the fixed image’s axes, that covers the fixed image: every fixed voxel centre has a nearest voxel in the field. A method that reads no OME-Zarr is given each image and mask at its finest level within the method’s "max_voxels" (2^24 voxels unless its method.json says otherwise) and "max_side" (NiftyReg smooths no axis longer than 2048 voxels), as NIfTI, and returns its field on that level’s grid. The zero field is on the coarsest level.

A wrong sign or axis order gives a plausible wrong score, so the convention is checked for every method:

  • fidopt check <method> registers a synthetic volume shifted by a known translation, on an oblique, axis-permuted, anisotropic grid, with the moving image once on the fixed grid and once on another. The field as written must recover the translation within 1 mm; read with a wrong sign, component order, frame, direction or spacing, it must miss it by more than 2.5 mm. A method that takes 2D is also checked on a 2D colour image, three 8-bit components a pixel, on a grid of 0.7 by 2.5 mm whose index axes are turned by 215 degrees: its adapter’s conversion to one value a pixel, where it makes one, is checked with it.

  • fidopt check <method> --on <dataset> <case> uses the real scorer on a corpus case: only the field as written may beat the zero field. This is the check for networks trained on anatomy. uniGradICON and multiGradICON pass the synthetic check on one grid only, since they read each image in its own voxel frame.

3. Scoring#

One evaluation for every method, run by KonfAI’s evaluator. evaluation/<task>.yml holds it, one block per kind of truth; a dataset takes the blocks its truth has, filled with the values of its descriptor, and the file KonfAI resolves from them is kept beside the rows it scored (results/<machine>/<method>/<dataset>.evaluation.yml). Where Learn2Reg defines a metric, it is computed like the organisers’ code (MDL-UzL/L2R, evaluation/), except the TRE: KonfAI reads the field at a landmark linearly, as ITK does, where that code takes a cubic B-spline.

metric

definition

Dice

Moving labels warped by nearest neighbour, as ITK resamples: a point mapped more than half a voxel outside the moving image is background. Mean over the labels present in both the fixed and the moving map, as the organisers’ code scores. A label the warp erased counts 0, otherwise a field that loses structures would be rewarded (labels_lost counts them).

HD95

95th percentile of the surface distances (DeepMind surface-distance), in mm (the dataset’s unit on a 2D dataset), mean over the labels the warp kept.

TRE

‖x + u(x) − y‖ in mm (the dataset’s unit on a 2D dataset) over the landmark pairs (x, y), with u read at x as ITK’s DisplacementFieldTransform reads it (linear, from the voxels around x). Mean and 95th percentile.

rTRE

The TRE over the length of the fixed image’s diagonal, as ANHIR relates its landmark errors, without a unit: it compares datasets of other units and sizes. Shown, never ranked: it is the TRE again.

SDlogJ

Standard deviation of log(clip(det J + 3, 1e-9, 1e9)), J the Jacobian of x + u(x), 2 voxels cropped on each side (the + 3 is Learn2Reg’s). Also inside the body and inside the fixed mask when there is one.

folds

Fraction of voxels with det J ≤ 0, over the whole field and inside the body (the Otsu foreground of the fixed image).

time, memory

Wall time of run.sh including model loading, peak RAM of its process tree, peak GPU memory of its processes. A run that downloads a model first is not timed.

On a large dataset the landmarks are scored at full resolution: the field is read at each fixed landmark from its world coordinates, keeping its edge values out to the edges of its voxels as ITK reads it. The labels are scored at a reduced level, the same for every method whatever its grid: the finest level of the label maps within 2^26 voxels, recorded in the row (labels_level), the moving labels warped through the field read at each voxel of that level. The regularity is that of the field on its own grid, with the fixed image’s body and mask brought onto it.

What is kept. One row per test case in results/<machine>/<method>/<dataset>.json: the metrics, time and memory, the method’s version, fidopt’s version and the date. The displacement field is scored and then deleted.

A failure is a row. A case with no field, an invalid field, or a timeout is recorded with its error and counts as the unregistered pair, as a challenge scores a missing submission.

A licence that allows commercial use. The official leaderboards, a public release of the corpus and a paid service hold only the datasets whose licence is open and allows commercial use ("spdx" in the descriptor, every part of it one of LICENCES in fidopt/corpus.py), as the methods carry only open licences. A dataset under non-commercial, no-derivatives or research-only terms, or with no licence stated, is built and scored the same way, locally and ranked apart, until its provider allows more: its truth is no weaker, its terms are.

The partition follows the image. One image may serve several tasks (a segmentation or a synthesis dataset takes its cases from a registration dataset’s pairs); an image of a test pair is a test case in every task, an image of a development pair a development case, never the reverse.

A segmentation is scored by structure, by name. The dataset names its labels; the method writes its own labels and their names; the two meet through datasets/structures.json (one name for oesophagus and esophagus; kidneys made of kidney_left and kidney_right). Dice, HD95 and the surface Dice (at the descriptor’s truth.tolerance_mm, one for all or one a structure, 1 mm unless it says otherwise) are computed for every structure the dataset names, with KonfAI’s criteria as for registration. A structure the case’s truth holds and the method does not produce scores a Dice of 0 and no distance; one the truth lacks is left out of the case’s mean, unless the method writes it there, a false positive that scores 0 (Metrics Reloaded’s rule). The line’s Dice is the mean over the structures so scored, and the line says how many of the truth’s structures the method produced. A failed case is an empty map.

A synthesis is scored inside the body. The synthetic image against the real image of the target modality, where the dataset’s mask (or the real image’s body) is: the mean absolute error in the target’s units, the PSNR and the SSIM at the descriptor’s dynamic range, with KonfAI’s criteria. How the real image was brought onto the source image’s grid is the dataset’s own business, and is not described here (a method could learn it). No image stands in for a failed case: the line is incomplete, shown and not ranked, until the case is produced. A dataset whose target is a CT may ask for the anatomy kept (truth.geometry): TotalSegmentator’s KonfAI port segments the synthetic and the real image, and the Dice and HD95 of its structures are shown beside the line, never ranked, the reference being a program’s reading of the real image.

Parity with Learn2Reg. fidopt table --parity compares the zero field on each L2R_* dataset with the organisers’ own zero-displacement submission (datasets/<name>/official_zero.json). Every listed metric matches within 1e-4. Two leaderboards cannot be fully reproduced from public data, as their descriptors state: ThoraxCBCT scores 15 labels, of which 13 are public, and NLST scores landmarks that were not released (the public keypoint TRE is reproduced).

Next to the challenge’s table. datasets/<name>/published.json holds the rows the organisers published, with their source. They come from the hidden test set, not the validation pairs scored here, and from entries tuned on the task. The website shows them under the leaderboard as the level of the challenge, never ranked with it.

4. How methods run#

The authors’ code with the authors’ defaults. Each method runs its authors’ code at a pinned version (setup.sh), through its command-line tool or the function that tool calls, with every setting at its default. A protocol the authors document for one modality or anatomy is a method of its own (unigradicon_lungmask).

Two tiers. Tier 1 compares each tool’s defaults. Tier 2 holds settings chosen for a task, by a challenge’s organisers, by the authors for their entry, or by anyone who names the cases they tuned on. A tier 2 line is a variant (<method>@<configuration>, declared in method.json with its datasets and source), shown next to those datasets’ leaderboards and never ranked with tier 1. A variant tuned on a dataset’s test cases is not shown on that dataset.

Deformable stage only, from the same alignment. Since every pair is aligned (section 1), no tool runs its own rigid or affine stage: ANTs runs SyN alone, NiftyReg runs reg_f3d alone, deeds runs deedsBCV alone, as the literature runs them on aligned pairs. The IMPACT-Reg 1.8.7 rows are the one exception: each preset ran as shipped, and FireANTs_SyN and Generic_Rigid_BSpline kept their linear stages. The next presets are deformable only.

Same inputs. The pair, the modality of each image (ct, cbct, mr, us) and the dataset’s masks. A method is not run on a modality it does not take ("modalities" in method.json), nor on a 2D dataset unless it names 2D in "dimensions". A 2D colour image is given as it is: an adapter whose tool reads one value a pixel converts it (ANTs and Demons take the mean of its components), one whose tool reads several passes them on (greedy weighs the components itself, NiftyReg registers them as the channels of one image), and its run.sh says so.

One job at a time. A GPU method runs alone on its GPU; CPU methods run on a stated number of threads (results/<machine>/machine.json). A machine’s rows go to its folder of results/: the one whose machine.json lists the host in "hostnames", unless $FIDOPT_MACHINE or --machine names another.

Each baseline against its official configuration#

Each baseline was compared with the published script, not with a description of it.

method

official reference

what runs here

ants

the Learn2Reg 2024 organisers’ LUMIR baseline, ants.registration(..., 'antsRegistrationSyN[so]') (script)

the antsRegistration command ANTsPy builds for it

the same since 2026-10-09; earlier rows ran antsRegistrationSyN.sh -t so, equal within ANTs’ run-to-run spread (methods/ants/NOTES.md)

unigradicon, multigradicon

the authors’ unigradicon-register with its defaults

the same command

the same

anatomix_convexadam

the commands of the authors’ README

the same settings

the same

corrfield

the Learn2Reg organisers’ NLST baseline (script)

the Lung250M-4B authors’ copy with its defaults: MIND sigma 1 instead of 0.8, stages added where the organisers compose them

close; corrfield@l2r is the organisers’ copy, called as their script calls it

convexadam_mind

one script per task in the authors’ repository, their Learn2Reg entries

the repository’s generic defaults

different; convexadam_mind@l2r2021-lungct is their LungCT entry

deeds

-G 6x5x4x3x2 -L 6x5x4x3x2 -Q 5x4x3x2x1 on LUMIR, a task not in this corpus; the ThoraxCBCT settings are not published

the tool’s defaults, 8x7x6x5x4

different

niftyreg

the organisers describe MIND with bending and Jacobian penalties and fewer levels, without values

reg_f3d defaults: NMI, no Jacobian penalty

different, not reproducible from what is published

Where the official settings are published, the benchmark gives the challenge’s numbers. On the three Learn2Reg LungCT validation pairs:

line

landmark error

SDlogJ in the lungs

the challenge’s row, on its test set

convexadam_mind, defaults

3.68 mm

0.086

convexadam_mind@l2r2021-lungct

2.01 mm

0.065

ConvexAdam 1.79 mm, 0.06

corrfield, defaults

2.14 mm

0.056

corrField 1.75 mm, 0.05

corrfield@l2r

2.12 mm

0.055

The validation pairs start further apart than the test pairs (14.6 mm against 10.2). On the NLST validation pairs, corrfield@l2r lands on the task’s public keypoints within 0.006 mm: those keypoints are that baseline’s own output, which is why no TRE on them counts for corrField (results/notes.json).

Not reproduced: the ConvexAdam entries on OASIS, Hippocampus and AbdomenCTCT register the output of segmentation networks trained on the task’s labels, which are not published, and its AbdomenMRCT script runs on the authors’ own cropped and resized images. The organisers’ NiftyReg and deeds settings would have to be asked for.

The methods#

method

version

device

similarity

settings

ants

ANTs 2.6.5 release binaries

CPU

CC radius 2

the Learn2Reg organisers’ antsRegistrationSyN[so] call

niftyreg

NiftyReg v2.0.0

CPU

NMI, 64 bins

reg_f3d defaults

deeds

deedsBCV 00b4653

CPU

MIND-SSC

deedsBCV defaults

convexadam_mind

convexAdam b229e52

GPU

MIND-SSC, SSD

repository defaults

anatomix_convexadam

anatomix 4c86aee, weights 842cfe9

GPU

anatomix and MIND-SSC features, SSD

command-line defaults, CT window of the README

unigradicon, multigradicon

unigradicon 1.0.5, weights 1.0

GPU

LNCC in the instance optimisation

command-line defaults

unigradicon_lungmask, multigradicon_lungmask

same

GPU

same

the authors’ lung protocol: images multiplied by their lung masks; lung CT only

corrfield

Lung250M-4B 53598ca

GPU

MIND-SSC, SSD

command-line defaults; lung datasets only

impact_reg@<preset>

impact-reg-konfai 1.8.7

GPU

the preset’s

the preset’s defaults

identity

the zero field: the “no registration” line

methods/fireants (the FireANTs library with its tutorial settings) is there for cost comparisons, not as a baseline. DINO-Reg is not run: its repository has no licence.

  • ANTs. SyN[0.2, 3, 0] from the identity, cross-correlation of radius 2, 100x70x50x20 iterations at shrink factors 8x4x2x1, smoothing 3x2x1x0 voxels, single precision, then antsApplyTransforms writes the field. The adapter sets the thread count and a fixed random seed. Not applied: a mutual information stage for multimodal pairs.

  • NiftyReg. reg_f3d: cubic B-spline with control points every 5 voxels, NMI with 64 bins, bending energy 0.001, first-order penalty 0.01, no Jacobian penalty, 3 levels, 150 iterations at the last one, CPU build. Version 2.0.0, not the later 2.1: commit bd9ffd65 (first released in 2.1.0) scaled the bending and first-order gradients down to about two thirds and one third without changing the default weights, so the same defaults regularise far less (brain MR to CT: Dice 0.64 with 2.0.0, 0.35 with 2.1.2).

  • deeds. deedsBCV: 5 levels, grid 8x7x6x5x4, search radius 8x7x6x5x4, quantisation 5x4x3x2x1, regularisation 1.6, symmetric. deeds counts all of these in voxels and reads two arrays of one size without geometry: a pair whose voxels are not cubic is first resampled onto a cubic grid of as many voxels, as its README asks for thick slices, and the field is brought back to the fixed grid.

  • ConvexAdam (MIND). convex_adam_pt(fixed, moving) with no argument: MIND-SSC radius 1, dilation 2; convex stage on a 6-voxel grid with search half-width 4 and inverse consistency; Adam stage on a 2-voxel grid, 80 iterations, learning rate 1, diffusion weight 1.25; half precision. The capture range is in voxels: 24 voxels are 48 mm on a 2 mm grid and 14 mm on a 0.57 mm grid. Not applied: the per-task settings of the Learn2Reg scripts, self-configuring ConvexAdam, and the nnU-Net features behind most published ConvexAdam Dice scores.

  • anatomix + ConvexAdam. The pipeline of the anatomix paper: diffusion weight 0.75, convex grid 2, search half-width 1, 80 Adam iterations on a grid of 2; 16 anatomix channels and 12 MIND-SSC channels compared by SSD; features on a 128³ sliding window with 0.8 overlap; CT clipped to [-450, 450] HU, as in the README. Not applied: the per-dataset grid search the paper describes.

  • uniGradICON and multiGradICON. unigradicon-register with its defaults: the network’s prediction, then 50 steps of instance optimisation with LNCC. The tool knows ct (also used for cbct), clamped to [-1000, 1000] HU, and mri, clamped to [minimum, 99th percentile]. With the lung protocol each image is multiplied by its lung mask, as in the authors’ Learn2Reg lung submissions; the field then means nothing outside the lungs, so the masked and unmasked versions are two lines.

  • corrField. corrfield.py with its defaults: regularisation 2.5, two non-rigid stages, Foerstner keypoints inside the fixed lung mask, thin-plate spline interpolation. Lung datasets only. It runs on the authors’ grid: an axis finer than 1 mm is brought to 1 mm, as they resample their scans, and a volume still above 50 million voxels is coarsened evenly (LungDIRQA case30 at 1.16 mm); the field comes back to the fixed grid.

  • IMPACT-Reg. impact-reg-konfai register <preset> as a user runs it, with the preset’s defaults and, since 2026-10-09, the masks the dataset allows (the 1.8.7 rows ran without them). Its result depends on the machine’s memory: a pair that fits is registered whole, a larger one by a coarse pass then tiles, and the two paths do not give the same field. A line is the result on its machine.

5. Leaderboards#

  • A dataset’s leaderboard has one line per method and machine: the mean of each metric over the test cases, with failures counted as unregistered, the number of failures, the mean time and the peak GPU memory.

  • Rank. A complete line is ranked on every metric all complete lines share (Dice and HD95, TRE, SDlogJ; higher Dice and lower everything else is better), except those the dataset lists as unranked. Its rank is the mean of those ranks. A line that misses cases is shown but not ranked. The zero field is the reference and is not ranked, since its perfect smoothness would put it above methods that actually register; nor is a line that failed on every case, which amounts to the zero field. A method measured on several machines is ranked once, on its latest complete line.

  • Uncertainty. A dataset has ten test cases, so many ranks are noise, and the boards say so instead of hiding it. Each mean carries its 95% t interval over the cases. Two neighbouring lines are tied when a paired Wilcoxon signed-rank test on their cases (two-sided, 5%) does not find the upper one better on the dataset’s main measure (the first of Dice, TRE, HD95, MAE, PSNR and SSIM it ranks); a tied line is shown with the places it could hold, those of the lines it is tied with. On the results of October 2026, two thirds of neighbouring lines are tied, and so are the first two lines of most datasets. A leaderboard of several datasets shows beside each mean rank the mean of those places, low and high, in the spirit of challengeR’s rank ranges (Wiesenfarth et al., 2021) without its bootstrap.

  • Methods under a licence that is not open. A method is ranked when its code and its weights are under MIT, Apache-2.0, BSD, ISC, Zlib, CC0 or Unlicense. A method under another licence (non-commercial, share-alike, copyleft, custom) is run and shown after the ranked lines, under its licence, ranked nowhere.

  • A track’s leaderboard (datasets/tracks.json) ranks the methods ranked on every dataset of the track by their mean rank. There is no ranking across tracks, and a track names its task ("task", registration unless it says otherwise). Tracks marked "truth": "automatic" hold datasets whose labels come from a network, and are shown apart; so is a track marked "apart" with its reason ("four point pairs a pair, from two blocks"), its truth keeping its grade. A track ranks on the datasets whose licence admits them (spdx: every part an open licence that allows commercial use); a dataset under another licence is shown with its results, after them, and ranked in no leaderboard.

  • Worse than no registration. Each score is shown with its change from the unregistered pair. On the SynthRAD pairs, which come rigidly aligned, a method can do worse than that: a single-modality similarity on an MR/CT pair, a network outside its training data, or a weakly regularised deformation can break what was already right. This is a real result, not a scoring artefact: the method’s own moved image matches the scorer’s reading of its field. On the other datasets, no ranked method is below the unregistered pair, except lung-only methods scored on whole-thorax labels (POPI).

  • Off the leaderboards. A method whose method.json sets "board": false is run and its rows are kept, but it is not shown. IMPACT-Reg is in that case while its presets are being retuned: the leaderboards hold the baselines.

  • Notes (results/notes.json) flag a result that needs a look, or remove a metric from a line when its ground truth is the method’s own output.

  • Time is not ranked yet. The rows come from machines that were not timing references, so times compare within one machine only. A timing table needs every method on one idle reference machine.