Roadmap#

Where fidopt is going, and in what order.

The vision#

More and more experiments are run by agents. An agent that develops an algorithm should be able to ask a trusted service how good it is, the way a person submits to a challenge, but in a loop: try, get scored, improve. We want to be that service, reached through commands an agent’s skills call, for three tasks that share one set of contracts:

Task

A method receives

It returns

Scored against

Measures

Registration

a fixed and a moving image, their modalities, masks

a displacement field

landmarks, labels

TRE, Dice, HD95, folding

Segmentation

an image and its modality

a label map

labels drawn by people

Dice, surface distance

Synthesis

an image of one modality (MR, CBCT)

the image of another (CT)

the real image of that modality

MAE, SSIM, and segmentation or dose computed on the synthetic image

Registration comes first because it is built: 98 datasets, 1,035 test pairs, a dozen baselines run from their authors’ code, one scorer, convention checks. Segmentation and synthesis reuse the same machinery. Their candidates are listed like registration’s: datasets/screening_segmentation.json (1,080 datasets) and datasets/screening_synthesis.json (278), and the SynthRAD sets of the corpus are synthesis data already.

What an agent can do#

  • List the problems: the tracks, their modalities and regions, what each is scored against.

  • Get the development pairs: images and truth, open, to iterate on freely.

  • Check its convention: the field convention check (fidopt check), the most common silent error.

  • Be scored: on development pairs as often as it likes, on the hidden test pairs a limited number of times, so the test cannot be learned by trial and error.

  • See where it stands: its numbers next to the baselines’, with time and memory when the service ran it.

The skills are a thin layer over what exists: the scorer, the run.sh contract, the convention check and the boards.

The rule that makes it trustworthy#

Development pairs are open; test truth never leaves the server. Today the test truth sits next to the images in the private copy. Before the service opens, the test truth moves to the server that scores, and test submissions are capped and logged. Every number stays traceable to a method version, a machine and the corpus checksums.

Order of work#

  1. Registration baselines complete. Every baseline on every dataset, and IMPACT-Reg’s presets tuned on the development pairs only.

  2. A task-agnostic core. A descriptor says its task; the scorer, the run contract and the result rows are chosen by task. Registration keeps working unchanged.

  3. Scoring as a service. Development scoring runs anywhere (the library, fidopt score). Test scoring runs on our server, which holds the test truth, caps submissions and records each one.

  4. The skills. Over steps 2 and 3: wrap a method, verify it, develop it on the development pairs, submit it to the test.

  5. Segmentation. Datasets from the screening with labels drawn by people, the Metrics Reloaded measures, baselines such as nnU-Net, TotalSegmentator and MedSAM run under the same contract.

  6. Synthesis. The SynthRAD sets and the paired MR/CT and CBCT/CT datasets of the corpus, image measures plus measures computed on the synthetic image, baselines run under the same contract.

  7. Compute on demand. People and agents bring a method or data and have it run for them (RunPod), and the run contract maps onto Nextflow so existing pipelines plug in.

Decisions still open#

  • Where the server that holds the test truth runs, and how test submissions are capped.

  • Which datasets a paid service may use: only those whose licence allows commercial use ("use": "open").

  • When the corpus and the boards become public.

  • How 2D, 2D/3D, video and very large images enter (a 2D builder, a pose error, point tracking, chunked reading).