Details of the Segmentation Challenge

As part of this workshop, there are three datasets with two sub-challenges per dataset. For each dataset, we will provide training data (atlas image and labels) as well as a collection of testing images. Within each dataset there will be two sub-challenges.

  1. Segment all testing images using whatever method you want (i.e., use your favorite pre-processing, registration, label fusion, and post-processing tools).
  2. Segment all testing images using provided (non-rigid) registrations. This sub-challenge will provide important context on the quality of label fusion algorithms, as opposed to the quality of your analysis pipeline as a whole.

All results of the challenge will be entirely based on performance on the testing data.

Get the Data

You must register and agree to the license terms in order to download the data. Please use the form here. Do not send the direct link to anyone. Registration only takes 30 seconds, but it is vital to our record keeping and agreements with the data sponsors. You may register more than once. SATA Data Registration Form

Data Updates

  • 4/25/2013 All raw (as provided by sponsors) training and testing data
  • 6/3/2013 We are still working with our collaborators to provide the standardized registrations. All standardized registrations should be available within the next two weeks.
  • 6/13/2013 All data for the standardized registration challenges are now available.

Data Details

General Data Information

Name Contact # Training # Testing Acquisition Dimensions Resolution # Labels (no BG)
Diencephalon Challenge (Mid-brain) Neuromorphometrics, Inc. URL 35 12 MPRAGE 256x256x287 1x1x1mm 14 Protocol
Canine Leg Dr. Styner URL 22 23 T2w, T2FS 256x256x180x2 1x1x1mm 11 Protocol
Cardiac Atlas Project (CAP) Dr. Young URL CAP 83 72 CINE MRI ≈192x192x16x30 ≈1.5x1.5x6mm 1 Protocol

Standardized Registration Data Information

Details for the standardized registration data

Sub-Challenge #'s

Free Competition Standardized Registration
Diencephalon (Mid-brain) #1 #4
Canine Leg #2 #5
Cardiac Atlas Project (CAP) #3 #6

How to Enter

The submission site

  • All submissions will be processed through the submission site
  • Through the submission site you will need to provide the following information:
    • Team/Algorithm Name
    • E-Mail Address
    • Dataset you are processing (Canine Legs / Diencephalon / CAP)
    • Challenge Type (Free-for-all / Standardized Registration)
    • An archive file containing the estimated segmentations
  • Example results for each datatype can be found here.
    • Ignore the actual quantitative data, it is probably meaningless.
  • Once your submission is processed, the current leaderboard will be automatically updated.
  • The results from your submission will be e-mailed to the specified e-mail address.
    • Please allow for at least an hour before expecting your results.

Notes

  • Given the same team name, only the most recent submission will be considered.
    • If you are submitting multiple algorithms, or variations on multiple algorithms, then please use a unique team name for each submission.
  • The archive file must be in one of the following formats: .zip, .tar, .tar.gz, or .tar.bz2
  • All estimated segmentations must be in .nii(.gz) format.
  • The filenames for the estimated segmentations must be of the form:
    • For all Canine Leg data: DD_###*.nii(.gz)
    • For all Diencephalon data: ####_#*.nii(.gz)
    • For all CAP data: DET#######*.nii(.gz)
    • Where "#" is a number specific to a given testing image, and "*" can be anything you want.
  • For any additional comments/questions/concerns please e-mail miccai.multi.atlas@gmail.com

Anticipated Potential Problems

  • For the CAP data, all estimated segmentations MUST be in the same 4-D format as the original volumes.
    • See here for additional details on how to convert the 3-D data back out to 4-D.
  • If your results look very strange please verify:
    • The header information matches the provided testing images.
    • Your estimated segmentations are in .nii(.gz) format
    • You are following the required filenaming conventions.

Rules / Additional Details

These details apply for all 6 challenges (i.e. 3 datasets x 2 sub-challenges per dataset).

  • A given individual or group may submit multiple algorithms – each algorithm will be treated as an entirely unique entry.
  • A representative of the authors must agree to register for the MICCAI 2013 SATA workshop and present the entry either as a poster.
  • A unique result for a given algorithm may be submitted once a day for any given challenge – i.e., you can submit to each sub-challenge once a day. The submission number will be noted on the final report.
  • The submitter will hear back from the challenge committee within 5 business days of the submission via e-mail. Submitting early will lead to more opportunities for resubmission.
  • A leader board will be maintained for each of the 6 challenges and will be updated at least once per week.
    • Algorithms will be scored based on Dice Similarity Coefficient (DSC) and symmetrized Hausdorff distance (SHD).
  • The PC reserves to right to disqualify entries that do not address the challenge.

"Winning"

Everyone is a winner.

"Best" algorithms will be recognized based on the average DSC performance across the 3 datasets either on the 3 "free form" competitions or on the 3 "standardized registration" challenges. The (up to 6) algorithms that score the best will be invited to present in the oral portion of the program. To be considered for this award, the algorithm must be evaluated on all 3 datasets. Special awards / consideration may be also be considered by the program committee. These will be decided based upon the results.