Acknowledgments

The provided liver segmentations are part of the training data from the "VISCERAL Organ Segmentation and Landmark Detection Challenge" at the IEEE International Symposium on Biomedical Imaging (ISBI) in May 1st, 2014 in Beijing, China. For more information on the ongoing VISCERAL Anatomy Benchmark series, please visit www.visceral.eu (which should also be acknowledged by citation, footnote, or other suitable means in any publication using any part of its dataset).

Please reference the Virtual Skeleton Database in your work as:

View online | plain text | as END | as RIS | as BibTex | Mendeley | CiteULike

Shape 2015 Challenge - Rules

• All participants obtain a set of 47 training liver data as binary images. Additionally, a set of 10 triangle meshes of partial liver shapes is also provided.
• The goal of the challenge is to obtain the best possible reconstruction (shape completion) for the 10 given partial livers.
• The participants are free in the choice of learning method as well as the completion method.
• Participants are simply required to hand in the reconstructed 10 shapes, each aligned with its original incomplete mesh.
• The evaluation of each reconstruction quality is performed once the reconstruction is uploaded on the virtual skeleton database.
• All 10 reconstructions need to be uploaded to the virtual skeleton database, in order for a participation to be valid.

UPDATE: Challenge informations

The evaluation system for the Shape Symposium Challenge is now online.

Submissions

To submit your liver reconstructions, please follow the instructions on this page!

Below are a few important administrative information:

• The deadline for submitting your reconstructions to the challenge is September 23rd 23h59 CEST
• Multiple reconstruction uploads are possible

Shape Symposium

• At the deadline expiration, the 3 best performing submitters will be kindly requested to attend the symposium and give a 10 min presentation on their method.
• If you do not intend to attend the event, please do inform us early on such that we can select the presenters accordingly.
• With your reconstruction submission, you have the possibility to submit a one page abstract, similar to the shape symposium poster abstracts, where you detail you method. The procedure to submit the abstract will be detailed on : http://shapesymposium.org/
• The deadline for the abstract submission is : September 15th 2015.

Evaluation

The reconstruction will be evaluated according to the following criteria:

1. Average mesh distance to ground truth:

Description :

For every point of the first mesh, its distance to the closest point on the second triangle mesh is computed. The average over all points is returned. The average distance will be evaluated symmetrically, i.e. from the ground truth to the reconstruction, and from the reconstruction to the ground truth. The returned value is the average of the evaluation in both directions.

Goal:

The lower the value, the better the reconstruction.

2. Hausdorff mesh distance to ground truth

Description :

For every point of the reconstructed triangle mesh, its distance to the closest point on the ground truth mesh is computed. Also, for every point of the ground truth triangle mesh, its distance to the closest point on the reconstructed mesh is computed. The maximum over all values is returned.

Goal:

The lower the value, the better the reconstruction.

Both metrics are implemented as part of the Scalismo library. The source code is available here

No automatic rigid alignment is performed !

so please make sure that your reconstructions are correctly aligned with the provided incomplete data to obtain meaningful values.

Scoring

For each submitted reconstructed mesh, both metrics are evaluated : - Average mesh distance - Hausdorff mesh distance

ranking of the participant according to their performance (e.g. if there are 10 participants, the best performing algorithm gets 1 point and the worst performing algorithm gets 10 point). Each participant is therefore ranked 20 times. The final ranking will be the average of the individual rankings.

Upload

Reconstruction format:

Reconstructions are to be submitted as triangle meshes, in the STL file format

Reconstrution naming:

1. You have to name your results exactly like this template NAMESPACE.description.vsdid.stl: For case 79 use VSD.shape2015_case_079.87048.stl. Here's the list for all cases:

VSD.shape2015_case_079.87048.stl
VSD.shape2015_case_082.87049.stl
VSD.shape2015_case_085.87050.stl
VSD.shape2015_case_132.87041.stl
VSD.shape2015_case_133.87042.stl
VSD.shape2015_case_134.87043.stl
VSD.shape2015_case_135.87044.stl
VSD.shape2015_case_136.87045.stl
VSD.shape2015_case_337.87046.stl
VSD.shape2015_case_340.87047.stl

2. Upload your files using the Upload & Evaluation page

3. Multiple versions: To keep track ouf your versions, you can change the description part in the filename.

A 1st version:

VSD.case_079_v1.87048.stl

and a modified 2nd version:

VSD.case_079_v2.87048.stl

4. Additional information for the template definition

namespace 
    string
    required
    static
    value: VSD

description: 
    string
    optional 
    user defined 
    no special character except undersocores `_` or dashes `-`
    example: shape2015_case_079

vsdid (of the partials image)
    integer
    required
    static per case
    example: 07048

extension:
    string
    required
    static
    value: .stl

Evaluation Results: Challenge