Criteria for Running a Successful Challenge

Each Challenge requires substantial time, effort, and expertise to run. A well run Challenge provides valuable research and training opportunities for the Challenge organizers and participants alike, while a poorly run Challenge can reflect poorly on our community and even cause harm.

We appreciate when we receive suggestions that we run a particular Challenge, and we want them to be aware of the requirements or running a Challenge. This list is not exhaustive, and we note that the some of the tasks in this list often take several months or even years to prepare:

  1. Challenge data
    1. Sufficient, high-qualify, real-world data
    2. Clinically meaningful labels
    3. Curated in collaboration with domain experts
    4. Public training set and hidden validation and test sets
    5. Clearly documented
    6. Appropriate and standard file formats
    7. IRB approved
    8. Fully deidentified
    9. Ethically sound
  2. Challenge objective
    1. Meaningful, clinically relevant, and relevant to PhysioNet
    2. Clearly defined and stated in collaboration with domain experts
  3. Challenge evaluation metrics
    1. Clinically relevant
    2. Capture the Challenge objective
    3. Clearly defined and stated in collaboration with clinical domain experts
  4. Challenge algorithms
    1. Publicly available example implementations available in common languages with clear documentation and instructions
    2. Flexible to encourage a diversity of approaches
    3. Restrictive to ensure that algorithms address the defined problem
  5. Challenge timeline
    1. Challenge data, objective, evaluation metrics, and algorithms must be available from the beginning of the Challenge
    2. Run time must be at least 3 months, with at least 6 months preferred
  6. Challenge deliverables
    1. Clear definition of scientific outputs
    2. Generate publicly available software
    3. Archive both public training data and hidden validation and test data at PhysioNet
    4. Publications and code links provided to PhysioNet after the Challenge
    5. Code for Challenge evaluation metrics must be submitted to PhysioNet
    6. Affiliation with a public scientific event recommended
  7. Challenge organizers
    1. PI(s) explicitly assume responsibility of Challenge, including the throughout and after the Challenge
    2. Sufficient and responsive staff
    3. Sufficient computational resources for at least 100 teams and 1000 code submissions
    4. Successful completion of a previous data science competition

Supported by the National Institute of Biomedical Imaging and Bioengineering (NIBIB) under NIH grant number R01EB030362.

© PhysioNet Challenges. Website content licensed under the Creative Commons Attribution 4.0 International Public License.

Back