PhysioNet/CinC Challenges

Logo

Quick links for this year's Challenge:

Please post questions and comments in the forum. However, if your question reveals information about your entry, then please email challenge at physionet.org. We may post parts of our reply publicly if we feel that all Challengers should benefit from it. We will not answer emails about the Challenge to any other address.

Frequently Asked Questions (FAQ)

Can I attend this year’s conference remotely? Will I be eligble for prizes?

Yes, due to the unique circumstances of this year, remote attendance is allowed for CinC 2020. You are still eligible for prizes if you attend remotely (as long as you satisfy the other criteria).

Are the scores currently on the leaderboard the final scores for the Challenge?

No, the leaderboard contains scores on a subset of the test data during the unofficial and official phases of the Challenge. The final scores on the full test data are released after the conference for the “best” model selected by each team.

How will you choose which model to evaluate on the full test data? The latest entry? The best-scoring entry?

You will be able to choose which model you would like to have scored on the full test set. We will ask for teams to choose their “best” model shortly before the end of the official phase of the Challenge. If you do not choose a model, or if there is any ambiguity about your choice, then we will use the model with the highest score on the current subset of the test data.

Are we allowed to do transfer learning using pre-trained networks?

Yes, most certainly. We encourage you to do this. You do not need to include your data in the code stack for training the algorithm, but you do need to include the pre-trained model in the code and provide code to retrain (continue training) on the training data we provide. You must also thoroughly document the content of the database you used to pre-train.

Do we need to provide our training code?

Yes, this is a new and required (and exciting) part of this year’s Challenge.

Can I provide my training code but request that you not use/run it?

No, the training code is an important part of this year’s Challenge.

What computational resources do you provide for our code?

We run your training code on Google Cloud using 8 vCPUs, 64 GB RAM, and an optional NVIDIA T4 Tensor Core GPU. The training code has a 72 hour time limit.

We run each your trained model on Google Cloud using 2 vCPUs, 13 GB RAM, and an optional NVIDIA T4 Tensor Core GPU. Your trained model has a 24 hour time limit on the test set.

For the license, can we make it open source but restrict to non-commercial use?

Yes, the philosophy of the Challenge is to encourage researchers to make their code free to use for research. We hope that companies will approach you to license the code, too! If you do not specify any license, then we will assume that the license is the BSD 3-Clause License.

Are the provided records a more accurate representation of the hold-out evaluation data that what was previously provided?

We are creating a large database of heterogeneous data with varying labels, some of which are wrong or incomplete. Leads can be inverted, noisy, mislabeled. We have deliberately made no attempt to clean this up. The test data contains better labels, but it is not perfect either, and although it roughly correspond to the training data, it includes some deliberate differences.

Should I submit your example code to test the submission system?

No, please only submit your code to the submission system.

Should I submit an empty repository to test the submission system?

No, please only submit an entry after you have finished and tested your code.

I left out a file, or I missed the deadline, or something else. Can I email you my code?

No, please use the submission form to submit your entry through a repository.

Do I need to upload your training data? What about the code for evaluating my algorithm?

No, we have our training and test data and our evaluation code.

Do you run the code that was in my repository at the time of submission?

No, not yet. If you change your code after submitting, then we may or may not run the updated version of your code. If you want to update your code but do not want us to run the updates (yet), then please make changes in a subdirectory or in another branch of your repository.

Why is my entry unsuccessful on your submission system? It works on my computer.

If you used Python, then make sure that it runs in Docker.

My entry had some kind of error. Did I lose one of my total entries?

No, only scored entries (submitted entries that receive a score) count against the total number of allowed entries.


Back