We have all been there: we read a paper that is promising the release of source code and data, we go on their webpage and we only find a
release date or we do not succeed to run the code. This is very frustrating, and raises the question how reproducible our results are.

Whilst papers can be reproducible without code release if the paper describes all components with enough detail, it is definitely much easier if the code is available. The promise to release code is typically appreciated by reviewers, but what does it look like in terms of accountability after?

Before we complain and make a point, we want to find out how big of a problem this actually is – together with you! Let’s focus on one of the strongest conferences in the field of computer vision, the Conference for Computer Vision and Pattern Recognition (CVPR). All papers since 2013 are available via the Computer Vision Foundation as open access and many of them promise code or directly include github links. Let’s investigate for this conference how easy it is to reproduce results.

For the moment, we would like to focus only on papers that indeed release code. We plan to do this project in stages:​

Stage 1 

A hackathon organized in a hybrid format with PhD students. The goal of this first stage is to figure out how easy or hard
it is to reproduce a single or multiple result figures from a paper if the code is available. This will give us a first impression how much time is
needed per paper and what the common pitfalls could be.  We believe that this project will be beneficial in multiple ways:

– Highlight strengths and weaknesses of recent papers in terms of reproducibility at a premier CV conference
– Identify best practices on how to ensure reproducibility; thereby also improving accountability and providing guidelines for reviewers on how to judge this.
– Give early career researchers & students opportunities to dive deep into different research topics, beyond just “theory”, and engage with a worldwide community.

And most importantly: it will be a lot of fun!

Who can participate?

You are invited to participate in stage 1 if you are a PhD student in a computer vision / machine learning related field (or if you can convince us that you have similar qualifications). PhD students from the EELISA partner universities of FAU are eligible to apply. For travel and accommodation funding, please contact your local EELISA project team.

REGISTER HREE

REGISTRATION DEADLINE 

October 31st

WHEN

December 8th to 10th

WHERE

Onsite at FAU and Online (a link will be sent to those elected to participate)