Zoom Link

13:00 - 13:15 Opening and Welcome

13:15 - 14:45 Black Mirror: BHCC Edition

14:45 - 15:00 Break

15:00 - 16:00 Paper Session I

1. We Don't Need No Expectations: On the Observer-expectancy Effect in Crowdsourcing, Eddy Maddalena, Alessandro Checco, and Elena Simperl

2. Designing Human-in-the-loop Approaches for Closed Deferral Pipelines, Vijay Keswani, Matthew Lease, and Krishnaram Kenthapadi

3. Detecting Bias in Complex Annotations, Alexander Braylan and Matthew Lease

4. Investigating Biases in Crowdsourced Hate Speech Annotation, Danula Hettiachchi, Indigo Holcombe-James, and Anjalee de Silva

16:00 - 16:45 Invited Talk: Ming Yin

17:00 - 18:00 Panel Discussion: Open Science & Good Research Practice

(Samantha Teplitzky, Denny Borsboom, Pietro Michelucci, Michael Hind)

Zoom Link

13:00 - 14:00 Meet & Geet While You Eat – Bring Your Own Lunch!

14:10 - 15:15 Paper Session II

1. The Life Course: An Interdisciplinary Framework for Broadening the Scope of Research on Crowdwork, Anoush Margaryan

2. A Practical Framework for Building and Managing Digital Crowdsourcing Communities, Wuraola Fisayo Oyewusi, Olubayo Adekanmbi, Olawale Victor Abimbola, Chinazo Anebelundu, Oluwaseun Abel Ofinni, and Sandra Oriji

3. A Quantum Approach to Crowdsourcing, Kevin Roitero

4. The Effect of Individual Cognitive Style when Mitigating Confirmation Bias during Search, Alisa Rieger and Nava Tintarev

15:15 - 15:30 Break

15:30 - 16:30 Invited Talk: Saiph Savage

16:30 - 16:45 Break

16:45 - 17:00 Synthesis of the Crowd Bias Workshop at CSCW 2021 - Danula Hettiachchi

17:00 - 18:00 Delft (Virtual) City Tour

Zoom Link

14:00 - 14:10 Welcome

14:10 - 15:15 Paper Session III

1. Preserving the memory of the first wave of COVID-19 pandemic: Crowdsourcing a collection of image search queries, Kalia Orphanou, Evgenia Christoforou, Jahna Otterbacher, Monica Lestari Paramita, and Frank Hopfgartner

2. Can Survey Methodology Improve Machine Learning? A Proposed Pilot Study Examining the Effect of Stratified Sampling on Color Naming Model Accuracy and Diversity, Anurata Hridi, Johanne Christensen, and Benjamin Watson

3. Bias in Computer Vision Algorithms for Natural Disaster Damage Assessment, Thomas Chen

4. Efficacy of Bengali Captioning Guideline in TV programs, Tanvir Ahmed and Akhter Al Amin

5. Mandating model cards and metrics before allowing to fork (pre-trained model) repos on Github, Sundaraparipurnan Narayanan

15:15 - 15:30 Break

15:30 - 15:45 Drive Business Success with Crowdsourcing-Practical Use Cases:Toloka

15:45 - 16:45 Invited Talk: Alexandra Olteanu

16:45 - 17:00 Break

17:00 - 18:00 BHCC Meme Making