Participants: Femke Snelting, Manetta Berends, Cristina Cochior, Gijs de Heij, Natacha Roussel, Catherine Lenoble, Julie Boschat Thorez, Anne Laforet, Maxigas, Roel Roscam Abbing
The starting point of this group was the observation that the act of annotating the Gold standard data is one of the key elements for the construction of a supervised machine learning classifier like the ones present in the Pattern for Python package. While it is a key element and its process is long, soit la clef, le proces de l'annotation qui est long, tedious and manual, the act itself is not visible in the documentation of the package.
The participants decided to organise an annotation process for a paternalist classifier and to document it in detail : the date and place, basic information on the annotators, a manual for decision making during the annotation process and comments on the scores. They also decided to make visible the element of disagreement, a factor that is often ignored or Another element that is often ignored or amicably resolved in projects of economical nature.
The annotators realised that their criteria of judgement kept changing throughout the process, depending on the texts they were annotating. Some decided to go back to earlier annotated data to adapt their scores, others didn't. Another element of uncertainty was the concept's definition of paternalism. It is quite complicated to establish specific criteria and classes for paternalism. Because the group was composed of members of different ages and cultural backgrounds, discussions were at the centre of the reviewing part of the process. Each one made highly individual choices marking sentences with 1, 0 or -1. In the end they create da Python scipt that displays the disagreement in the analysis of their text corpus.