Call for Labs proposals

Researchers and practitioners from all areas of information access and related communities are invited to submit proposals for evaluation labs for consideration for inclusion in CLEF 2015. Lab proposals will be reviewed by the lab selection committee. Authors of selected proposals will be invited to include their lab in the CLEF 2015 lab programme, possibly subject to small suggested modifications to their proposal to better suit the CLEF lab workflow or timeline.

Scope of CLEF Labs

Proposals are accepted for two different types of labs:

  • Evaluation Labs that are a "campaign-style" evaluation for specific information access problems (during the twelve month period preceding the conference), similar in nature to the traditional CLEF campaign "tracks" (see 2014 Labs at http://clef2014.clef-initiative.eu/ ). Topics covered by campaign-style labs can be inspired by any information access-related domain or task.
  • Labs that follow a more classical "workshop" pattern, exploring issues of evaluation methodology, metrics, processes etc. in information access and closely related fields, such as natural language processing, machine translation, and human-computer interaction.
If the organisers of the proposal are new to CLEF or other shared task evaluation campaigns, we highly recommend that a lab workshop first be organised to discuss the format, the problem space, and the practicalities of the shared task. The CLEF 2015 programme will reserve about half of the conference schedule for lab sessions which will take place within the conference in Toulouse. The labs will present their overall results "overview presentations" during the plenary scientific paper sessions to allow non-participants to get a sense of where the research frontiers are moving. Organisers of each lab are expected to organise the separate sessions for their lab at the conference so that they contain ample time for general discussion and engagement by all participants - not just those presenting campaign results and papers. Organisers should plan time in their sessions for activities such as panels, demos, poster sessions etc. as appropriate. CLEF is always interested in receiving and facilitating innovate and creative lab proposals. Potential task proposers who are unsure of the suitability of their task proposal or its format for inclusion at CLEF are encouraged to contact the CLEF 2015 Lab Organizing Committee Chairs to discuss its suitability or design at an early stage.

Proposal Submission

Lab proposals must provide sufficient information to enable to be able to judge the relevance, timeliness, scientific quality, benefits for the research community, and the competence of the proposers to co-ordinate the lab. Each lab proposal should identify one or more organizers as responsible for ensuring the timely execution of the lab.

Proposals should be 2-4 pages long and should provide the following information:

  1. Title of the proposed lab.
  2. A brief description of the lab topic and goals, its relevance to CLEF and the significance for the field.
  3. A brief but clear statement of usage scenarios or domain to which the activity is intended to contribute, including the evaluation setup and metrics.
  4. Details on the lab organizer(s), including identifying the task chair(s) responsible for the ensuring the running of the task. This should include details of any previous of involvement in organising or participating in evaluation tasks at CLEF or similar campaigns.
  5. The planned format of the lab, i.e. campaign-style ("track") or workshop.
  6. A statement on the intended development/growth path if the proposal is for a continuation of activities previously undertaken at CLEF workshops.
  7. Details of the expected target audience, i.e. who do you expect to participate in the task, and how do you propose to reach them.
  8. The number of tasks to be carried out in a lab. The proposal should clearly motivate the need for each of the proposed tasks and provide evidence of its capability of attracting enough participation. It is fine for a lab to have a single task, but labs often contain multiple closely related tasks.
  9. Expected length of the lab session at the conference: half-day, one day, two days. This should include very high level details of planned structure of the session, e.g. participant presentations, invited speaker(s), panels, etc, to justify the requested session length.
  10. Arrangements for the organization of the lab campaign: who will be responsible for activities within the task; how will data to acquired or created, including highlighting any IP issues that need to be addressed to enable data to be distributed to lab participants; tools or methods which will be used, e.g. how will necessary queries be created or relevance assessment carried out; any other information which is relevant to the conduct of your lab.
  11. If the lab proposes to set up a steering committee to oversee and advise its activities, include names, addresses, and homepage links of people your propose to be involved.

Reviewing Process

Each submitted proposal will be reviewed by the CLEF 2015 lab organizing committee. The decision will be sent by email to the responsible organizer by September 19, 2014. The final length of the lab session will be determined based on the overall organization of the conference and the number of participant submissions received by a lab.

Important Dates

  • Final lab proposals: 10 September 2014
  • Notification of lab acceptance: 19 September 2014
  • 3 November 2014: Labs registration opens

Submission Details

Lab proposals (or questions) should be submitted via e-mail (either plain text or PDF format, using "CLEF labs proposal" in the subject line) to both Lab Organizing Committee Chairs:

  • Eric San-Juan, eric.sanjuan[at]univ-avignon.fr
  • Gareth Jones, Gareth.Jones[at]computing.dcu.ie