Work Part 5: Evaluation

Work Part 5 - ``Evaluation'' is organized in the following way.

A short description of WP4 as a whole follows.

WP5 - Objectives

The objective of Work Part 5 is the investigation of the efficiency and effectiveness of the theories and of the related prototypical implementations developed within Work Parts 1 to 4. This is done both from a theoretical viewpoint, i.e. by studying the properties of the proposed theories in terms of computational complexity, and from an experimental viewpoint, i.e. by first devising and then running experiments that are adequate for understanding the effectiveness of information retrieval systems based on this logic.

WP5 - Approach

The computational complexity of the proposed theories will be studied by first relying on standard analysis techniques, and then, if negative results were obtained from these, by relying on techniques dealing with ``probabilistic'' computational complexity. In order to study the retrieval effectiveness of the prototypes resulting from Work Parts 1 to 4, instead, retrieval experiments will be performed and the results will be compared with those of similar approaches.

WP5 - Expected results

The computational properties of the theories resulting from Work Parts 1 to 3 and of the MIRLOG logic will have been characterized. Retrieval results will show the overall retrieval quality that can be achieved based on these theories and on the MIRLOG logic. In addition, these results will show strengths and weaknesses of our approach, thus indicating possibilities for further improvements.

Work Part 4 is further structured into Tasks T41 to T45. We now give a concise description of the objectives, approaches taken, and results expected from each of these tasks.

T51 - Objectives

The objective of T51 ``Computational studies'' is to assess the characteristics of the theories resulting from Work Parts 1 to 4 in terms of computational complexity.

T51 - Approach

The approach that will be followed in T51 will be to use the standard analysis techniques of computational complexity. In case that the results of this evaluation are negative (i.e. in case these theories turn out to have bad computational properties), techniques from probabilistic computational complexity will be used in order to try to assess whether these theories admit instead a tractable algorithm whose answers are correct only with probability close to 1. In such a case, given the imprecision already inherently present in the information retrieval endeavour, the imprecision introduced by a probabilistic algorithm would be negligible, and the problem would be deemed tractable.

T51 - Expected results

The result expected from T51 is a characterization of the computational properties of the the theories resulting from Work Parts 1 to 4, in deterministic terms and (if these were to be negative) in probabilistic terms. Results of this task will obviously influence the course of action that will be taken in the accomplishment of Work Part 4..

T52 - Objectives

In T52 ``Design of experiments'', retrieval experiments for evaluating the effectiveness of the theories resulting from Work Parts 1 to 4 are designed. This includes the experimental design as well as the setup of a prototypical system for performing the retrieval experiments.

T52 - Approach

We will use the routing queries from the TREC collection for setting up retrieval experiments. This means that besides the query formulation itself, a number of documents with relevance judgements are available for each query. Based on this data, query-specific knowledge bases can be set up. Other research groups with similar (advanced) representation techniques have taken the same approach. In addition, there are retrieval figures for a number of other approaches to which we can compare our final results. For the experiments, it has to be clearly defined which knowledge sources are used and what kinds of manual intervention is allowed. The prototypical system will be implemented by coupling the prototypes developed by the different groups with a search engine like e.g. SMART.

T52 - Expected results

The result of this task will be a precise specification of the experiments to be performed, and a prototypical IR system.

T53 - Objectives

In Task T53 ``Data acquisitions'', the data required for performing the retrieval experiments has to be collected.

T53 - Approach

Since the UNIDO-CS group is participating in the TREC initiative, the queries, documents and relevance judgements can be provided by this group. In addition, there are retrieval figures for a number of other approaches to which we can compare our results. Based on this material, the queries will be transformed manually into queries in terms of the specific theories to be evaluated. Furthermore, query-specific knowledge bases will be set up semi-automatically by considering the feedback documents.

T53 - Expected results

As a result of this task, a collection of experimental data is provided which also can be used by other research groups for performing similar experiments.

T54 - Objectives

In Task T54 ``Experiments'', the retrieval runs are performed in order to assess the effectiveness of retrieval based on the various theories to be evaluated.

T44 - Approach

By using the prototypical system set up in T42 and the data from T43, first retrieval runs are performed. Then effectiveness figures are computed by using standard evaluation measures.

T44 - Expected results

Retrieval output lists, overall and query-specific evaluation figures will be the results of this task.

T55 - Objectives

In the final task ``Analysis of the experimental results'', the results from T54 are analysed.

T55 - Approach

The overall figures show the overall quality of the various approaches tested, whereas query-specific comparisons with other approaches indicate specific strengths and weaknesses; furthermore, these results may point towards possible improvements.

T55 - Expected results

Besides figures for the overall quality of MIRLOG-based retrieval, we expect findings about its strengths and weaknesses and indications for further improvements.

The participating (P) consortium members for each of the Tasks in Work Part 4 are listed in the following table.