While Cranfield Methodology seems to be very good to compare your system to what other researchers achieved, but I think it is very subjective. For example, following this evaluation you can find out that after you modified your retrieval system in some way it started to perform worse then before, or some other systems. The problem is that these changes you made could be a “Holy Grail” in IR from the end user point of view, but you will get rid of them without knowing about it, because automated evaluation showed that performance got worse. So I think that it is essential to also conduct evaluation by end users.
No comments:
Post a Comment