To Phrase or Not to Phrase – Impact of User versus System Term Dependence upon Retrieval

Christina Lioma, Birger Larsen, Peter Ingwersen

Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

10 Downloads (Pure)

Resumé

When submitting queries to information retrieval (IR) systems, users often have the option of specifying which, if any, of the query terms are heavily dependent on each other and should be treated as a fixed phrase, for instance by placing them between quotes.In addition to such cases where users specify term dependence, automatic ways also exist for IR systems to detect dependent terms in queries. Most IR systems use both user and algorithmic approaches. It is not however clear whether and to what extent user-defined term dependence agrees with algorithmic estimates of term dependence, nor which of the two may fetch higher performance gains. Simply put, is it better to trust users or the system to detect term dependence in queries? To answer this question, we experiment with 101 crowdsourced search engine users and 334 queries (52 train and 282 test TREC queries) and we record 10 assessments per query. We find that (i) user assessments of term dependence differ significantly from algorithmic assessments of term dependence (their overlap is approximately 30%); (ii) there is little agreement among users about term dependence in queries, and this disagreement increases as queries become longer; (iii) the potential retrieval gain that can be fetched by treating term dependence (both user- and system-defined) over a bag of words baseline is reserved to a small subset (approximately 8%) of the queries, and is much higher for low-depth than deep precision measures. Points (ii) and (iii) constitute novel insights into term dependence.
OriginalsprogEngelsk
TidsskriftJournal of Data and Information Management
Vol/bind2
Udgave nummer1
Sider (fra-til)1-14
Antal sider14
ISSN2543-9251
DOI
StatusUdgivet - 1 jun. 2018

Fingerprint

Information retrieval systems
Search engines
Experiments

Citer dette

@article{7caab187d09047fbbcc0742e32d2d931,
title = "To Phrase or Not to Phrase – Impact of User versus System Term Dependence upon Retrieval",
abstract = "When submitting queries to information retrieval (IR) systems, users often have the option of specifying which, if any, of the query terms are heavily dependent on each other and should be treated as a fixed phrase, for instance by placing them between quotes.In addition to such cases where users specify term dependence, automatic ways also exist for IR systems to detect dependent terms in queries. Most IR systems use both user and algorithmic approaches. It is not however clear whether and to what extent user-defined term dependence agrees with algorithmic estimates of term dependence, nor which of the two may fetch higher performance gains. Simply put, is it better to trust users or the system to detect term dependence in queries? To answer this question, we experiment with 101 crowdsourced search engine users and 334 queries (52 train and 282 test TREC queries) and we record 10 assessments per query. We find that (i) user assessments of term dependence differ significantly from algorithmic assessments of term dependence (their overlap is approximately 30{\%}); (ii) there is little agreement among users about term dependence in queries, and this disagreement increases as queries become longer; (iii) the potential retrieval gain that can be fetched by treating term dependence (both user- and system-defined) over a bag of words baseline is reserved to a small subset (approximately 8{\%}) of the queries, and is much higher for low-depth than deep precision measures. Points (ii) and (iii) constitute novel insights into term dependence.",
keywords = "information retrieval, phrase search, term dependence",
author = "Christina Lioma and Birger Larsen and Peter Ingwersen",
year = "2018",
month = "6",
day = "1",
doi = "10.2478/dim-2018-0001",
language = "English",
volume = "2",
pages = "1--14",
journal = "Journal of Data and Information Management",
issn = "2543-9251",
publisher = "De Gruyter",
number = "1",

}

To Phrase or Not to Phrase – Impact of User versus System Term Dependence upon Retrieval. / Lioma, Christina; Larsen, Birger; Ingwersen, Peter.

I: Journal of Data and Information Management, Bind 2, Nr. 1, 01.06.2018, s. 1-14.

Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

TY - JOUR

T1 - To Phrase or Not to Phrase – Impact of User versus System Term Dependence upon Retrieval

AU - Lioma, Christina

AU - Larsen, Birger

AU - Ingwersen, Peter

PY - 2018/6/1

Y1 - 2018/6/1

N2 - When submitting queries to information retrieval (IR) systems, users often have the option of specifying which, if any, of the query terms are heavily dependent on each other and should be treated as a fixed phrase, for instance by placing them between quotes.In addition to such cases where users specify term dependence, automatic ways also exist for IR systems to detect dependent terms in queries. Most IR systems use both user and algorithmic approaches. It is not however clear whether and to what extent user-defined term dependence agrees with algorithmic estimates of term dependence, nor which of the two may fetch higher performance gains. Simply put, is it better to trust users or the system to detect term dependence in queries? To answer this question, we experiment with 101 crowdsourced search engine users and 334 queries (52 train and 282 test TREC queries) and we record 10 assessments per query. We find that (i) user assessments of term dependence differ significantly from algorithmic assessments of term dependence (their overlap is approximately 30%); (ii) there is little agreement among users about term dependence in queries, and this disagreement increases as queries become longer; (iii) the potential retrieval gain that can be fetched by treating term dependence (both user- and system-defined) over a bag of words baseline is reserved to a small subset (approximately 8%) of the queries, and is much higher for low-depth than deep precision measures. Points (ii) and (iii) constitute novel insights into term dependence.

AB - When submitting queries to information retrieval (IR) systems, users often have the option of specifying which, if any, of the query terms are heavily dependent on each other and should be treated as a fixed phrase, for instance by placing them between quotes.In addition to such cases where users specify term dependence, automatic ways also exist for IR systems to detect dependent terms in queries. Most IR systems use both user and algorithmic approaches. It is not however clear whether and to what extent user-defined term dependence agrees with algorithmic estimates of term dependence, nor which of the two may fetch higher performance gains. Simply put, is it better to trust users or the system to detect term dependence in queries? To answer this question, we experiment with 101 crowdsourced search engine users and 334 queries (52 train and 282 test TREC queries) and we record 10 assessments per query. We find that (i) user assessments of term dependence differ significantly from algorithmic assessments of term dependence (their overlap is approximately 30%); (ii) there is little agreement among users about term dependence in queries, and this disagreement increases as queries become longer; (iii) the potential retrieval gain that can be fetched by treating term dependence (both user- and system-defined) over a bag of words baseline is reserved to a small subset (approximately 8%) of the queries, and is much higher for low-depth than deep precision measures. Points (ii) and (iii) constitute novel insights into term dependence.

KW - information retrieval

KW - phrase search

KW - term dependence

U2 - 10.2478/dim-2018-0001

DO - 10.2478/dim-2018-0001

M3 - Journal article

VL - 2

SP - 1

EP - 14

JO - Journal of Data and Information Management

JF - Journal of Data and Information Management

SN - 2543-9251

IS - 1

ER -