Microsoft word - intentionalqueryexpansion_0501_cr-copy_changes_off_ms.doc

Intentional Query Suggestion:
Making User Goals More Explicit During Search
ABSTRACT
~25% of queries have a clear navigational intent, and up to ~75% The degree to which users' make their search intent explicit can of queries need to be understood as informational or transactional be assumed to represent an upper bound on the level of service queries, meaning they are not directed towards a specific set of that search engines can provide. In a departure from traditional expected documents. Recent studies even estimate more drastic query expansion mechanisms, we introduce ratios [13]. While users crafting informational or transactional search queries often have a high level search intent ("plan a trip to Suggestion as a novel idea that is attempting to make users' intent Europe"), in many situations they have no clear idea or knowledge more explicit during search. In this paper, we present a prototypical algorithm for Intentional Query Suggestion and we about the specific documents they expect to retrieve. This makes discuss corresponding data from comparative experiments with it difficult for users to craft successful queries and makes query traditional query suggestion mechanisms. Our preliminary results suggestion a particularly important and challenging problem. indicate that intentional query suggestions 1) diversify search In this paper we are interested in exploring the following question: result sets (i.e. it reduces result set overlap) and 2) have the What if search engines would, rather than letting users guess potential to yield higher click-through rates than traditional query arbitrary words from the set of documents they are expected to retrieve, encourage users to tell them their original search intent in a more unambiguous and natural way? In other words, what if Categories and Subject Descriptors
search engines would encourage users to make their search intent H.1.2 [User/Machine Systems]: Human Factors; H.3.3
more explicit (e.g. "buy a car") rather than formulating their query [Information Storage and Retrieval]: Query Formulation,
in a rather artificial manner ("car dealership")? In future search interfaces (such as audio search interfaces for cell phones or natural language search interfaces), current mechanisms for query General Terms
suggestion might become inadequate and natural language search Algorithms, Human Factors, Experimentation queries might play a more important role. This work is interested Keywords
in understanding how current search methods would cope with such a development. For this purpose, we introduce and study a novel approach to query suggestion: Intentional Query Suggestion or query suggestion by user intent. While traditional query suggestion often 1. INTRODUCTION
aims to make a query resemble more closely the documents a user In IR literature, the purpose of query suggestion has often been is expected to retrieve (which might be unknown to the user), we described as the process of making a user query resemble more want to study an alternative: expanding queries to make searchers' closely the documents it is expected to retrieve ([26]). In other words, the goal of query suggestion is commonly understood as maximizing the similarity between query terms and expected To give an example: In traditional query suggestion, a query "car" documents. The task of a searcher then is to envision the expected might receive the following suggestions: "car rental", "car documents, and craft queries that reflect their contents. insurance", "enterprise car rental", "car games" (actual suggestions produced by Yahoo.com on Nov 27th 2008). In query suggestion However, research on query log analysis suggests that many based on explicit user intent, the suggestions could be "buy a car", queries exhibit a lack of user understanding about the specific "rent a car", "sel your car", "repair your car" (see Table 1 for documents users expect to retrieve. Broder [6] found that only examples). We can speculate that in innovative search interfaces (such as audio search interfaces), such suggestions would be easier to verify with a user than verifying traditional query Permission to make digital or hard copies of all or part of this work for suggestions (e.g. "Do you want to: buy a car OR sell a car OR …?"). personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. WSCD'09 at WSDM'09, February 9, 2009, Barcelona, Spain. Copyright 2009 ACM 1-58113-000-0/00/0004…$5.00. Table 1: Comparison of suggested queries provided by
query and expand it based on a better understanding of the Yahoo!, MSN and Intentional Query Suggestion.
corresponding information need - thereby aiming to make user intent more explicit. We define Intentional Query Suggestion as the incremental process of transforming a query into a new query based on intentional structures found in a given domain, in our case: a search query log. An initial query is replaced by the most probable intentions that underlay the query. To give an example: for the query "playground mat", an Intentional Query Suggestion mechanism might suggest the following 5 user intentions: "buy full tilt poker, free learn to play poker, playground equipment", "build a swing set", "covering dirt in a of poker, party poker games, free buy poker table, playground", "buy children plastic slides", "raise money for our In our case, we extract the proposed intentions from search query logs, but they could potentially be extracted from other knowledge house plans, white house TV show, insure my house, bases containing common human goals as well, such as house, house of houses for sale, sell your house, ConceptNet [19] or others. To the best of our knowledge, the fraser, columbia houses for rent, make offer on house, house of house plans, house house, buy house application of explicit search intent [24] to query suggestion represents a novel idea that has not been studied yet. 3. EXPERIMENTAL SETUP
We are interested in studying the effects of this idea on the search In traditional query suggestion, an initial query formulation is result sets obtained from experiments with a current search engine replaced by some other query that refines, disambiguates or provider. In particular, we are interested in seeking answers to the clarifies the original query. In our approach, the initial query is following questions: How do today's search engines deal with replaced by a query that exhibits a higher degree of intentional queries that contain explicit user goals? How would queries explicitness, meaning that it makes user intent more explicit [24]. expanded by user intent influence search results and click Definition: We define this replacement as query suggestion based
on user intent. The suggested queries can be considered to This paper introduces Intentional Query Suggestion as a novel represent Intentional Query Suggestions whenever they 1) contain type of query suggestion. Specifically, this paper 1) introduces a at least one verb and 2) describe a plausible state of affairs that definition of Intentional Query Suggestion 2) presents a the user may want to achieve or avoid (cf.) in 3) a recognizable preliminary algorithm to perform intentional query suggestion based on historic query log data and 3) discusses experimental We developed a parametric algorithm that executes the function results and potential implications for future research on search f(q) → RQE = {qe,1, qe,2 … qe,k}, mapping implicit intentional queries (length <= 2) to a set of potential explicit intentional query suggestions (e.g. "car" → "buy a car", "rent a car", "repair my car"). 2. QUERY SUGGESTION
The general idea of query suggestion is to support the searcher in
3.1 Datasets
formulating queries that have a better chance to retrieve relevant The MSN Search query log excerpt contains about 15 million documents [21], [3]. Methods offered to expand queries can be queries (from US users) that were sampled over one month in divided into two major categories. Global methods employ entire May, 2006. The search query log data is split into two files, one document collections or external sources such as thesauri as file containing attributes Time, Query, QueryID and ResultCount, the corpora for producing suggestions. Local methods reformulate the other one attributes QueryID, Query, Time, URL and Position initial query based on the result set it has retrieved. Relevance providing click-through data. The queries were modified via the feedback represents another query reformulation strategy in which following normalization steps (i) trimming of each query, and (ii) a searcher is involved by marking retrieved documents as relevant space sequence reduction to one space character. Queries and or not. Global as well as local methods aim to eventually move corresponding click-through data containing adult content were the initial query closer to the entire cluster of relevant documents. filtered out (and were not taken into account in our study). A set of ~46.000 explicit intentional queries was extracted from the MSN Search Asset Data Spring 2006 applying the algorithm 2.1 Intentional Query Suggestion
described in [24]. The resulting set has an estimated precision of While traditional query suggestion techniques aim at narrowing 77% of explicit intentional queries (based on the evaluations the gap between the initial query and the set of relevant reported in [25]) and represents our knowledge base for documents, we seek to approximate the user's intentions behind a Intentional Query Suggestion. We call this subset of queries the Explicit Intentional Query Dataset from here on. 1 Related query suggestion results from Yahoo! Our parametric algorithm for Intentional Query Suggestion approximates the searcher's intent by combining two different yet 2 Related query suggestion results from MSN complementary approaches, i.e. text-based Intentional Query Suggestion (see Section 3.2) and neighborhood-based Intentional We define the neighborhood of an explicit intentional query q as Query Suggestion (see Section 3.3). The two approaches can be N(qe, Pd), where the parameter Pd determines the number of combined yielding a ranked list of potential intentional query queries that are considered before and after the query q . The neighborhood N(qe, Pd) contains 2 * Pd queries where q ϵ QU holds. Queries qi ϵ N(qe, Pd) are processed to serve as tags (dimensions of the characteristic vector describing explicit 3.2 Text-Based Intentional Query Suggestion
intentional queries) for the corresponding intentional query q . In the text-based approach, the tokens of input queries are After stop words have been removed, the remaining tokens are textually compared to all query tokens in the Explicit Intentional combined into a set of words and form a tag set T(qe)={t1, t2 … tm} Query Dataset. We experimented with several text-based of the explicit intentional query q . In addition to parameter P similarity measures including Cosine Similarity, Dice Similarity, introduce the parameter Pi that denotes the intersection size Jaccard Similarity and Overlap Similarity [11], [3]. Because the between explicit intentional queries and neighboring queries. This similarity measures did not exhibit significant differences, we parameter can be considered as a quality filter. Tokens of one decided on using Jaccard Similarity throughout our experiments query are only admitted to the tag set T(qe) if the query shares at for reasons of simplicity. In text-based intentional query least Pi tokens with qe. Let qe be "lose weight fast", qu be "weight suggestion, we calculate Jaccard Similarity in the following way: loss supplements" and Pi = 1: qe and qu share one common term ("weight"). Consequently, the tokens of q qe, i.e. T(qe) = {"weight", "loss", "supplement"}. We suspect this parameter to be related to the quality of the tags admitted to the tag set and consequently related to the quality of the entire model. A and qB are the respective token sets representing two This yields a characteristic vector of tags for each explicit intentional query based on session-neighborhood. Figure 1 shows a bipartite graph that was partly generated from 3.3 Neighborhood-Based Intentional Query
the query log excerpt in Table 2 with a parameter setting Pd = 3 Suggestion
and Pi = 1. The graph illustrates relations between explicit In addition to Intentional Query Suggestion based on text, we are intentional queries and meaningful terms in the session using a similarity construct based on query log session neighborhood, representing characteristic term vectors for explicit neighborhood. This has the potential to include behavioural intentional queries. The example also shows that the intentional structures in our algorithm. For that purpose, we are neighborhood-based approach is agnostic to misspellings. The conceptualizing query logs as consisting of two types of nodes (a bipartite graph is useful in at least two ways: Bottom-up, it can bipartite graph), where nodes of one type correspond to help to produce intentional query suggestions based on co- intentional queries and nodes of the other type correspond to occurrence (e.g. "upplements" → "lose weight fast"). Top-down, the graph can help to transform explicit intentional queries into implicit intentional queries. We construct a bipartite graph based on session proximity between these two types of nodes. Thereby, implicit ones (which is not further pursued in this paper). Note we use neighboring queries to further describe and characterize that qu,3 and qu,4 both represent explicit intentional queries and are explicit intentional queries, building characteristic term vectors therefore neglected in the graph generation process. for explicit intentional queries. In the following, we introduce the parametric algorithm for intentional query expansion in a more formal way. Table 2: Search query log excerpt illustrating the explicit
intentional query qe,1 and its neighborhood N(qe,1, 3).
Figure 1: Bipartite graph partly generated from search query
log excerpt in Table 2 with parameter setting P
d=3 and Pi=1.
Similarity between an input query ("upplements") and a number of explicit intentional queries ("lose weight fast") can now be calculated with traditional similarity metrics. Again, we experimented with different similarity measures and opted for the Jaccard similarity measure due to insignificant differences between the measures. In neighbourhood-based intentional query suggestion, we calculate Jaccard similarity in the following way: T (q ) ∩ T (q ) T (q ) ∪ T (q ) 1, q2 … qn} denote the set of n queries in a search query log. Q consists of two disjoint sets QE={qe,1, qe,2 … qe,s} and where T(qA) and T(qB) are the respective token sets representing U={qu,1, qu,2 … qu,t } so that Q = QE ∪ QU represents the set of explicit intentional queries, such as "lose weight fast", and QU the neighboring implicit intentional queries such as "weight loss supplements" as illustrated in Table 2. 3.4 Query Suggestion based on User Intent
suggestions, i.e. query suggestions that were assigned to relevance When input queries are processed by our algorithm, both class 1. Achieved precision values are illustrated in Table 3. similarity measures are calculated. In our approach, a linear Table 3: Precision values of our algorithm as rated by three
combination determines the overall similarity between an input human annotators (X, Y and Z).
query and every explicit intentional query in our dataset yielding a ranked list of potential user intentions. The parameter α defines S (q , q ) = α * S (q , q ) + 1( − α ) * S (q , q ) The average precision amounts to 0.71, i.e. in seven out of ten cases the algorithm returns a potential user intention. In this work we do not intend to identify an optimized parameter set to generate the model. We rather chose a simple parameter set In addition, we calculated the inter-rater agreement κ [8] between for the purpose of seeking answers to the exploratory questions of all pairs of human subjects X, Y, and Z. Cohen's κ measures the this paper. Future work might explore the utility of parameter average pair-wise agreement corrected for chance agreement when classifying N items into C mutually exclusive categories. Cohen's κ formula reads: The parametric algorithm for Intentional Query Suggestion can be P(O) − P(C) d = 3, Pi = 1 and α = 0.5 in our experiments. where P(O) is the proportion of times that a hypothesis agrees An evaluation of the selected model is provided in Section 3.5. with a standard (or another rater), and P(C) is the proportion of times that a hypothesis and a standard would be expected to agree by chance. The κ value is constrained to the interval [-1,1]. A κ- 3.5 Evaluation
value of 1 indicates total agreement, 0 indicates agreement by We conducted a user study to learn more about the quality of chance and -1 indicates total disagreement. Table 4 shows the intentions that were suggested by our algorithm. Annotators were achieved κ-values in our human subject study. asked to categorize the 10 top-ranked suggested explicit intentional queries for 30 queries into one of the following two Table 4: Kappa values amongst three annotators (X, Y and Z)
for the two relevance classes.
Relevance Classes:
(1) Potential User Intention: the suggested query represents
Cohen's Kappa (κ) 0.6416 0.5125 0.6703 a plausible intention behind a short query. The κ-values (see Table 4) range from 0.51 to 0.67 (0.61 on "anime" "draw anime", "draw manga" average) containing two values above 0.6 indicating some level of "playground mat" "buy playground equipment", "build a swing set" or the suggested query represents an unlikely yet still related user intention as illustrated by following examples: 4. PRELIMINARY RESULTS
In this section we discuss two potential implications of Intentional Query Suggestion for web search: First, diversity of search results "Boston herald" "getting around Boston", "sightseeing in has recently gained importance in web search [9]. For example in "ginseng coffee" "moving coffee stains", "fix my keyboard" informational queries, web search results should not provide monolithic search result sets but rather cover as many different aspects (topics) as possible. We are interested in exploring the (2) Clear Misinterpretation: the suggested query has no
influence of explicit intentional queries on the diversity of search relation with the initial query. Suggestions that do not result sets. If result sets of explicit intentional queries would be conform to our definition (see Section 3) are assigned this more diverse, Intentional Query Suggestion could help to better focus and guide searchers' intent in exploratory searches. Second, click through rates have been frequently used as a proxy "Boston herald" "care for Boston fern", "flying to Nantucket" for measuring relevance in large document collections (cf. [10]). "playground mat" "raise money for our playground", "weave a We are interested in studying whether explicit intentional queries would yield other/better click-through rates than implicit intentional queries. If explicit intentional queries would yield higher click-through rates, making user intent more explicit would 30 queries of length 1 or 2 were randomly drawn from the MSN represent an interesting new mechanism to improve search engine search query log. The prospective queries were filtered with regard to (i) reasonableness, i.e. discarding queries such as "wiseco" or "drinkingmate" and to (ii) non American raters, i.e. 4.1 Influence on Diversity of Search Results
discarding queries such as "target" or "espn". We examine the diversity within search results by calculating the In order to evaluate intentional query suggestions that are intersection size between different URL result sets produced by provided by our algorithm, we calculated the percentage of correct different/same query suggestion mechanisms. Two experiments were conducted, seeking answers to the following questions: Table 6: Average intersection sizes for URL sets expanded by
(i) Intersection between different Query Suggestion Yahoo! Suggestions and Intentional Query Suggestion.
Mechanisms: How many URLs (top level domains only) intersect between URL result sets retrieved by 1) the Compared URL result sets
Average Intersection
original queries, 2) the corresponding Yahoo! expanded queries and 3) the corresponding intentional query suggestions? (ii) Intersection within same Query Suggestion Mechanisms: How many URLs (top level domains only) intersect between result sets that were retrieved by Considering the presented results, we can speculate that search different query suggestions (produced by the same processes could be made more focused if the searchers' intention query suggestion mechanism) regarding one original is explicitly included in the search process. It appears that intentional query suggestions diversify search results and cover a wider range of topics than Yahoo!'s suggestions. 400 queries of length 1 or 2 were randomly drawn from the MSR search query log. Following constraints were made: original queries (i) should yield at least 10 suggestions by our algorithm, 4.2 Influence on Click-Through
(ii) should not contain misspellings and (iii) must not be 'adult' To study the influence of explicit intentional queries on click phrases. For each selected query, the top 10 suggestions were through, we analyzed the number of click-through events for produced by using the Yahoo! API and by the Intentional Query different token lengths. We obtained the click-through numbers Suggestion algorithm. We processed the top 50 result URLs for for different token lengths in the MSR query dataset and created each suggestion, totalling 500 URLs per selected query. Searches the following token length bins: one token queries, two token were conducted by applying the Yahoo! BOSS API3. In order to queries, three to four token queries, five token queries, six to ten compare the original query results with both expanded results sets, token queries and queries consisting of more than ten tokens 500 resulting URLs are retrieved for every original query. For (excluding explicit intentional queries). Five token queries were each query, we calculated how many URLs are shared on average of particular interest, since the average length of queries in our between the URL result sets taking into account only unique Explicit Intentional Query Dataset amounts to 5.33 tokens. For URLs as well as only top level domains of the resulting set. each category, a random sample of 5,000 queries was drawn from Again, we used Jaccard as a metric for intersection/similarity. The the MSN search query log and all corresponding click-through averaged results over all candidate queries are shown in Table 5. events were registered and counted. Table 7 shows the number of Table 5: Average intersection sizes for URL sets of original
click through events for each bin and also for the set of explicit queries and their corresponding suggestions.
Compared URL result sets
Avg.Inter-
Table 7: Click-through distribution for different query
lengths and explicit intentional queries
Original Queries vs. Intentional Suggestions Yahoo! Suggestions vs. Intentional Suggestions 855,649 358,327 64,313 5,559 2,728 960 7,236 The results in Table 5 imply that original query results share more URLs with results from Yahoo! expanded queries than with It can be observed that explicit intentional queries appear to have results yielded by queries that reflect potential user intent. This a ~ 30% higher number of click through events (#click-through = suggests that if queries are expanded by user intent more diverse 7,236) than implicit intentional queries of comparable length result sets can be achieved. In addition, we calculated the inner (length 5, #click-through = 5,559). The higher click-through intersection size of the result sets, i.e. the overlap between numbers of explicit intentional queries suggest that such queries different result sets produced by the same suggestion mechanism. retrieve more relevant results, which appears to be an interesting The results were again averaged over all queries and are shown in finding and preliminary evidence for the potential utility of The results in Table 6 suggest that queries expanded by Yahoo! yield more overlapping URLs than queries expanded by user 5. RELATED WORK
intent. These results suggest that queries that express a specific Two areas of research are particularly relevant to our work: intention lead to more diverse results than queries that attempt to Studies of search intent in query logs and query suggestion. approximate the expected document content to retrieve. Studies of search intent in query logs: Peter Norvig discussed4 search intent as one of the outstanding problems in the future of search. One interpretation of understanding the users' needs is to 3 http://developer.yahoo.com/search/boss/ 4 Interview in the Technology Review (Monday, July 16, 2007) understand the intentions behind search queries. Intentional query 6. CONCLUSIONS
suggestions could be regarded as a first step in this direction by While there is a significant body of research on understanding helping users to make their search intent more explicit. In user intent during search ([6], [23], [13], [5], [18], [10], [7]), to previous years, several different definitions of user intent emerged the best of our knowledge, the application of user intent to query [6], [10], [12],[25]. Broder [6] for example introduced a high level suggestion is a novel idea which has not been studied yet. In this taxonomy of search intent by categorizing search queries into paper, we introduce and define the concept of Intentional Query three categories: navigational, informational and transactional. Suggestion and present a prototypical algorithm as first evidence This has stimulated a series of follow up research on automatic for the feasibility of this idea. In a number of experiments, query categorization by [18], [13], [15], [12] and [23]. Evolutions we.could highlight interesting differences to traditional query of Broder's taxonomy include collapsing categories, adding suggestion mechanisms: 1) Differences in the diversity of search categories [5] and/or focusing on subsets only [18]. In contrast to results. Our results suggest that intentional query expansions can Broder, we do not incorporate high-level categories of search be used to diversify result sets. One implication of this finding is intent but rather focus on instances of user intentions that search engine vendors might be able to make search processes (informational vs. "things to consider when buying a car"). more focused if the searchers' intention is explicitly included in He et al. [12] used syntactic structures, i.e. verb-object pairs, to the search process. 2) Different click-through distributions for classify queries into Broder's categories. In a similar way, explicit intentional queries. Our experiments showed a higher Strohmaier et al. [25] employed part-of-speech trigrams as click-through ratio for explicit intentional queries compared to features to extract instances of user intentions in search query implicit intentional queries of similar length. The higher click- logs. In this paper, user intent is understood as a certain type of through numbers suggest that such queries retrieve more relevant verb phrases that explicitly state the user's goal. Downey et al results. This interesting finding might inspire novel ways to [10] view the information seeking process differently: Actions approach query suggestion in the future. that follow a search query are proposed as characterizations of the Our results could be relevant for a number of currently open searcher's information goal. The last URL visited in a search research problems. 1) Query disambiguation: Similar to Allan [2], session serves as a proxy for the user intent. While their approach where the problem of query disambiguation was approached by is useful to study user behavior during search sessions, it can not posing questions, Intentional Query Suggestion could provide a easily be used in an interactive way - to enable users to make their mechanism to identify the original user goal during search. 2) Search intent: A better understanding of the user's intent could In addition to studies of user intent, research on query suggestion give search engine vendors a better picture of users' needs. In the is related to our work as well. Query expansion [27], query long run, approximating user intent could help making search substitution [14], query recommendation [4] and query refinement more focused and prevent topic drift. 3) Search session: Along [17] are different concepts that share a similar objective: with a better understanding of users' search intent, new, more transforming an initial query into a 'better' query that is capable useful definitions of search sessions might be necessary. New of satisfying the searcher's information need by retrieving more definitions could differ from existing definitions by, for example, relevant documents. We deviate from these traditional approaches putting emphasis on a set of coherent, goal-related queries rather that focus on query vs. expected documents by focusing on than time-based notions, where multi-tasking behavior of users is queries and potential user intentions. Xu et al. [27] for example hard to capture. 4) Evaluation: Kinney et al. [16] point out the employed local and global documents in query expansion by difficulty of finding expert annotators when it comes to annotating applying the measure of global analysis to the selection of query web search results for evaluation purposes. In order to alleviate terms in local feedback. Query suggestion is closely related to the annotation task, the authors proposed statements that query substitution as well where the original query is extended by described the user intent behind a query. Intentional query new search terms to narrow the search scope. Jones et al. [14] suggestion might serve as a link between plain queries and the investigated a query substitution mechanism that does not exhibit intent statements by offering a list of empirically-grounded, query drift which represents a common drawback of query expansion techniques. The authors make use of search query sessions to infer relations between queries. Baeza-Yates et al. [4] proposed an approach that suggests related 7. ACKNOWLEDGMENTS
queries based on query log data and clustering. Former queries We would like to thank Microsoft Research for providing the were transformed into a new term-vector representation by taking search query log and Peter Prettenhofer for his support in into account the content of the clicked URLs. Another approach extracting the Explicit Intentional Query Dataset. This work is reported in [17] employed anchor texts for the purpose of query funded by the FWF Austrian Science Fund Grant P20269 refinement. It is based on the observation that queries and anchor TransAgere. The Know-Center is funded within the Austrian texts are highly similar. Query transformation techniques have COMET Program under the auspices of the Austrian Ministry of already spread to other areas such as question answering [1]. Transport, Innovation and Technology, the Austrian Ministry of Economics and Labor and by the State of Styria. COMET is Work on query suggestion has recently been done by [20], [22]. managed by the Austrian Research Promotion Agency FFG. Both papers apply their algorithms on bipartite graphs (user - query and/or query - URL) that were generated from search query logs. In a similar way, our work generates a bipartite graph from a search query log. However, our approach focuses on explicit 8. REFERENCES
intentional queries and their implicit intentional query [1] Agichtein E., Lawrence S. and Gravano L. Learning search neighborhood, thereby focusing on explicit user intent rather than engine specific query transformations for question the generation of syntactic or semantic query suggestions. answering. In 'WWW '01: Proceedings of the 10th International Conference on World Wide Web', ACM, New [15] Kang, I. and Kim, G. Query type classification for web document retrieval. In 'SIGIR '03: Proceedings of the 26th Annual International ACM SIGIR Conference on Research Allan J. and Raghavan H. Using part-of-speech patterns to and Development in Information Retrieval', ACM, New International ACM SIGIR Conference on Research and Development in Information Retrieval', ACM Press New [16] Kinney, K. A., Huffman, S. B. and Zhai, J. How evaluator domain expertise affects search result relevance judgments. Baeza-Yates R. and Ribeiro-Neto B. Modern Information 'CIKM '08: Proceedings of the 17th ACM Conference on Information and Knowledge Management', ACM, New [4] Baeza-Yates R., Hurtado C.A. and Mendoza M. Query recommendation using query logs in search engines. In [17] Kraft, R. and Zien, J. Mining anchor text for query Lindner W., Mesiti M., Türker C., Tzitzikas Y. and Vakali refinement. In 'WWW '04: Proceedings of the 13th A., 'EDBT Workshops', Springer, pp. 588--596, 2004. International Conference on World Wide Web', ACM, New York, NY, USA, pp. 666--674, 2004. [5] Baeza-Yates R., Calderón-Benavides L. and González-Caro [18] Lee, U., Liu, Z. and Cho, J. Automatic identification of user and Information Retrieval, pp. 98--109, 2006. 'WWW '05: Proceedings of the 14th International Conference on World Wide Web', ACM, New [6] Broder A. A taxonomy of web search. In ACM SIGIR Forum [19] Liu, H. and Singh, P. 'ConceptNet - A practical [7] Chang Y., He, K., Yu S. and Lu, W. Identifying user goals commonsense reasoning tool-kit'. In BT Technology Journal from web search results. In 'WI '06: Proceedings of the 2006 IEEE/WIC/ACM International Conference on Web Intelligence', IEEE Computer Society, Washington, DC, [20] Ma, H., Yang, H., King, I. and Lyu, M. R. Learning latent semantic relations from clickthrough data for query suggestion. In 'CIKM '08: Proceedings of the 17th ACM [8] Cohen, J. A coefficient of agreement for nominal scales. In Conference on Information and Knowledge Management', Educational and Psychological Measurement 20(1), 37, ACM, New York, NY, USA, pp. 709--718, 2008. [21] Manning, C. D., Raghavan, P. and Schütze, H. Introduction [9] Crabtree, D. W., Andreae, P. and Gao, X. Exploiting to Information Retrieval, Cambridge University Press, 2008. underrepresented query aspects for automatic query expansion. In [22] Mei, Q., Zhou, D. and Church, K. Query suggestion using 'KDD '07: Proceedings of the 13th ACM SIGKDD international conference on Knowledge Discovery hitting time. In 'CIKM '08: Proceedings of the 17th ACM and Data Mining', ACM, New York, NY, USA, pp. 191-- Conference on Information and Knowledge Management', ACM, New York, NY, USA, pp. 469--478, 2008. Downey, D., Liebling, D. and Dumais, S. Understanding the Rose, D. E. and Levinson, D. Understanding user goals in relationship between searchers, queries and information web search. In 'WWW '04: Proceedings of the 13th goals. 'CIKM '08: Proceedings of the 17th ACM Conference International Conference on World Wide Web', ACM, New on Information and Knowledge Management', ACM, New [24] Strohmaier, M., Prettenhofer, P. and Lux, M. Different degrees of explicitness in intentional artifacts - studying user Ferber, R. Information Retrieval, Dpunkt.Verlag, ISBN 978- goals in a large search query log. In 'CSKGOI'08: Proceedings of the Workshop on Commonsense Knowledge [12] He, K., Chang, Y. and Lu, W. Improving identification of and Goal Oriented Interfaces, in conjunction with IUI'08', latent user goals through search-result snippet classification. In 'WI '07: Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence', IEEE Computer Society, [25] Strohmaier, M., Prettenhofer, P. and Kröll, M. Acquiring Washington, DC, USA, pp. 683--686, 2007. explicit user goals from search query logs. In 'International Workshop on Agents and Data Mining Interaction ADMI' [13] Jansen, B. J., Booth, D. L. and Spink, A. Determining the informational, navigational, and transactional intent of web queries. In [26] Strzalkowski, T. and Carballo, J. Natural Language Inf. Process. Manage. 44(3), pp. 1251--1266, Information Retrieval: TREC-5 Report. In 'Text REtrieval Conference', pp. 164--173, 1998. [14] Jones, R., Rey, B., Madani, O. and Greiner, W. Generating [27] Xu, J. and Croft, W. B. Query expansion using local and 'WWW '06: Proceedings of the 15th International Conference on World Wide Web', ACM, New global document analysis. In 'SIGIR '96: Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval', ACM, New York, NY, USA, pp. 4--1, 1996

Source: http://www.kmi.tugraz.at/staff/markus/documents/2009_WSCD09_Intentional-Query-Suggestion.pdf