Overview¶

The basic keyword search workflow:

• The query is tokenized (including shallow parsing for key=val patterns and quotes)
• Then by a number of entity matchers the entry points are generated (matches of each keyword or the few nearby keywords into some of the schema or value terms which make part of a structured query)
• Exploring the various combinations of the entry points the candidates for query suggestions are evaluated and ranked
• The top results are presented to the users (generating a valid DAS query and a providing a readable query description)

Adapters to access Metadata¶

Contain meta-data related functions:

• accessing integration schema: fields, values, constraints on inputs/queries
• tracking fields available
• tracking known (input field) values

Tokenizing and parsing the query¶

Module description:

• first clean up input keyword query (rm extra spaces, standardize notation)

• then it tokenizes the query into:

  • individual query terms
  • compound query terms in brackets (e.g. “number of events”)
  • phrases: “terms operator value” (e.g. nevent > 1, “number of events”=100)

DAS.keywordsearch.tokenizer.cleanup_query(query)¶

Returns cleaned query by applying a number of transformation patterns that removes spaces and simplifies the conditions

>>> cleanup_query('number of events = 33')
'number of events=33'

>>> cleanup_query('number of events >    33')
'number of events>33'

>>> cleanup_query('more than 33 events')
'>33 events'

>>> cleanup_query('X more than 33 events')
'X>33 events'

>>> cleanup_query('find datasets where X more than 33 events')
'datasets where X>33 events'

>>> cleanup_query('=2012-02-01')
'= 20120201'

>>> cleanup_query('>= 2012-02-01')
'>= 20120201'

DAS.keywordsearch.tokenizer.get_keyword_without_operator(keyword)¶

splits keyword on operator

>>> get_keyword_without_operator('number of events >= 10')
'number of events'

>>> get_keyword_without_operator('dataset')
'dataset'

>>> get_keyword_without_operator('dataset=Zmm')
'dataset'

DAS.keywordsearch.tokenizer.get_operator_and_param(keyword)¶

splits keyword on operator

>>> get_operator_and_param('number of events >= 10')
{'type': 'filter', 'param': '10', 'op': '>='}

>>> get_operator_and_param('dataset')

>>> get_operator_and_param('dataset=Zmm')
{'type': 'filter', 'param': 'Zmm', 'op': '='}

DAS.keywordsearch.tokenizer.test_operator_containment(keyword)¶

returns whether a keyword token contains an operator (this is useful then processing a list of tokens, as only the last token may have an operator)

>>> test_operator_containment('number of events >= 10')
True

>>> test_operator_containment('number')
False

DAS.keywordsearch.tokenizer.tokenize(query)¶

tokenizes the query retaining the phrases in brackets together it also tries to group “word operator word” sequences together, such as

"number of events">10 or dataset=/Zmm/*/raw-reco

so it could be used for further processing.

special characters currently allowed in data values include: _*/-

For example:

>>> tokenize('file dataset=/Zmm*/*/raw-reco lumi=20853 nevents>10'                     '"number of events">10 /Zmm*/*/raw-reco')
['file', 'dataset=/Zmm*/*/raw-reco', 'lumi=20853', 'nevents>10',     'number of events>10', '/Zmm*/*/raw-reco']

>>> tokenize('file dataset=/Zmm*/*/raw-reco lumi=20853 dataset.nevents>10'                     '"number of events">10 /Zmm*/*/raw-reco')
['file', 'dataset=/Zmm*/*/raw-reco', 'lumi=20853', 'dataset.nevents>10',     'number of events>10', '/Zmm*/*/raw-reco']

>>> tokenize("file dataset=/Zmm*/*/raw-reco lumi=20853 dataset.nevents>10"                      "'number of events'>10 /Zmm*/*/raw-reco")
['file', 'dataset=/Zmm*/*/raw-reco', 'lumi=20853', 'dataset.nevents>10',     'number of events>10', '/Zmm*/*/raw-reco']


>>> tokenize('user=vidmasze@cern.ch')
['user=vidmasze@cern.ch']

Entity matchers¶

Contain entity matching related functions:

• Name matching / custom String distance
• Chunk matching (multi-word terms into names of service output fields)
• Value matching
• And CMS specific dataset matching

Generating and Ranking the Query suggestion¶

Presenting the Results to the user¶

Presentation of query suggestions

The module contain functions for presenting the results as DASQL and formatting/coloring them in HTML.

DAS.keywordsearch.presentation.result_presentation.dasql_to_nl(dasql_tuple)¶

Returns natural language representation of a generated DAS query

so to explain users what does it mean.

DAS.keywordsearch.presentation.result_presentation.fescape(value)¶

escape a value to be included in html

DAS.keywordsearch.presentation.result_presentation.result_to_dasql(result, frmt='text', shorten_html=True, max_value_len=26)¶

returns proposed query as DASQL in there formats:

• text, standard DASQL

• html, colorified DASQL with long values shortened down (if shorten_html

  is specified)

DAS.keywordsearch.presentation.result_presentation.shorten_value(value, max_value_len)¶

provide a shorter version of a very long value for displaying in (html) results. Examples include long dataset or block names.