DAS utilities

DAS provides a useful set of utilities.

CERN SSO authentication utils

CERN SSO toolkit. Provides get_data method which allow to get data begind CERN SSO protected site.

class DAS.utils.cern_sso_auth.HTTPSClientAuthHandler(key=None, cert=None, level=0)

Simple HTTPS client authentication class based on provided key/ca information

get_connection(host, timeout=300)

Connection method

https_open(req)

Open request method

DAS.utils.cern_sso_auth.get_data(url, key, cert, debug=0)

Main routine to get data from data service behind CERN SSO. Return file-like descriptor object (similar to open).

DAS.utils.cern_sso_auth.timestamp()

Construct timestamp used by Shibboleth

DAS config utils

Config utilities

DAS.utils.das_config.das_configfile(fname=None)

Return DAS configuration file name $DAS_ROOT/etc/das.cfg

DAS.utils.das_config.das_readconfig(fname=None, debug=False)

Return DAS configuration

DAS.utils.das_config.das_readconfig_helper(fname=None, debug=False)

Read DAS configuration file and store DAS parameters into returning dictionary.

DAS.utils.das_config.read_configparser(dasconfig)

Read DAS configuration

DAS.utils.das_config.read_wmcore(filename)

Read DAS python configuration file and store DAS parameters into returning dictionary.

DAS.utils.das_config.wmcore_config(filename)

Return WMCore config object for given file name

DAS.utils.das_config.write_configparser(dasconfig, use_default)

Write DAS configuration file

DAS DB utils

DAS option module

DAS option’s class

class DAS.utils.das_option.DASOption(section, name, itype='string', default=None, validator=None, destination=None, description='')

Class representing a single DAS option, independent of storage mechanism. Must define at least a section and name.

In configparser

[section] name = value

In WMCORE config

config.section.name = value

The type parameter forces conversion as appropriate. The default parameter allows the option to not be specified in the config file. If default is not set then not providing this key will throw an exception. The validator parameter should be a single-argument function which returns true if the value supplied is appropriate. The destination argument is for values which shouldn’t end up in config[section][name] = value but rather config[destination] = value Description is provided as a future option for some sort of automatic documentation.

get_from_configparser(config)

Extract a value from a configparser object.

get_from_wmcore(config)

As per get_from_configparser.

write_to_configparser(config, use_default=False)

Write the current value to a configparser option. This assumes it has already been read or the values in self.value somehow changed, otherwise if use_default is set only those values with defaults are set.

write_to_wmcore(config, use_default=False)

As per write_to_configparser.

DAS timer module

DAS json wrapper

JSON wrapper around different JSON python implementations. We use simplejson (json), cjson and yajl JSON implementation.

NOTE: different JSON implementation handle floats in different way Here are few examples

..doctest:

r1={"ts":time.time()}
print r1
{'ts': 1374255843.891289}

Python json:

..doctest:

print json.dumps(r1), json.loads(json.dumps(r1))
{"ts": 1374255843.891289} {u'ts': 1374255843.891289}

CJSON:

..doctest::
print cjson.encode(r1), cjson.decode(cjson.encode(r1)) {“ts”: 1374255843.89} {‘ts’: 1374255843.89}

YAJL:

..doctest:

print yajl.dumps(r1), yajl.loads(yajl.dumps(r1))
{"ts":1.37426e+09} {u'ts': 1374260000.0}

Therefore when records contains timestamp it is ADVISED to round it to integer. Then json/cjson implementations will agreee on input/output, while yajl will still differ (for that reason we can’t use yajl).

class DAS.utils.jsonwrapper.JSONDecoder(**kwargs)

JSONDecoder wrapper

decode(istring)

Decode JSON method

raw_decode(istring)

Decode given string

class DAS.utils.jsonwrapper.JSONEncoder(**kwargs)

JSONEncoder wrapper

encode(idict)

Decode JSON method

iterencode(idict)

Encode input dict

DAS.utils.jsonwrapper.dump(doc, source)

Use json.dump for back-ward compatibility, since cjson doesn’t provide this method. The dump method works on file-descriptor objects.

DAS.utils.jsonwrapper.dumps(idict, **kwargs)

Based on default MODULE invoke appropriate JSON encoding API call

DAS.utils.jsonwrapper.load(source)

Use json.load for back-ward compatibility, since cjson doesn’t provide this method. The load method works on file-descriptor objects.

DAS.utils.jsonwrapper.loads(idict, **kwargs)

Based on default MODULE invoke appropriate JSON decoding API call

DAS logger

General purpose DAS logger class. PrintManager class is based on the following work http://stackoverflow.com/questions/245304/how-do-i-get-the-name-of-a-function-or-method-from-within-a-python-function-or-m http://stackoverflow.com/questions/251464/how-to-get-the-function-name-as-string-in-python

class DAS.utils.logger.NullHandler(level=0)

Do nothing logger

emit(record)

This method does nothing.

handle(record)

This method does nothing.

class DAS.utils.logger.PrintManager(name='function', verbose=0)

PrintManager class

debug(msg)

print debug messages

error(msg)

print warning messages

info(msg)

print info messages

warning(msg)

print warning messages

DAS.utils.logger.funcname()

Extract caller name from a stack

DAS.utils.logger.print_msg(msg, cls, prefix='')

Print message in a form cls::caller msg, suitable for class usage

DAS.utils.logger.set_cherrypy_logger(hdlr, level)

set up logging for CherryPy

DAS pycurl manager

DAS query utils

DAS query utils.

DAS.utils.query_utils.compare_dicts(input_dict, exist_dict)

Helper function for compare_specs. It compares key/val pairs of Mongo dict conditions, e.g. {‘$gt’:10}. Return true if exist_dict is superset of input_dict

DAS.utils.query_utils.compare_specs(input_query, exist_query)

Function to compare set of fields and specs of two input mongo queries. Return True if results of exist_query are superset of resulst for input_query.

DAS.utils.query_utils.compare_str(query1, query2)

Function to compare string from specs of query. Return True if query2 is supperset of query1. query1&query2 is the string in the pattern:

([a-zA-Z0-9_\-\#/*\.])*

* is the sign indicates that a sub string of *:

([a-zA-Z0-9_\-\#/*\.])*

case 1. if query2 is flat query(w/out *) then query1 must be the same flat one

case 2. if query1 is start/end w/ * then query2 must start/end with *

case 3. if query2 is start/end w/out * then query1 must start/end with query2[0]/query[-1]

case 4. query1&query2 both include *

Way to perform a comparision is spliting:

query1 into X0*X1*X2*X3 query2 into Xa*Xb*Xc

foreach X in (Xa, Xb, Xc):

case 5. X is ‘’:

continue

special case:

when X0 & Xa are ‘’ or when X3 & Xc are ‘’ we already cover it in case 2

case 6. X not in query1 then return False

case 7. X in query1 begin at index:

case 7-1. X is the first X not ‘’ we looked up in query1.(Xa)
last_idx = index ; continue

case 7-2. X is not the first:

try to find the smallest Xb > Xa if and Only if we could find a sequence:

satisfy Xc > Xb > Xa, otherwise return False ‘=’ will happen when X0 = Xa then we could return True
DAS.utils.query_utils.convert2pattern(query)

In MongoDB patterns are specified via regular expression. Convert input query condition into regular expression patterns. Return new MongoDB compiled w/ regex query and query w/ debug info.

DAS.utils.query_utils.decode_mongo_query(query)

Decode query from storage format into mongo format.

DAS.utils.query_utils.encode_mongo_query(query)

Encode mongo query into storage format. MongoDB does not allow storage of dict with keys containing ”.” or MongoDB operators, e.g. $lt. So we convert input mongo query spec into list of dicts whose “key”/”value” are mongo query spec key/values. For example

spec:{"block.name":"aaa"}

converted into

spec:[{"key":"block.name", "value":'"aaa"'}]

Conversion is done using JSON dumps method.

DAS regex expressions

Regular expression patterns

DAS.utils.regex.word_chars(word, equal=True)

Creates a pattern of given word as series of its characters, e.g. for given word dataset I’ll get ‘^d$|^da$|^dat$|^data$|^datas$|^datase$|^dataset$’ which can be used later in regular expressions

DAS task manager

DAS threadpool utils

URL utils

Generic utils