DAS utilities¶
DAS provides a useful set of utilities.
CERN SSO authentication utils¶
CERN SSO toolkit. Provides get_data method which allow to get data begind CERN SSO protected site.
-
class
DAS.utils.cern_sso_auth.
HTTPSClientAuthHandler
(key=None, cert=None, level=0)¶ Simple HTTPS client authentication class based on provided key/ca information
-
get_connection
(host, timeout=300)¶ Connection method
-
https_open
(req)¶ Open request method
-
-
DAS.utils.cern_sso_auth.
get_data
(url, key, cert, debug=0)¶ Main routine to get data from data service behind CERN SSO. Return file-like descriptor object (similar to open).
-
DAS.utils.cern_sso_auth.
timestamp
()¶ Construct timestamp used by Shibboleth
DAS config utils¶
Config utilities
-
DAS.utils.das_config.
das_configfile
(fname=None)¶ Return DAS configuration file name $DAS_ROOT/etc/das.cfg
-
DAS.utils.das_config.
das_readconfig
(fname=None, debug=False)¶ Return DAS configuration
-
DAS.utils.das_config.
das_readconfig_helper
(fname=None, debug=False)¶ Read DAS configuration file and store DAS parameters into returning dictionary.
-
DAS.utils.das_config.
read_configparser
(dasconfig)¶ Read DAS configuration
-
DAS.utils.das_config.
read_wmcore
(filename)¶ Read DAS python configuration file and store DAS parameters into returning dictionary.
-
DAS.utils.das_config.
wmcore_config
(filename)¶ Return WMCore config object for given file name
-
DAS.utils.das_config.
write_configparser
(dasconfig, use_default)¶ Write DAS configuration file
DAS DB utils¶
DAS option module¶
DAS option’s class
-
class
DAS.utils.das_option.
DASOption
(section, name, itype='string', default=None, validator=None, destination=None, description='')¶ Class representing a single DAS option, independent of storage mechanism. Must define at least a section and name.
In configparser
[section] name = value
In WMCORE config
config.section.name = value
The type parameter forces conversion as appropriate. The default parameter allows the option to not be specified in the config file. If default is not set then not providing this key will throw an exception. The validator parameter should be a single-argument function which returns true if the value supplied is appropriate. The destination argument is for values which shouldn’t end up in config[section][name] = value but rather config[destination] = value Description is provided as a future option for some sort of automatic documentation.
-
get_from_configparser
(config)¶ Extract a value from a configparser object.
-
get_from_wmcore
(config)¶ As per get_from_configparser.
-
write_to_configparser
(config, use_default=False)¶ Write the current value to a configparser option. This assumes it has already been read or the values in self.value somehow changed, otherwise if use_default is set only those values with defaults are set.
-
write_to_wmcore
(config, use_default=False)¶ As per write_to_configparser.
-
DAS timer module¶
DAS json wrapper¶
JSON wrapper around different JSON python implementations. We use simplejson (json), cjson and yajl JSON implementation.
NOTE: different JSON implementation handle floats in different way Here are few examples
..doctest:
r1={"ts":time.time()}
print r1
{'ts': 1374255843.891289}
Python json:
..doctest:
print json.dumps(r1), json.loads(json.dumps(r1))
{"ts": 1374255843.891289} {u'ts': 1374255843.891289}
CJSON:
- ..doctest::
- print cjson.encode(r1), cjson.decode(cjson.encode(r1)) {“ts”: 1374255843.89} {‘ts’: 1374255843.89}
YAJL:
..doctest:
print yajl.dumps(r1), yajl.loads(yajl.dumps(r1))
{"ts":1.37426e+09} {u'ts': 1374260000.0}
Therefore when records contains timestamp it is ADVISED to round it to integer. Then json/cjson implementations will agreee on input/output, while yajl will still differ (for that reason we can’t use yajl).
-
class
DAS.utils.jsonwrapper.
JSONDecoder
(**kwargs)¶ JSONDecoder wrapper
-
decode
(istring)¶ Decode JSON method
-
raw_decode
(istring)¶ Decode given string
-
-
class
DAS.utils.jsonwrapper.
JSONEncoder
(**kwargs)¶ JSONEncoder wrapper
-
encode
(idict)¶ Decode JSON method
-
iterencode
(idict)¶ Encode input dict
-
-
DAS.utils.jsonwrapper.
dump
(doc, source)¶ Use json.dump for back-ward compatibility, since cjson doesn’t provide this method. The dump method works on file-descriptor objects.
-
DAS.utils.jsonwrapper.
dumps
(idict, **kwargs)¶ Based on default MODULE invoke appropriate JSON encoding API call
-
DAS.utils.jsonwrapper.
load
(source)¶ Use json.load for back-ward compatibility, since cjson doesn’t provide this method. The load method works on file-descriptor objects.
-
DAS.utils.jsonwrapper.
loads
(idict, **kwargs)¶ Based on default MODULE invoke appropriate JSON decoding API call
DAS logger¶
General purpose DAS logger class. PrintManager class is based on the following work http://stackoverflow.com/questions/245304/how-do-i-get-the-name-of-a-function-or-method-from-within-a-python-function-or-m http://stackoverflow.com/questions/251464/how-to-get-the-function-name-as-string-in-python
-
class
DAS.utils.logger.
NullHandler
(level=0)¶ Do nothing logger
-
emit
(record)¶ This method does nothing.
-
handle
(record)¶ This method does nothing.
-
-
class
DAS.utils.logger.
PrintManager
(name='function', verbose=0)¶ PrintManager class
-
debug
(msg)¶ print debug messages
-
error
(msg)¶ print warning messages
-
info
(msg)¶ print info messages
-
warning
(msg)¶ print warning messages
-
-
DAS.utils.logger.
funcname
()¶ Extract caller name from a stack
-
DAS.utils.logger.
print_msg
(msg, cls, prefix='')¶ Print message in a form cls::caller msg, suitable for class usage
-
DAS.utils.logger.
set_cherrypy_logger
(hdlr, level)¶ set up logging for CherryPy
DAS pycurl manager¶
DAS query utils¶
DAS query utils.
-
DAS.utils.query_utils.
compare_dicts
(input_dict, exist_dict)¶ Helper function for compare_specs. It compares key/val pairs of Mongo dict conditions, e.g. {‘$gt’:10}. Return true if exist_dict is superset of input_dict
-
DAS.utils.query_utils.
compare_specs
(input_query, exist_query)¶ Function to compare set of fields and specs of two input mongo queries. Return True if results of exist_query are superset of resulst for input_query.
-
DAS.utils.query_utils.
compare_str
(query1, query2)¶ Function to compare string from specs of query. Return True if query2 is supperset of query1. query1&query2 is the string in the pattern:
([a-zA-Z0-9_\-\#/*\.])*
* is the sign indicates that a sub string of *:
([a-zA-Z0-9_\-\#/*\.])*
case 1. if query2 is flat query(w/out *) then query1 must be the same flat one
case 2. if query1 is start/end w/ * then query2 must start/end with *
case 3. if query2 is start/end w/out * then query1 must start/end with query2[0]/query[-1]
case 4. query1&query2 both include *
Way to perform a comparision is spliting:
query1 into X0*X1*X2*X3 query2 into Xa*Xb*Xcforeach X in (Xa, Xb, Xc):
case 5. X is ‘’:
continue
special case:
when X0 & Xa are ‘’ or when X3 & Xc are ‘’ we already cover it in case 2case 6. X not in query1 then return False
case 7. X in query1 begin at index:
- case 7-1. X is the first X not ‘’ we looked up in query1.(Xa)
- last_idx = index ; continue
case 7-2. X is not the first:
try to find the smallest Xb > Xa if and Only if we could find a sequence:
satisfy Xc > Xb > Xa, otherwise return False ‘=’ will happen when X0 = Xa then we could return True
-
DAS.utils.query_utils.
convert2pattern
(query)¶ In MongoDB patterns are specified via regular expression. Convert input query condition into regular expression patterns. Return new MongoDB compiled w/ regex query and query w/ debug info.
-
DAS.utils.query_utils.
decode_mongo_query
(query)¶ Decode query from storage format into mongo format.
-
DAS.utils.query_utils.
encode_mongo_query
(query)¶ Encode mongo query into storage format. MongoDB does not allow storage of dict with keys containing ”.” or MongoDB operators, e.g. $lt. So we convert input mongo query spec into list of dicts whose “key”/”value” are mongo query spec key/values. For example
spec:{"block.name":"aaa"} converted into spec:[{"key":"block.name", "value":'"aaa"'}]
Conversion is done using JSON dumps method.
DAS regex expressions¶
Regular expression patterns
-
DAS.utils.regex.
word_chars
(word, equal=True)¶ Creates a pattern of given word as series of its characters, e.g. for given word dataset I’ll get ‘^d$|^da$|^dat$|^data$|^datas$|^datase$|^dataset$’ which can be used later in regular expressions