secml.utils¶

CLog¶

class secml.utils.c_log.CLog(level=None, logger_id=None, add_stream=True, file_handler=None, propagate=False)[source]¶

Bases: object

Manager for logging and logfiles.

Logger can be used to save important runtime code informations to disk instead of built-in function ‘print’. Along with any print-like formatted string, the logger stores full time stamp and calling class name.

The default filename of a log file is logs.log. This will be placed in the same directory of the calling file.

Logging levels currently available and target purpose:

DISABLE - 100: disable all logging.
CRITICAL - 50: critical error messages.
ERROR - 40: standard error messages.
WARNING - 30: standard error messages.
INFO - 20: general info logging.
DEBUG - 10: debug logging only.

Logger is fully integrated to the CTimer class in order to log performance of a desired method or routine.

Parameters

levelLOG_LEVEL, int or None, optional: Initial logging level. Default is None, meaning that the current logging level will be preserved if the logger has already been created.
logger_idstr or None, optional: Identifier of the logger. Default None. If None, creates a logger which is the root of the hierarchy
add_streambool, optional: If True, attach a stream handler to the logger. Default True. A stream handler prints to stdout the logged messages.
file_handlerstr or None, optional: If a string, attach a file handler to the logger. Default None. A file handler stores to the specified path the logged messages.
propagatebool, optional: If True, messages logged to this logger will be passed to the handlers of higher level (ancestor) loggers, in addition to any handler attached to this logger. Default False.

See also

CTimer: Manages performance monitoring and logging.

Notes

Unlike most of the Python logging modules, our implementation can be fully used inside parallelized code.

Examples

>>> from secml.array import CArray
>>> from secml.utils import CLog

>>> log = CLog().warning("{:}".format(CArray([1,2,3])))  
... - WARNING - CArray([1 2 3])

Attributes

level: Return logging level.
logger_id: Return identifier of the logger.
propagate: If True, events logged will be passed to the handlers of higher level (ancestor) loggers.

Methods

`attach_file`(self, filepath)	Adds a file handler to the logger.
`attach_stream`(self)	Adds a stream handler to the logger.
`catch_warnings`()	A context manager that copies and restores the warnings filter upon exiting the context.
`critical`(self, msg, \args, \\*kwargs)	Logs a message with level CRITICAl on this logger.
`debug`(self, msg, \args, \\*kwargs)	Logs a message with level DEBUG on this logger.
`error`(self, msg, \args, \\*kwargs)	Logs a message with level ERROR on this logger.
`filterwarnings`(action[, message, category, …])	Insert an entry into the list of warnings filters (at the front).
`get_child`(self, name)	Return a child logger associated with ancestor.
`info`(self, msg, \args, \\*kwargs)	Logs a message with level INFO on this logger.
`log`(self, level, msg, \args, \\*kwargs)	Logs a message with specified level on this logger.
`remove_handler_file`(self, filepath)	Removes the file handler from the logger.
`remove_handler_stream`(self)	Removes the stream handler from the logger.
`set_level`(self, level)	Sets logging level of the logger.
`timed`(self[, msg])	Timer decorator.
`timer`(self[, msg])	Starts a timed codeblock.
`warning`(self, msg, \args, \\*kwargs)	Logs a message with level WARNING on this logger.

attach_file(self, filepath)[source]¶: Adds a file handler to the logger.

attach_stream(self)[source]¶: Adds a stream handler to the logger.

static catch_warnings()[source]¶

A context manager that copies and restores the warnings filter upon exiting the context.

Wrapper of warnings.catch_warnings.

critical(self, msg, *args, **kwargs)[source]¶

Logs a message with level CRITICAl on this logger.

See CLog.log for details on args and kwargs.

debug(self, msg, *args, **kwargs)[source]¶

Logs a message with level DEBUG on this logger.

See CLog.log for details on args and kwargs.

error(self, msg, *args, **kwargs)[source]¶

Logs a message with level ERROR on this logger.

See CLog.log for details on args and kwargs.

static filterwarnings(action, message='', category=<class 'Warning'>, module='', lineno=0, append=False)[source]¶

Insert an entry into the list of warnings filters (at the front).

Wrapper of warnings.filterwarnings.

Parameters

actionstr: One of “error”, “ignore”, “always”, “default”, “module”, or “once”.
messagestr, optional: A regex that the warning message must match.
categoryclass, optional: A class that the warning must be a subclass of. Default Warning.
modulestr, optional: A regex that the module name must match.
linenoint, optional: An integer line number, 0 (default) matches all warnings.
appendbool, optional: If true, append to the list of filters.

get_child(self, name)[source]¶

Return a child logger associated with ancestor.

Parameters

namestr-like: Identifier of the child logger. Can be any object safely convertible to string (int, float, etc.)

Returns

child_loggerlogger: Instance of the child logger.

info(self, msg, *args, **kwargs)[source]¶

Logs a message with level INFO on this logger.

See CLog.log for details on args and kwargs.

property level¶: Return logging level.

log(self, level, msg, *args, **kwargs)[source]¶

Logs a message with specified level on this logger.

The msg is the message format string, and the args are the arguments which are merged into msg using the string formatting operator.

There are two keyword arguments in kwargs which are inspected: exc_info which, if it does not evaluate as false, causes exception information to be added to the logging message. If an exception tuple (in the format returned by sys.exc_info()) is provided, it is used; otherwise, sys.exc_info() is called to get the exception information.

The second keyword argument is extra which can be used to pass a dictionary which is used to populate the __dict__ of the LogRecord created for the logging event with user-defined attributes.

property logger_id¶: Return identifier of the logger.

property propagate¶: If True, events logged will be passed to the handlers of higher level (ancestor) loggers.

remove_handler_file(self, filepath)[source]¶: Removes the file handler from the logger.

remove_handler_stream(self)[source]¶: Removes the stream handler from the logger.

set_level(self, level)[source]¶: Sets logging level of the logger.

timed(self, msg=None)[source]¶

Timer decorator.

Returns a decorator that can be used to measure execution time of any method. Performance data will be stored inside the calling logger. Messages will be logged using the DEBUG logging level. As this decorator accepts optional arguments, must be called as a method. See examples.

Parameters

msgstr or None, optional: Custom message to display when entering the timed block. If None, “Entering timed block function_name…” will printed.

Examples

>>> from secml.array import CArray
>>> from secml.utils import CLog

>>> log = CLog()
>>> log.set_level(10)
>>> @log.timed()
... def abc():
...     print("Hello world!")

>>> abc()  
Hello world!

timer(self, msg=None)[source]¶

Starts a timed codeblock.

Returns an instance of context manager CTimer. Performance data will be stored inside the calling logger. Messages will be logged using the DEBUG logging level.

Parameters

msgstr or None, optional: Custom message to display when entering the timed block. If None, “Entering timed block…” will printed.

Examples

>>> from secml.array import CArray
>>> from secml.utils import CLog

>>> log = CLog()
>>> log.set_level(10)
>>> with log.timer("Timing the instruction..."):
...     a = CArray([1,2,3])  
2... - root - DEBUG - Timing the instruction...
2... - root - DEBUG - Elapsed time: ... ms

warning(self, msg, *args, **kwargs)[source]¶

Logs a message with level WARNING on this logger.

See CLog.log for details on args and kwargs.

class secml.utils.c_log.CTimer(log=None, msg=None)[source]¶

Bases: object

Context manager for performance logging

The code inside the specific context will be timed and performance data printed and/or logged.

This class fully integrates with CLog in order to store to disk performance data. When no logger is specified, data is printed on the console output.

Times are always stored in milliseconds (ms).

Parameters

logCLog or None, optional: Instance of CLog class to be used as performance logger. If a logger is specified, timer data will not be printed on console.
msgstr or None, optional: Custom message to display when entering the timed block. If None, “Entering timed block function_name…” will printed.

See also

CLog: CLog and store runtime informations on disk.

Examples

>>> from secml.array import CArray
>>> from secml.utils import CTimer

>>> with CTimer() as t:
...     a = CArray([1,2,3])  
Entering timed block...
Elapsed time: ... ms

>>> with CTimer(msg="Timing the instruction...") as t:
...     a = CArray([1,2,3])  
Timing the instruction...
Elapsed time: ... ms

>>> from secml.utils import CLog
>>> logger = CLog()
>>> logger.set_level(10)
>>> with CTimer(logger) as t:
...     a = CArray([1,2,3])  
2... - root - DEBUG - Entering timed block...
2... - root - DEBUG - Elapsed time: ... ms

Attributes

step: Return time elapsed from timer start (milliseconds).

Methods

timed([log, msg])

Timer decorator.

property step¶: Return time elapsed from timer start (milliseconds).

static timed(log=None, msg=None)[source]¶

Timer decorator.

Returns a decorator that can be used to measure execution time of any method. As this decorator accepts optional arguments, must be called as a method. See examples.

Parameters

logCLog or None, optional: Instance of CLog class to be used as performance logger. If a logger is specified, timer data will not be printed on console.
msgstr or None, optional: Custom message to display when entering the timed block. If None, “Entering timed block…” will printed.

See also

CLog: CLog and store runtime informations on disk.

Examples

>>> from secml.array import CArray
>>> from secml.utils import CTimer

>>> @CTimer.timed()
... def abc():
...     print("Hello world!")

>>> abc()  
Entering timed block `abc`...
Hello world!
Elapsed time: ... ms

c_file_manager¶

secml.utils.c_file_manager.folder_exist(folder_path)[source]¶

Test whether a folder exists.

Returns False for broken symbolic links.

secml.utils.c_file_manager.file_exist(file_path)[source]¶

Test whether a file exists.

Returns False for broken symbolic links.

secml.utils.c_file_manager.make_folder_incwd(folder_name, mode=511)[source]¶

Create a directory named folder_name inside current working directory (cwd).

Parameters

folder_namestr: Desired name for the new folder.
modeoct, optional: Octal literal representing the numeric mode to use. On some systems, mode is ignored. Where it is used, the current umask value is first masked out. If bits other than the last 9 (i.e. the last 3 digits of the octal representation of the mode) are set, their meaning is platform-dependent. On some platforms, they are ignored and you should call chmod() explicitly to set them. Default 0o777.

See also

make_folder: Create a directory given full path.

secml.utils.c_file_manager.make_folder(folder_path, mode=511)[source]¶

Create a directory inside folder_path with numeric mode ‘mode’.

All intermediate-level directories needed to contain the leaf directory will be recursively made.

Parameters

folder_pathstr: Desired path for the new folder.
modeoct, optional: Octal literal representing the numeric mode to use. On some systems, mode is ignored. Where it is used, the current umask value is first masked out. If bits other than the last 9 (i.e. the last 3 digits of the octal representation of the mode) are set, their meaning is platform-dependent. On some platforms, they are ignored and you should call chmod() explicitly to set them. Default 0o777.

See also

make_folder_inpath: Create a directory inside a specific folder.

secml.utils.c_file_manager.remove_folder(folder_path, force=False)[source]¶

Remove (delete) the directory path.

Path must point to a directory (but not a symbolic link to a directory).

Parameters

folder_pathstr: Absolute or relative path to folder to remove.
forcebool, optional: By default, if force is False, directory is removed only if empty, otherwise, OSError is raised. Set to True in order to remove the whole directory and its subdirectories.

secml.utils.c_file_manager.make_rand_folder(folder_path=None, custom_name=None)[source]¶

Create a random named folder.

Random name will be selected inside the integer range between 1 and 1 million [1, 1kk).

Parameters

folder_pathstr, optional: Path where to create the new directory. If None, folder will be created inside calling file folder.
custom_namestr, optional: Custom name to add before the random ID number. An underscore is placed between ID and custom_name.

Returns

target_pathstr: Absolute path of created directory.

Notes

There is a small chance that randomly generated folder already exists. Just run the function again :)

secml.utils.c_file_manager.abspath(file_name)[source]¶

Return the absolute path to file.

File name, as well as directory separator, is not added to the end of the returned path.

Examples

>>> import secml.utils.c_file_manager as fm

>>> cur_file = fm.split(__file__)[1]  # Getting only the filename
>>> cur_file  
'c_folder_manager.py'
>>> fm.abspath(cur_file)[-12:]
'/secml/utils'

secml.utils.c_file_manager.normpath(path)[source]¶

Normalize a pathname.

Normalize a pathname by collapsing redundant separators and up-level references so that A//B, A/B/, A/./B and A/foo/../B all become A/B. This string manipulation may change the meaning of a path that contains symbolic links. On Windows, it converts forward slashes to backward slashes.

Examples

>>> import secml.utils.c_file_manager as fm

>>> cur_path = fm.split(__file__)[0]  # Getting only the filename
>>> cur_path  
'---/src/secml/utils'
>>> upper_path = fm.join(cur_path, '..', '..')
>>> upper_path  
'---/src/secml/utils/../..'
>>> fm.normpath(upper_path)  
'---/src'

secml.utils.c_file_manager.join(*paths)[source]¶

Join one or more path components intelligently.

The return value is the concatenation of path and any members of *paths with exactly one directory separator (os.sep) following each non-empty part except the last, meaning that the result will only end in a separator if the last part is empty. If a component is an absolute path, all previous components are thrown away and joining continues from the absolute path component.

See also

split: Split the pathname path into a pair (head, tail).

secml.utils.c_file_manager.split(path)[source]¶

Split the pathname path into a pair (head, tail).

Tail is the last pathname component and head is everything leading up to that. The tail part will never contain a slash; if path ends in a slash, tail will be empty. If there is no slash in path, head will be empty. If path is empty, both head and tail are empty. Trailing slashes are stripped from head unless it is the root (one or more slashes only). In all cases, join(head, tail) returns a path to the same location as path (but the strings may differ).

Returns

out_splittuple of str: A tuple of strings consisting of (head, tail), where tail is the last pathname component and head is everything leading up to that.

See also

join: Join one or more path components intelligently.

Examples

>>> import secml.utils.c_file_manager as fm

>>> path = fm.join('dir1','dir2','dir3')
>>> path
'dir1/dir2/dir3'
>>> print(fm.split(path))
('dir1/dir2', 'dir3')

secml.utils.c_file_manager.expanduser(path)[source]¶

Replace user path shortcut with real user path.

On Unix and Windows, return path with an initial ~ or ~user replaced by that user’s home directory.

On Unix, an initial ~ is replaced by the environment variable HOME if it is set; otherwise the current user’s home directory is looked up in the password directory through the built-in module pwd. An initial ~user is looked up directly in the password directory.

On Windows, HOME and USERPROFILE will be used if set, otherwise a combination of HOMEPATH and HOMEDRIVE will be used. An initial ~user is handled by stripping the last directory component from the created user path derived above.

If the expansion fails or if the path does not begin with a tilde, the path is returned unchanged.

Examples

>>> import secml.utils.c_file_manager as fm

>>> fm.expanduser('~')  
'/home/username'
>>> fm.expanduser(fm.join('~','documents'))  
'/home/username/documents'

secml.utils.c_file_manager.dirsep()[source]¶: The character used by the operating system to separate pathname components. This is ‘/’ for POSIX and ‘’ for Windows. Note that knowing this is not sufficient to be able to parse or concatenate pathnames, use CFileManager.split() and CFileManager.join() instead, but it is occasionally useful.

secml.utils.c_file_manager.get_tempfile()[source]¶

Returns an handle to a temporary file.

The file will be destroyed as soon as it is closed (including an implicit close when the object is garbage collected).

pickle_utils¶

secml.utils.pickle_utils.save(file_path, obj)[source]¶

Save object to file using cPickle.

This functions stores a generic python object into a compressed gzip file (*.gz).

Saved objects can be loaded using .load.

Parameters

file_pathstr: Path to destination file.
objobject: Any python object to save.

Returns

obj_pathstr: Full path to the stored object.

secml.utils.pickle_utils.load(file_path, encoding='bytes')[source]¶

Load object from cPickle file.

Load a generic gzip compressed python object created by .save.

Parameters

file_pathstr: Path to target file to read.
encodingstr, optional: Encoding to use for loading the file. Default ‘bytes’.

download_utils¶

secml.utils.download_utils.dl_file(url, output_dir, user=None, chunk_size=1024, md5_digest=None)[source]¶

Download file from input url and store in output_dir.

Parameters

urlstr: Url of the file to download.
output_dirstr: Path to the directory where the file should be stored. If folder does not exists, will be created.
userstr or None, optional: String with the user[:password] if required for accessing url.
chunk_sizeint, optional: Size of the data chunk to read from url in bytes. Default 1024.
md5_digeststr or None, optional: Expected MD5 digest of the downloaded file. If a different digest is computed, the downloaded file will be removed and ValueError is raised.

secml.utils.download_utils.md5(fname, blocksize=65536)[source]¶

Generate RSA’s MD5 digest for input file.

Parameters

fnamestr: Path to the file to parse
blocksizeint: Size in bytes of the file chunks to read. Default 65536.

Returns

str: MD5 hex digest of input file.

dict_utils¶

secml.utils.dict_utils.load_dict(file_path, values_dtype=<class 'str'>, encoding='ascii')[source]¶

Load dictionary from textfile.

Each file’s line should be <key: value>

Parameters

file_pathstr: Full path to the file to read.
values_dtypedtype: Datatype of the values. Default str (string).
encodingstr, optional: Encoding to use for reading the file. Default ‘ascii’.

Returns

dictionarydict: Loaded dictionary with one key for each line in the input text file.

secml.utils.dict_utils.merge_dicts(*dicts)[source]¶

Shallow copy and merge any number of input dicts.

Precedence goes to key value pairs in latter dicts.

Parameters

dictsdict1, dict2, …: Any sequence of dict objects to merge.

Examples

>>> from secml.utils import merge_dicts

>>> d1 = {'attr1': 100, 'attr2': 200}
>>> d2 = {'attr3': 300, 'attr1': 999}  # Redefining `attr1`

>>> merge_dicts(d1, d2)  # Value of `attr1` will be set according to `d2` dictionary
{'attr3': 300, 'attr2': 200, 'attr1': 999}

secml.utils.dict_utils.invert_dict(d)[source]¶

Returns a new dict with keys as values and values as keys.

Parameters

ddict: Input dictionary. If one value of the dictionary is a list or a tuple, each element of the sequence will be considered separately.

Returns

dict: The new dictionary with d keys as values and d values as keys. In the case of duplicated d values, the value of the resulting key of the new dictionary will be a list with all the corresponding d keys.

Examples

>>> from secml.utils.dict_utils import invert_dict

>>> a = {'k1': 2, 'k2': 2, 'k3': 1}
>>> print(invert_dict(a))
{1: 'k3', 2: ['k1', 'k2']}

>>> a = {'k1': 2, 'k2': [2,3,1], 'k3': 1}
>>> print(invert_dict(a))
{1: ['k2', 'k3'], 2: ['k1', 'k2'], 3: 'k2'}

class secml.utils.dict_utils.LastInDict[source]¶

Bases: collections.abc.MutableMapping

Last In Dictionary.

A standard dictionary that keeps in memory the key of the last set item. The setting behaviour is queue-like: a single element can be inserted in the dictionary each time.

The last key can be changes manually calling LastInDict.lastitem_id = key.

Examples

>>> from secml.utils import LastInDict

>>> li = LastInDict()

>>> li['key1'] = 123
>>> li['key2'] = 102030

>>> li.lastin_key
'key2'
>>> li.lastin
102030

Attributes

lastin
lastin_key

Methods

`clear`(self)
`get`(self, key[, default])
`items`(self)
`keys`(self)
`pop`(self, key[, default])	If key is not found, d is returned if given, otherwise KeyError is raised.
`popitem`(self)	as a 2-tuple; but raise KeyError if D is empty.
`setdefault`(self, key[, default])
`update`(\args, \\*kwds)	If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
`values`(self)

property lastin¶

property lastin_key¶

class secml.utils.dict_utils.SubLevelsDict(data)[source]¶

Bases: collections.abc.MutableMapping

Sub-Levels Dictionary.

A standard dictionary that allows easy access to attributes of contained objects at infinite deep.

Examples

>>> from secml.utils import SubLevelsDict

>>> class Foo(object):
...     attr2 = 5

>>> li = SubLevelsDict({'attr1': Foo()})

>>> print(type(li['attr1']))
<class 'dict_utils.Foo'>
>>> print(li['attr1.attr2'])
5

>>> li['attr1.attr2'] = 10  # Subattributes can be set in the same way
>>> print(li['attr1.attr2'])
10

Methods

`clear`(self)
`get`(self, key[, default])
`items`(self)
`keys`(self)
`pop`(self, key[, default])	If key is not found, d is returned if given, otherwise KeyError is raised.
`popitem`(self)	as a 2-tuple; but raise KeyError if D is empty.
`setdefault`(self, key[, default])
`update`(\args, \\*kwds)	If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
`values`(self)

list_utils¶

secml.utils.list_utils.find_duplicates(l)[source]¶

Find and returns a python set with input list duplicates.

Parameters

llist: List to examine.

Returns

duplicatesset: Python set with input list duplicates.

References

http://stackoverflow.com/questions/9835762/find-and-list-duplicates-in-python-list

Examples

>>> from secml.utils.list_utils import find_duplicates
>>> l = ['1', '1', 2, '3', 2]
>>> print(find_duplicates(l))
set(['1', 2])

mixed_utils¶

class secml.utils.mixed_utils.AverageMeter[source]¶

Bases: object

Computes and stores the average and current value.

Attributes

valfloat: Current value.
avgfloat: Average.
sumfloat: Cumulative sum of seen values.
countint: Number of seen values.

Methods

update(self, val[, n])

Updated average and current value.

reset

reset(self)[source]¶

update(self, val, n=1)[source]¶

Updated average and current value.

Parameters

valfloat: New current value.
nint, optional: Multiplier for the current value. Indicates how many times the value should be counted in the average. Default 1.

class secml.utils.mixed_utils.OrderedFlexibleClass(*items)[source]¶

Bases: object

A flexible class exposing its attributes in a specific order when iterated.

Order of the attributes inside the class follows the inputs sequence. Any attribute set after class initialization will be placed at the end of attributes sequence (see examples).

Parameters

itemstuple1, tuple2, …: Any custom sequence of tuples with the attributes to set. Each tuple must be a (key, value) pair.

Examples

>>> from secml.utils import OrderedFlexibleClass

>>> c = OrderedFlexibleClass(('attr1', None), ('attr2', 5))
>>> print(tuple(attr for attr in c))
(None, 5)

>>> c.attr3 = 123
>>> print(tuple(attr for attr in c))
(None, 5, 123)

Attributes

attr_order: Returns a list specifing current attributes order.

property attr_order¶: Returns a list specifing current attributes order.

secml.utils.mixed_utils.check_is_fitted(obj, attributes, msg=None, check_all=True)[source]¶

Check if the input object is trained (fitted).

Checks if the input object is fitted by verifying if all or any of the input attributes are not None.

Parameters

objobject: Instance of the class to check. Must implement .fit() method.
attributesstr or list of str: Attribute or list of attributes to check. Es.: [‘classes’, ‘n_features’, …], ‘classes’
msgstr or None, optional: If None, the default error message is: “this {name} is not trained. Call .fit() first.”. For custom messages if ‘{name}’ is present in the message string, it is substituted by the class name of the checked object.
check_allbool, optional: Specify whether to check (True) if all of the given attributes are not None or (False) just any of them. Default True.

Raises

NotFittedError: If check_all is True and any of the attributes is None; if check_all is False and all of attributes are None.