secml.utils¶
CLog¶
-
class
secml.utils.c_log.
CLog
(level=None, logger_id=None, add_stream=True, file_handler=None, propagate=False)[source]¶ Bases:
object
Manager for logging and logfiles.
Logger can be used to save important runtime code information to disk instead of built-in function ‘print’. Along with any print-like formatted string, the logger stores full time stamp and calling class name.
The default filename of a log file is
logs.log
. This will be placed in the same directory of the calling file.- Logging levels currently available and target purpose:
DISABLE - 100: disable all logging.
CRITICAL - 50: critical error messages.
ERROR - 40: standard error messages.
WARNING - 30: standard error messages.
INFO - 20: general info logging.
DEBUG - 10: debug logging only.
Logger is fully integrated to the
CTimer
class in order to log performance of a desired method or routine.- Parameters
- levelLOG_LEVEL, int or None, optional
Initial logging level. Default is None, meaning that the current logging level will be preserved if the logger has already been created.
- logger_idstr or None, optional
Identifier of the logger. Default None. If None, creates a logger which is the root of the hierarchy
- add_streambool, optional
If True, attach a stream handler to the logger. Default True. A stream handler prints to stdout the logged messages.
- file_handlerstr or None, optional
If a string, attach a file handler to the logger. Default None. A file handler stores to the specified path the logged messages.
- propagatebool, optional
If True, messages logged to this logger will be passed to the handlers of higher level (ancestor) loggers, in addition to any handler attached to this logger. Default False.
See also
CTimer
Manages performance monitoring and logging.
Notes
Unlike most of the Python logging modules, our implementation can be fully used inside parallelized code.
Examples
>>> from secml.array import CArray >>> from secml.utils import CLog
>>> log = CLog().warning("{:}".format(CArray([1,2,3]))) ... - WARNING - CArray([1 2 3])
- Attributes
Methods
attach_file
(self, filepath)Adds a file handler to the logger.
attach_stream
(self)Adds a stream handler to the logger.
catch_warnings
([record])A context manager that copies and restores the warnings filter upon exiting the context.
critical
(self, msg, *args, **kwargs)Logs a message with level CRITICAl on this logger.
debug
(self, msg, *args, **kwargs)Logs a message with level DEBUG on this logger.
error
(self, msg, *args, **kwargs)Logs a message with level ERROR on this logger.
filterwarnings
(action[, message, category, …])Insert an entry into the list of warnings filters (at the front).
get_child
(self, name)Return a child logger associated with ancestor.
info
(self, msg, *args, **kwargs)Logs a message with level INFO on this logger.
log
(self, level, msg, *args, **kwargs)Logs a message with specified level on this logger.
remove_handler_file
(self, filepath)Removes the file handler from the logger.
remove_handler_stream
(self)Removes the stream handler from the logger.
set_level
(self, level)Sets logging level of the logger.
timed
(self[, msg])Timer decorator.
timer
(self[, msg])Starts a timed codeblock.
warning
(self, msg, *args, **kwargs)Logs a message with level WARNING on this logger.
-
static
catch_warnings
(record=False)[source]¶ A context manager that copies and restores the warnings filter upon exiting the context.
Wrapper of warnings.catch_warnings.
- Parameters
- recordbool, optional
If False (the default), the context manager returns None on entry. If True, a list is returned that is progressively populated with warning objects as seen by the context manager.
-
critical
(self, msg, *args, **kwargs)[source]¶ Logs a message with level CRITICAl on this logger.
See CLog.log for details on args and kwargs.
-
debug
(self, msg, *args, **kwargs)[source]¶ Logs a message with level DEBUG on this logger.
See CLog.log for details on args and kwargs.
-
error
(self, msg, *args, **kwargs)[source]¶ Logs a message with level ERROR on this logger.
See CLog.log for details on args and kwargs.
-
static
filterwarnings
(action, message='', category=<class 'Warning'>, module='', lineno=0, append=False)[source]¶ Insert an entry into the list of warnings filters (at the front).
Wrapper of warnings.filterwarnings.
- Parameters
- actionstr
One of “error”, “ignore”, “always”, “default”, “module”, or “once”.
- messagestr, optional
A regex that the warning message must match.
- categoryclass, optional
A class that the warning must be a subclass of. Default Warning.
- modulestr, optional
A regex that the module name must match.
- linenoint, optional
An integer line number, 0 (default) matches all warnings.
- appendbool, optional
If true, append to the list of filters.
-
get_child
(self, name)[source]¶ Return a child logger associated with ancestor.
- Parameters
- namestr-like
Identifier of the child logger. Can be any object safely convertible to string (int, float, etc.)
- Returns
- child_loggerlogger
Instance of the child logger.
-
info
(self, msg, *args, **kwargs)[source]¶ Logs a message with level INFO on this logger.
See CLog.log for details on args and kwargs.
-
property
level
¶ Return logging level.
-
log
(self, level, msg, *args, **kwargs)[source]¶ Logs a message with specified level on this logger.
The msg is the message format string, and the args are the arguments which are merged into msg using the string formatting operator.
There are two keyword arguments in kwargs which are inspected: exc_info which, if it does not evaluate as false, causes exception information to be added to the logging message. If an exception tuple (in the format returned by sys.exc_info()) is provided, it is used; otherwise, sys.exc_info() is called to get the exception information.
The second keyword argument is extra which can be used to pass a dictionary which is used to populate the __dict__ of the LogRecord created for the logging event with user-defined attributes.
-
property
logger_id
¶ Return identifier of the logger.
-
property
propagate
¶ If True, events logged will be passed to the handlers of higher level (ancestor) loggers.
-
timed
(self, msg=None)[source]¶ Timer decorator.
Returns a decorator that can be used to measure execution time of any method. Performance data will be stored inside the calling logger. Messages will be logged using the DEBUG logging level. As this decorator accepts optional arguments, must be called as a method. See examples.
- Parameters
- msgstr or None, optional
Custom message to display when entering the timed block. If None, “Entering timed block function_name…” will printed.
Examples
>>> from secml.array import CArray >>> from secml.utils import CLog
>>> log = CLog() >>> log.set_level(10) >>> @log.timed() ... def abc(): ... print("Hello world!")
>>> abc() Hello world!
-
timer
(self, msg=None)[source]¶ Starts a timed codeblock.
Returns an instance of context manager
CTimer
. Performance data will be stored inside the calling logger. Messages will be logged using the DEBUG logging level.- Parameters
- msgstr or None, optional
Custom message to display when entering the timed block. If None, “Entering timed block…” will printed.
Examples
>>> from secml.array import CArray >>> from secml.utils import CLog
>>> log = CLog() >>> log.set_level(10) >>> with log.timer("Timing the instruction..."): ... a = CArray([1,2,3]) 2... - root - DEBUG - Timing the instruction... 2... - root - DEBUG - Elapsed time: ... ms
-
class
secml.utils.c_log.
CTimer
(log=None, msg=None)[source]¶ Bases:
object
Context manager for performance logging
The code inside the specific context will be timed and performance data printed and/or logged.
This class fully integrates with
CLog
in order to store to disk performance data. When no logger is specified, data is printed on the console output.Times are always stored in milliseconds (ms).
- Parameters
- logCLog or None, optional
Instance of
CLog
class to be used as performance logger. If a logger is specified, timer data will not be printed on console.- msgstr or None, optional
Custom message to display when entering the timed block. If None, “Entering timed block function_name…” will printed.
See also
CLog
CLog and store runtime information on disk.
Examples
>>> from secml.array import CArray >>> from secml.utils import CTimer
>>> with CTimer() as t: ... a = CArray([1,2,3]) Entering timed block... Elapsed time: ... ms
>>> with CTimer(msg="Timing the instruction...") as t: ... a = CArray([1,2,3]) Timing the instruction... Elapsed time: ... ms
>>> from secml.utils import CLog >>> logger = CLog() >>> logger.set_level(10) >>> with CTimer(logger) as t: ... a = CArray([1,2,3]) 2... - root - DEBUG - Entering timed block... 2... - root - DEBUG - Elapsed time: ... ms
- Attributes
step
Return time elapsed from timer start (milliseconds).
Methods
timed
([log, msg])Timer decorator.
-
property
step
¶ Return time elapsed from timer start (milliseconds).
-
static
timed
(log=None, msg=None)[source]¶ Timer decorator.
Returns a decorator that can be used to measure execution time of any method. As this decorator accepts optional arguments, must be called as a method. See examples.
- Parameters
- logCLog or None, optional
Instance of
CLog
class to be used as performance logger. If a logger is specified, timer data will not be printed on console.- msgstr or None, optional
Custom message to display when entering the timed block. If None, “Entering timed block…” will printed.
See also
CLog
CLog and store runtime information on disk.
Examples
>>> from secml.array import CArray >>> from secml.utils import CTimer
>>> @CTimer.timed() ... def abc(): ... print("Hello world!")
>>> abc() Entering timed block `abc`... Hello world! Elapsed time: ... ms
c_file_manager¶
-
secml.utils.c_file_manager.
folder_exist
(folder_path)[source]¶ Test whether a folder exists.
Returns False for broken symbolic links.
-
secml.utils.c_file_manager.
file_exist
(file_path)[source]¶ Test whether a file exists.
Returns False for broken symbolic links.
-
secml.utils.c_file_manager.
make_folder_incwd
(folder_name, mode=511)[source]¶ Create a directory named folder_name inside current working directory (cwd).
- Parameters
- folder_namestr
Desired name for the new folder.
- modeoct, optional
Octal literal representing the numeric mode to use. On some systems, mode is ignored. Where it is used, the current umask value is first masked out. If bits other than the last 9 (i.e. the last 3 digits of the octal representation of the mode) are set, their meaning is platform-dependent. On some platforms, they are ignored and you should call chmod() explicitly to set them. Default 0o777.
See also
make_folder
Create a directory given full path.
-
secml.utils.c_file_manager.
make_folder
(folder_path, mode=511)[source]¶ Create a directory inside folder_path with numeric mode ‘mode’.
All intermediate-level directories needed to contain the leaf directory will be recursively made.
- Parameters
- folder_pathstr
Desired path for the new folder.
- modeoct, optional
Octal literal representing the numeric mode to use. On some systems, mode is ignored. Where it is used, the current umask value is first masked out. If bits other than the last 9 (i.e. the last 3 digits of the octal representation of the mode) are set, their meaning is platform-dependent. On some platforms, they are ignored and you should call chmod() explicitly to set them. Default 0o777.
See also
make_folder_inpath
Create a directory inside a specific folder.
-
secml.utils.c_file_manager.
remove_folder
(folder_path, force=False)[source]¶ Remove (delete) the directory path.
Path must point to a directory (but not a symbolic link to a directory).
- Parameters
- folder_pathstr
Absolute or relative path to folder to remove.
- forcebool, optional
By default, if force is False, directory is removed only if empty, otherwise, OSError is raised. Set to True in order to remove the whole directory and its subdirectories.
-
secml.utils.c_file_manager.
make_rand_folder
(folder_path=None, custom_name=None)[source]¶ Create a random named folder.
Random name will be selected inside the integer range between 1 and 1 million [1, 1kk).
- Parameters
- folder_pathstr, optional
Path where to create the new directory. If None, folder will be created inside calling file folder.
- custom_namestr, optional
Custom name to add before the random ID number. An underscore is placed between ID and custom_name.
- Returns
- target_pathstr
Absolute path of created directory.
Notes
There is a small chance that randomly generated folder already exists. Just run the function again :)
-
secml.utils.c_file_manager.
abspath
(file_name)[source]¶ Return the absolute path to file.
File name, as well as directory separator, is not added to the end of the returned path.
Examples
>>> import secml.utils.c_file_manager as fm
>>> cur_file = fm.split(__file__)[1] # Getting only the filename >>> cur_file 'c_folder_manager.py' >>> fm.abspath(cur_file)[-12:] '/secml/utils'
-
secml.utils.c_file_manager.
normpath
(path)[source]¶ Normalize a pathname.
Normalize a pathname by collapsing redundant separators and up-level references so that A//B, A/B/, A/./B and A/foo/../B all become A/B. This string manipulation may change the meaning of a path that contains symbolic links. On Windows, it converts forward slashes to backward slashes.
Examples
>>> import secml.utils.c_file_manager as fm
>>> cur_path = fm.split(__file__)[0] # Getting only the filename >>> cur_path '---/src/secml/utils' >>> upper_path = fm.join(cur_path, '..', '..') >>> upper_path '---/src/secml/utils/../..' >>> fm.normpath(upper_path) '---/src'
-
secml.utils.c_file_manager.
join
(*paths)[source]¶ Join one or more path components intelligently.
The return value is the concatenation of path and any members of *paths with exactly one directory separator (os.sep) following each non-empty part except the last, meaning that the result will only end in a separator if the last part is empty. If a component is an absolute path, all previous components are thrown away and joining continues from the absolute path component.
See also
split
Split the pathname path into a pair (head, tail).
-
secml.utils.c_file_manager.
split
(path)[source]¶ Split the pathname path into a pair (head, tail).
Tail is the last pathname component and head is everything leading up to that. The tail part will never contain a slash; if path ends in a slash, tail will be empty. If there is no slash in path, head will be empty. If path is empty, both head and tail are empty. Trailing slashes are stripped from head unless it is the root (one or more slashes only). In all cases, join(head, tail) returns a path to the same location as path (but the strings may differ).
- Returns
- out_splittuple of str
A tuple of strings consisting of (head, tail), where tail is the last pathname component and head is everything leading up to that.
See also
join
Join one or more path components intelligently.
Examples
>>> import secml.utils.c_file_manager as fm
>>> path = fm.join('dir1','dir2','dir3') >>> path 'dir1/dir2/dir3' >>> print(fm.split(path)) ('dir1/dir2', 'dir3')
-
secml.utils.c_file_manager.
expanduser
(path)[source]¶ Replace user path shortcut with real user path.
On Unix and Windows, return path with an initial ~ or ~user replaced by that user’s home directory.
On Unix, an initial ~ is replaced by the environment variable HOME if it is set; otherwise the current user’s home directory is looked up in the password directory through the built-in module pwd. An initial ~user is looked up directly in the password directory.
On Windows, HOME and USERPROFILE will be used if set, otherwise a combination of HOMEPATH and HOMEDRIVE will be used. An initial ~user is handled by stripping the last directory component from the created user path derived above.
If the expansion fails or if the path does not begin with a tilde, the path is returned unchanged.
Examples
>>> import secml.utils.c_file_manager as fm
>>> fm.expanduser('~') '/home/username' >>> fm.expanduser(fm.join('~','documents')) '/home/username/documents'
-
secml.utils.c_file_manager.
dirsep
()[source]¶ The character used by the operating system to separate pathname components. This is ‘/’ for POSIX and ‘’ for Windows. Note that knowing this is not sufficient to be able to parse or concatenate pathnames, use CFileManager.split() and CFileManager.join() instead, but it is occasionally useful.
pickle_utils¶
-
secml.utils.pickle_utils.
save
(file_path, obj)[source]¶ Save object to file using cPickle.
This functions stores a generic python object into a compressed gzip file (*.gz).
Saved objects can be loaded using .load.
- Parameters
- file_pathstr
Path to destination file.
- objobject
Any python object to save.
- Returns
- obj_pathstr
Full path to the stored object.
Notes
Objects are stored using protocol 4 data stream format. For more information see https://docs.python.org/3/library/pickle.html#data-stream-format
download_utils¶
-
secml.utils.download_utils.
dl_file
(url, output_dir, user=None, headers=None, chunk_size=1024, md5_digest=None)[source]¶ Download file from input url and store in output_dir.
- Parameters
- urlstr
Url of the file to download.
- output_dirstr
Path to the directory where the file should be stored. If folder does not exists, will be created.
- userstr or None, optional
String with the user[:password] if required for accessing url.
- headersdict or None, optional
Dictionary with any additional header for the download request.
- chunk_sizeint, optional
Size of the data chunk to read from url in bytes. Default 1024.
- md5_digeststr or None, optional
Expected MD5 digest of the downloaded file. If a different digest is computed, the downloaded file will be removed and ValueError is raised.
-
secml.utils.download_utils.
dl_file_gitlab
(repo_url, file_path, output_dir, branch='master', token=None, chunk_size=1024, md5_digest=None)[source]¶ Download file from a gitlab.com repository and store in output_dir.
- Parameters
- repo_urlstr
Url of the repository from which download the file. Can include the http(s)://gitlab.com/ prefix.
- file_pathstr
Path to the file to download, relative to the repository.
- output_dirstr
Path to the directory where the file should be stored. If folder does not exists, will be created.
- branchstr, optional
Branch from which the file should be downloaded. Default ‘master’.
- tokenstr or None, optional
Personal access token, required to access private repositories. See: https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html
- chunk_sizeint, optional
Size of the data chunk to read from url in bytes. Default 1024.
- md5_digeststr or None, optional
Expected MD5 digest of the downloaded file. If a different digest is computed, the downloaded file will be removed and ValueError is raised.
dict_utils¶
-
secml.utils.dict_utils.
load_dict
(file_path, values_dtype=<class 'str'>, encoding='ascii')[source]¶ Load dictionary from textfile.
Each file’s line should be <key: value>
- Parameters
- file_pathstr
Full path to the file to read.
- values_dtypedtype
Datatype of the values. Default str (string).
- encodingstr, optional
Encoding to use for reading the file. Default ‘ascii’.
- Returns
- dictionarydict
Loaded dictionary with one key for each line in the input text file.
-
secml.utils.dict_utils.
merge_dicts
(*dicts)[source]¶ Shallow copy and merge any number of input dicts.
Precedence goes to key value pairs in latter dicts.
- Parameters
- dictsdict1, dict2, …
Any sequence of dict objects to merge.
Examples
>>> from secml.utils import merge_dicts
>>> d1 = {'attr1': 100, 'attr2': 200} >>> d2 = {'attr3': 300, 'attr1': 999} # Redefining `attr1`
>>> merge_dicts(d1, d2) # Value of `attr1` will be set according to `d2` dictionary {'attr3': 300, 'attr2': 200, 'attr1': 999}
-
secml.utils.dict_utils.
invert_dict
(d)[source]¶ Returns a new dict with keys as values and values as keys.
- Parameters
- ddict
Input dictionary. If one value of the dictionary is a list or a tuple, each element of the sequence will be considered separately.
- Returns
- dict
The new dictionary with d keys as values and d values as keys. In the case of duplicated d values, the value of the resulting key of the new dictionary will be a list with all the corresponding d keys.
Examples
>>> from secml.utils.dict_utils import invert_dict
>>> a = {'k1': 2, 'k2': 2, 'k3': 1} >>> print(invert_dict(a)) {1: 'k3', 2: ['k1', 'k2']}
>>> a = {'k1': 2, 'k2': [2,3,1], 'k3': 1} >>> print(invert_dict(a)) {1: ['k2', 'k3'], 2: ['k1', 'k2'], 3: 'k2'}
-
class
secml.utils.dict_utils.
LastInDict
[source]¶ Bases:
collections.abc.MutableMapping
Last In Dictionary.
A standard dictionary that keeps in memory the key of the last set item. The setting behaviour is queue-like: a single element can be inserted in the dictionary each time.
The last key can be changes manually calling LastInDict.lastitem_id = key.
Examples
>>> from secml.utils import LastInDict
>>> li = LastInDict()
>>> li['key1'] = 123 >>> li['key2'] = 102030
>>> li.lastin_key 'key2' >>> li.lastin 102030
- Attributes
- lastin
- lastin_key
Methods
clear
(self)get
(self, key[, default])items
(self)keys
(self)pop
(self, key[, default])If key is not found, d is returned if given, otherwise KeyError is raised.
popitem
(self)as a 2-tuple; but raise KeyError if D is empty.
setdefault
(self, key[, default])update
(*args, **kwds)If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
values
(self)-
property
lastin
¶
-
property
lastin_key
¶
-
class
secml.utils.dict_utils.
SubLevelsDict
(data)[source]¶ Bases:
collections.abc.MutableMapping
Sub-Levels Dictionary.
A standard dictionary that allows easy access to attributes of contained objects at infinite deep.
Examples
>>> from secml.utils import SubLevelsDict
>>> class Foo: ... attr2 = 5
>>> li = SubLevelsDict({'attr1': Foo()})
>>> print(type(li['attr1'])) <class 'dict_utils.Foo'> >>> print(li['attr1.attr2']) 5
>>> li['attr1.attr2'] = 10 # Subattributes can be set in the same way >>> print(li['attr1.attr2']) 10
Methods
clear
(self)get
(self, key[, default])items
(self)keys
(self)pop
(self, key[, default])If key is not found, d is returned if given, otherwise KeyError is raised.
popitem
(self)as a 2-tuple; but raise KeyError if D is empty.
setdefault
(self, key[, default])update
(*args, **kwds)If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
values
(self)
list_utils¶
-
secml.utils.list_utils.
find_duplicates
(l)[source]¶ Find and returns a python set with input list duplicates.
- Parameters
- llist
List to examine.
- Returns
- duplicatesset
Python set with input list duplicates.
References
http://stackoverflow.com/questions/9835762/find-and-list-duplicates-in-python-list
Examples
>>> from secml.utils.list_utils import find_duplicates >>> l = ['1', '1', 2, '3', 2] >>> print(find_duplicates(l)) set(['1', 2])
mixed_utils¶
-
class
secml.utils.mixed_utils.
AverageMeter
[source]¶ Bases:
object
Computes and stores the average and current value.
- Attributes
- valfloat
Current value.
- avgfloat
Average.
- sumfloat
Cumulative sum of seen values.
- countint
Number of seen values.
Methods
update
(self, val[, n])Updated average and current value.
reset
-
class
secml.utils.mixed_utils.
OrderedFlexibleClass
(*items)[source]¶ Bases:
object
A flexible class exposing its attributes in a specific order when iterated.
Order of the attributes inside the class follows the inputs sequence. Any attribute set after class initialization will be placed at the end of attributes sequence (see examples).
- Parameters
- itemstuple1, tuple2, …
Any custom sequence of tuples with the attributes to set. Each tuple must be a (key, value) pair.
Examples
>>> from secml.utils import OrderedFlexibleClass
>>> c = OrderedFlexibleClass(('attr1', None), ('attr2', 5)) >>> print(tuple(attr for attr in c)) (None, 5)
>>> c.attr3 = 123 >>> print(tuple(attr for attr in c)) (None, 5, 123)
- Attributes
attr_order
Returns a list specifing current attributes order.
-
property
attr_order
¶ Returns a list specifing current attributes order.
-
secml.utils.mixed_utils.
check_is_fitted
(obj, attributes, msg=None, check_all=True)[source]¶ Check if the input object is trained (fitted).
Checks if the input object is fitted by verifying if all or any of the input attributes are not None.
- Parameters
- objobject
Instance of the class to check. Must implement .fit() method.
- attributesstr or list of str
Attribute or list of attributes to check. Es.: [‘classes’, ‘n_features’, …], ‘classes’
- msgstr or None, optional
If None, the default error message is: “this {name} is not trained. Call .fit() first.”. For custom messages if ‘{name}’ is present in the message string, it is substituted by the class name of the checked object.
- check_allbool, optional
Specify whether to check (True) if all of the given attributes are not None or (False) just any of them. Default True.
- Raises
- NotFittedError
If check_all is True and any of the attributes is None; if check_all is False and all of attributes are None.