secml.ml.peval

CPerfEvaluator

class secml.ml.peval.c_perfevaluator.CPerfEvaluator(splitter, metric)[source]

Bases: secml.core.c_creator.CCreator

Evaluate the best parameters for input estimator.

Parameters
splitterCDataSplitter or str

Object to use for splitting the dataset into train and validation.

metricCMetric or str

Name of the metric that we want maximize / minimize.

Attributes
class_type

Defines class type.

logger

Logger for current object.

verbose

Verbosity level of logger output.

Methods

compute_performance(self, estimator, dataset)

Compute estimator performance on input dataset.

copy(self)

Returns a shallow copy of current class.

create([class_item])

This method creates an instance of a class with given type.

deepcopy(self)

Returns a deep copy of current class.

evaluate_params(self, estimator, dataset, …)

Evaluate parameters for input estimator on input dataset.

get_class_from_type(class_type)

Return the class associated with input type.

get_params(self)

Returns the dictionary of class hyperparameters.

get_state(self)

Returns the object state dictionary.

get_subclasses()

Get all the subclasses of the calling class.

list_class_types()

This method lists all types of available subclasses of calling one.

load(path)

Loads object from file.

load_state(self, path)

Sets the object state from file.

save(self, path)

Save class object to file.

save_state(self, path)

Store the object state to file.

set(self, param_name, param_value[, copy])

Set a parameter of the class.

set_params(self, params_dict[, copy])

Set all parameters passed as a dictionary {key: value}.

set_state(self, state_dict[, copy])

Sets the object state using input dictionary.

timed([msg])

Timer decorator.

abstract compute_performance(self, estimator, dataset)[source]

Compute estimator performance on input dataset.

This must be reimplemented by subclasses.

Parameters
estimatorCClassifier

The classifier that we want evaluate.

datasetCDataset

Dataset that we want use for evaluate the classifier.

Returns
scorefloat

Performance score of estimator.

evaluate_params(self, estimator, dataset, parameters, pick='first', n_jobs=1)[source]

Evaluate parameters for input estimator on input dataset.

Parameters
estimatorCClassifier

The classifier for witch we want chose best parameters.

datasetCDataset

Dataset to be used for evaluating parameters.

parametersdict

Dictionary with each entry as {parameter: list of values to test}.

pick{‘first’, ‘last’, ‘random’}, optional

Defines which of the best parameters set pick. Usually, ‘first’ (default) correspond to the smallest parameters while ‘last’ correspond to the biggest. The order is consistent to the parameters dict passed as input.

n_jobsint, optional

Number of parallel workers to use. Default 1. Cannot be higher than processor’s number of cores.

Returns
best_param_dictdict

A dictionary with the best value for each evaluated parameter.

best_valueany

Metric value obtained on validation set by the estimator.

CPerfEvaluatorXVal

class secml.ml.peval.c_perfevaluator_xval.CPerfEvaluatorXVal(splitter, metric)[source]

Bases: secml.ml.peval.c_perfevaluator.CPerfEvaluator

Evaluate the best estimator parameters using Cross-Validation.

Parameters
splitterCXVal or str

XVal object to be used for splitting the dataset into train and validation.

metricCMetric or str

Name of the metric that we want maximize / minimize.

Attributes
class_type‘xval’

Defines class type.

Methods

compute_performance(self, estimator, dataset)

Split data in folds and return the mean estimator performance.

copy(self)

Returns a shallow copy of current class.

create([class_item])

This method creates an instance of a class with given type.

deepcopy(self)

Returns a deep copy of current class.

evaluate_params(self, estimator, dataset, …)

Evaluate parameters for input estimator on input dataset.

get_class_from_type(class_type)

Return the class associated with input type.

get_params(self)

Returns the dictionary of class hyperparameters.

get_state(self)

Returns the object state dictionary.

get_subclasses()

Get all the subclasses of the calling class.

list_class_types()

This method lists all types of available subclasses of calling one.

load(path)

Loads object from file.

load_state(self, path)

Sets the object state from file.

save(self, path)

Save class object to file.

save_state(self, path)

Store the object state to file.

set(self, param_name, param_value[, copy])

Set a parameter of the class.

set_params(self, params_dict[, copy])

Set all parameters passed as a dictionary {key: value}.

set_state(self, state_dict[, copy])

Sets the object state using input dictionary.

timed([msg])

Timer decorator.

compute_performance(self, estimator, dataset)[source]

Split data in folds and return the mean estimator performance.

Parameters
estimatorCClassifier

The Classifier that we want evaluate

datasetCDataset

Dataset that we want use for evaluate the classifier

Returns
scorefloat

Mean performance score of estimator computed on the K-Folds.

CPerfEvaluatorXValMulticlass

class secml.ml.peval.c_perfevaluator_xval_multiclass.CPerfEvaluatorXValMulticlass(splitter, metric)[source]

Bases: secml.ml.peval.c_perfevaluator.CPerfEvaluator

Evaluate the best parameters for each single binary classifier using Cross-Validation.

Parameters
splitterCXVal or str

XVal object to be used for splitting the dataset into train and validation.

metricCMetric or str

Name of the metric that we want maximize / minimize.

Attributes
class_type‘xval-multiclass’

Defines class type.

Methods

compute_performance(self, estimator, dataset)

Split data in folds and return the mean estimator performance.

copy(self)

Returns a shallow copy of current class.

create([class_item])

This method creates an instance of a class with given type.

deepcopy(self)

Returns a deep copy of current class.

evaluate_params(self, estimator, dataset, …)

Evaluate parameters for input estimator on input dataset.

get_class_from_type(class_type)

Return the class associated with input type.

get_params(self)

Returns the dictionary of class hyperparameters.

get_state(self)

Returns the object state dictionary.

get_subclasses()

Get all the subclasses of the calling class.

list_class_types()

This method lists all types of available subclasses of calling one.

load(path)

Loads object from file.

load_state(self, path)

Sets the object state from file.

save(self, path)

Save class object to file.

save_state(self, path)

Store the object state to file.

set(self, param_name, param_value[, copy])

Set a parameter of the class.

set_params(self, params_dict[, copy])

Set all parameters passed as a dictionary {key: value}.

set_state(self, state_dict[, copy])

Sets the object state using input dictionary.

timed([msg])

Timer decorator.

compute_performance(self, estimator, dataset)[source]

Split data in folds and return the mean estimator performance.

Parameters
estimatorCClassifier

The Classifier that we want evaluate

datasetCDataset

Dataset that we want use for evaluate the classifier

Returns
scoreslist

Mean performance score of each binary estimator computed on the K-Folds.