secml.ml.peval¶
CPerfEvaluator¶
-
class
secml.ml.peval.c_perfevaluator.
CPerfEvaluator
(splitter, metric)[source]¶ Bases:
secml.core.c_creator.CCreator
Evaluate the best parameters for input estimator.
- Parameters
- splitterCDataSplitter or str
Object to use for splitting the dataset into train and validation.
- metricCMetric or str
Name of the metric that we want maximize / minimize.
- Attributes
class_type
Defines class type.
logger
Logger for current object.
verbose
Verbosity level of logger output.
Methods
compute_performance
(self, estimator, dataset)Compute estimator performance on input dataset.
copy
(self)Returns a shallow copy of current class.
create
([class_item])This method creates an instance of a class with given type.
deepcopy
(self)Returns a deep copy of current class.
evaluate_params
(self, estimator, dataset, …)Evaluate parameters for input estimator on input dataset.
get_class_from_type
(class_type)Return the class associated with input type.
get_params
(self)Returns the dictionary of class hyperparameters.
get_state
(self)Returns the object state dictionary.
get_subclasses
()Get all the subclasses of the calling class.
list_class_types
()This method lists all types of available subclasses of calling one.
load
(path)Loads object from file.
load_state
(self, path)Sets the object state from file.
save
(self, path)Save class object to file.
save_state
(self, path)Store the object state to file.
set
(self, param_name, param_value[, copy])Set a parameter of the class.
set_params
(self, params_dict[, copy])Set all parameters passed as a dictionary {key: value}.
set_state
(self, state_dict[, copy])Sets the object state using input dictionary.
timed
([msg])Timer decorator.
-
abstract
compute_performance
(self, estimator, dataset)[source]¶ Compute estimator performance on input dataset.
This must be reimplemented by subclasses.
- Parameters
- estimatorCClassifier
The classifier that we want evaluate.
- datasetCDataset
Dataset that we want use for evaluate the classifier.
- Returns
- scorefloat
Performance score of estimator.
-
evaluate_params
(self, estimator, dataset, parameters, pick='first', n_jobs=1)[source]¶ Evaluate parameters for input estimator on input dataset.
- Parameters
- estimatorCClassifier
The classifier for witch we want chose best parameters.
- datasetCDataset
Dataset to be used for evaluating parameters.
- parametersdict
Dictionary with each entry as {parameter: list of values to test}.
- pick{‘first’, ‘last’, ‘random’}, optional
Defines which of the best parameters set pick. Usually, ‘first’ (default) correspond to the smallest parameters while ‘last’ correspond to the biggest. The order is consistent to the parameters dict passed as input.
- n_jobsint, optional
Number of parallel workers to use. Default 1. Cannot be higher than processor’s number of cores.
- Returns
- best_param_dictdict
A dictionary with the best value for each evaluated parameter.
- best_valueany
Metric value obtained on validation set by the estimator.
CPerfEvaluatorXVal¶
-
class
secml.ml.peval.c_perfevaluator_xval.
CPerfEvaluatorXVal
(splitter, metric)[source]¶ Bases:
secml.ml.peval.c_perfevaluator.CPerfEvaluator
Evaluate the best estimator parameters using Cross-Validation.
- Parameters
- splitterCXVal or str
XVal object to be used for splitting the dataset into train and validation.
- metricCMetric or str
Name of the metric that we want maximize / minimize.
- Attributes
class_type
‘xval’Defines class type.
Methods
compute_performance
(self, estimator, dataset)Split data in folds and return the mean estimator performance.
copy
(self)Returns a shallow copy of current class.
create
([class_item])This method creates an instance of a class with given type.
deepcopy
(self)Returns a deep copy of current class.
evaluate_params
(self, estimator, dataset, …)Evaluate parameters for input estimator on input dataset.
get_class_from_type
(class_type)Return the class associated with input type.
get_params
(self)Returns the dictionary of class hyperparameters.
get_state
(self)Returns the object state dictionary.
get_subclasses
()Get all the subclasses of the calling class.
list_class_types
()This method lists all types of available subclasses of calling one.
load
(path)Loads object from file.
load_state
(self, path)Sets the object state from file.
save
(self, path)Save class object to file.
save_state
(self, path)Store the object state to file.
set
(self, param_name, param_value[, copy])Set a parameter of the class.
set_params
(self, params_dict[, copy])Set all parameters passed as a dictionary {key: value}.
set_state
(self, state_dict[, copy])Sets the object state using input dictionary.
timed
([msg])Timer decorator.
-
compute_performance
(self, estimator, dataset)[source]¶ Split data in folds and return the mean estimator performance.
- Parameters
- estimatorCClassifier
The Classifier that we want evaluate
- datasetCDataset
Dataset that we want use for evaluate the classifier
- Returns
- scorefloat
Mean performance score of estimator computed on the K-Folds.
CPerfEvaluatorXValMulticlass¶
-
class
secml.ml.peval.c_perfevaluator_xval_multiclass.
CPerfEvaluatorXValMulticlass
(splitter, metric)[source]¶ Bases:
secml.ml.peval.c_perfevaluator.CPerfEvaluator
Evaluate the best parameters for each single binary classifier using Cross-Validation.
- Parameters
- splitterCXVal or str
XVal object to be used for splitting the dataset into train and validation.
- metricCMetric or str
Name of the metric that we want maximize / minimize.
- Attributes
class_type
‘xval-multiclass’Defines class type.
Methods
compute_performance
(self, estimator, dataset)Split data in folds and return the mean estimator performance.
copy
(self)Returns a shallow copy of current class.
create
([class_item])This method creates an instance of a class with given type.
deepcopy
(self)Returns a deep copy of current class.
evaluate_params
(self, estimator, dataset, …)Evaluate parameters for input estimator on input dataset.
get_class_from_type
(class_type)Return the class associated with input type.
get_params
(self)Returns the dictionary of class hyperparameters.
get_state
(self)Returns the object state dictionary.
get_subclasses
()Get all the subclasses of the calling class.
list_class_types
()This method lists all types of available subclasses of calling one.
load
(path)Loads object from file.
load_state
(self, path)Sets the object state from file.
save
(self, path)Save class object to file.
save_state
(self, path)Store the object state to file.
set
(self, param_name, param_value[, copy])Set a parameter of the class.
set_params
(self, params_dict[, copy])Set all parameters passed as a dictionary {key: value}.
set_state
(self, state_dict[, copy])Sets the object state using input dictionary.
timed
([msg])Timer decorator.
-
compute_performance
(self, estimator, dataset)[source]¶ Split data in folds and return the mean estimator performance.
- Parameters
- estimatorCClassifier
The Classifier that we want evaluate
- datasetCDataset
Dataset that we want use for evaluate the classifier
- Returns
- scoreslist
Mean performance score of each binary estimator computed on the K-Folds.