LECA.active_learning.ActiveLearner

class LECA.active_learning.ActiveLearner(wf: WorkFlow, data_pool: DataFrame | ndarray | None = None)

Bases: object

Interface object between LECAs WorkFlow object and the PyALAF library.

Parameters:

• wf (LECA WorkFlow object) – An initially trained LECA workflow, providing data and model information.

• data_pool (Optional[Union[pd.DataFrame, np.ndarray]]) –

  A pool of data that may be considered for Active Learning. If None is provided, population-based active learning will be performed instead of pool-based active learning.

  Default value None.

wf

An initially trained LECA workflow, providing data and model information.

Type:

LECA WorkFlow object

active_set_X

Features of the data set used for active learning. The models are trained based on these data and further data is selected.

Type:

pd.DataFrame

active_set_y

Objectives for the active learning set.

Type:

pd.DataFrame

test_set_X

Features for a test set, similar to the WorkFlows validation holdout set.

Type:

pd.DataFrame

test_set_y

Objectives for a test set, similar to the WorkFlows validation holdout set.

Type:

pd.DataFrame

X

Features for the active set and test set together.

Type:

pd.DataFrame

y

Objectives for the active set and test set together.

Type:

pd.DataFrame

data_pool

A pool of data that may be considered for Active Learning. If a pool is provided, data from this pool might be added to the active set. If None is provided, population-based active learning will be performed instead of pool-based active learning.

Type:

Union[pd.DataFrame, np.ndarray]

objective_funcs

List of all objective functions.

Type:

List[str]

estimator_names

Name of all estimators used in the LECA workflow.

Type:

List[str]

suggested_data

Suggested data to add to the training set, which was not evaluated yet.

Type:

np.ndarray

collected_data

Collected data, which was already evaluated but not added to the LECA workflow.

Type:

np.ndarray

__init__(wf: WorkFlow, data_pool: DataFrame | ndarray | None = None) → None

automatic_al(estimator: str | None = None, objective_funcs: str | List[str] | None = None, acquisition_function: str | List[str] | None = 'ideal', data_evaluation_models: callable | None = None, aggregation_function: callable | None = None, alpha: float = 10.0, lim: ndarray | None = None, batch_size: int = 10, active_learning_steps: int = 2, random_state: int | None = None, opt_method: str | None = 'PSO', **kwargs)

Perform automatic active learning and labeling of data

Parameters:

• estimator (Optional[str], optional) –

  Name of the LECA estimator to use for AL

  Default value None

• objective_funcs (Optional[Union[str, List[str]]], optional) –

  Objective functions to use for AL.

  Default value None

• acquisition_function (Optional[Union[str, List[str]]], optional) –

  Name of the acquisition function. Chose from GSx, GSy, iGS, ideal, max for non-GPR models and UCB, PI, EI additionally for GPR models.

  Default value ideal

• data_evaluation_models (Optional[callable], optional) –

  Definition of a data_evaluation_model, which can be used to automatically combine active learning and data acquisition. If None is used, an Interactive Model is chosen, for which data needs to be entered manually.

  Default value None

• aggregation_function (Optional[callable], optional) –

  An aggregation function, which combines multiple objectives into a single one.

  Default value None

• alpha (float, optional) –

  Hyperparameter for the acquisition function. Not all acquisition functions make use of it. For more details see the PyALAF documentation.

  Default value 10

• lim (Optional[np.ndarray], optional) –

  Boundaries for the search space of active learning. None means an automatic determination of the boundaries. The format needs to be [[l1, l2, …], [u1, u2, …]]

  Default value None

• batch_size (int, optional) –

  Size of the batch to suggest via AL

  Default value 10

• active_learning_steps (int, optional) –

  Size of the batch to suggest via AL

  Default value 2

  Default value None

• opt_method (Optional[str], optional) –

  Optimization method to search for the maximum of acquisition functions. Possible options are PSO (Particle Swarm Optimization, global optimization) and lbfgs (local optimization).

  Default value PSO

Returns:

• np.nd_array – All feature values that belong to sampled data points in the order of sampling.

• dict – Results of the AL.

Raises:

Exception – Pool-based learning not implemented yet. The data_pool attribute must be None.

data_importance(estimators: str | List[str] | None = None, objective_funcs: str | List[str] | None = None, acquisition_function: str | List[str] | None = 'ideal', aggregation_function: callable | None = None, lim_features: List[float] | None = [-1, 1], repeat: int = 1, initial_samples: int = 10, alpha: float | List[float] = 10.0, shuffle_sets=False, random_state: int | None = None, initialization: str | None = 'random', training_subset=None, select_test_from_sets_equally=True, cv=None, sets_for_selection=None, **kwargs)

Perform an analysis of how much data of all collected data is really needed. An initial model is trained and afterwards sequentially data points are added to the pool of training data. Metrics are evaluated for each step.

Parameters:

• estimators (Optional[Union[str, List[str]]]) –

  String or list with model name(s) to perform datasize performance analysis. None will use all workflow models.

  Default value None.

• objective_funcs (Optional[Union[str, List[str]]]) –

  String or list of strings declaring which objective functions on which to perform datasize performance analysis. When None is passed, defaults to all objective functions. If a list of objective functions are passed, the function returns a list of objects.

  Default value None.

• acquisition_function (Optional[Union[str, List[str]]]) –

  String or list of strings declaring which acquisition functions should be used. The active learning is performed for each of the listed acquisition functions.

  Valid acquisition functions for GPR models are: ucb, poi, ei, GSx, GSy, iGS, ideal, qbc, std

  Valid acquisition functions for non-GPR models are: GSx, GSy, iGS, ideal, qbc

  Default value ideal

• aggregation_function (Optional[Callable]) –

  If several objectives are given data may be selected based on the improvement for all objectives instead of only one objective. This function defines how different objectives are combined to a single objective. It must have uncert as a boolean parameter, which defines how to handle the calculation of the uncertainty for the combined objective, which might be different to the calculation of the combined objective itself. If None is provided, active learning will be performed for each objective seperately.

  Default value None

• repeat (Optional[int]) –

  Number of times to repeat the active learning.

  Default value 1.

• initial_samples (Optional[int]) –

  Number of data points used to fit an initial model.

  Default value 10.

• alpha (Optional[Union[float, List[float]]]) –

  Hyperparameter for the active learning algorithm. For following acquisition functions a hyperparameter is used: ei: Weighing of exploration vs. exploitation ideal: IDEAL hyperparameter qbc: Number of models in the ensemble

  Although other models ignore this parameter a value must be given.

  Default value 10.

• suffle_sets (Optional[bool]) –

  When True the trainining and validation holdout set from the workflow are mixed. Use this, when the result depends extremely on the chosen validation holdout set.

  Default value False.

• training_subset (Optional[pd.DataFrame]) –

  You should always use a workflow containing all data available. To use only a subset of data for training the models for datasize performans, you can specify training_subset. This way, the test set will always be selected from all available data, even if the models are only trained on part of the data. This ensures consistency in test errors.

  Example: We assume data was collected during 3 Active Learning iterations. Data named Gen1 corresponds to the first, Gen2 to the second and Gen3 to the third iteration. The initial data is named Gen0. We will always take data from Gen0 to Gen3 for testing. Let’s say we want to evaluate the improvement of the models when adding Gen2 data. This means, we use Gen0 and Gen1 as initial data (Initial samples needs to be specified, together with initialization=’data’). Then the datasize_performance is only run for Gen2.

  If you want to perform a similar analysis for Gen3, the test errors will be consistent.

  Default value None.

• random_state (Optional[int]) –

  Sets a numpy random seed for reproducibility.

  Default value None.

Returns:

score_dict – Dictionary with active learning results

Return type:

Dict

get_collected_data()

Get collected data.

Returns:

Data Frame with all data collected during single or multiple active learning runs if not already added to the LECA workflow.

Return type:

pd.DataFrame

get_suggested_data()

Get suggested data.

Returns:

Array with feature values of suggested data

Return type:

np.nd_array

single_step_al(estimator: str | None = None, objective_funcs: str | List[str] | None = None, acquisition_function: str | List[str] | None = 'ideal', aggregation_function: callable | None = None, alpha: float = 10.0, lim: ndarray | None = None, batch_size: int = 10, random_state: int | None = None, opt_method: str | None = 'PSO', **kwargs)

Performs a single active learning step.

Parameters:

• estimator (Optional[str], optional) –

  Name of the LECA estimator to use for AL

  Default value None

• objective_funcs (Optional[Union[str, List[str]]], optional) –

  Objective functions to use for AL.

  Default value None

• acquisition_function (Optional[Union[str, List[str]]], optional) –

  Name of the acquisition function. Chose from GSx, GSy, iGS, ideal, max for non-GPR models and UCB, PI, EI additionally for GPR models.

  Default value ideal

• aggregation_function (Optional[callable], optional) –

  An aggregation function, which combines multiple objectives into a single one.

  Default value None

• alpha (float, optional) –

  Hyperparameter for the acquisition function. Not all acquisition functions make use of it. For more details see the PyALAF documentation.

  Default value 10

• lim (Optional[np.ndarray], optional) –

  Boundaries for the search space of active learning. None means an automatic determination of the boundaries. The format needs to be [[l1, l2, …], [u1, u2, …]]

  Default value None

• batch_size (int, optional) –

  Size of the batch to suggest via AL

  Default value 10

• random_state (Optional[int], optional) –

  Random state

  Default value None

• opt_method (Optional[str], optional) –

  Optimization method to search for the maximum of acquisition functions. Possible options are PSO (Particle Swarm Optimization, global optimization) and lbfgs (local optimization).

  Default value PSO

Returns:

• np.nd_array – All feature values that belong to sampled data points in the order of sampling.

• dict – Results of the AL.

Raises:

Exception – Pool-based learning not implemented yet. The data_pool attribute must be None.

update_wf()

Add the collected data to the LECA workflow.