Evaluation & Experiments¶

class pynlpl.evaluation.AbstractExperiment(inputdata=None, **parameters)¶

defaultparameters()¶

delete()¶

done(warn=True)¶: Is the subprocess done?

duration()¶

run()¶

sample(size)¶: Return a sample of the input data

score()¶

start()¶: Start as a detached subprocess, immediately returning execution to caller.

startcommand(command, cwd, stdout, stderr, *arguments, **parameters)¶

wait()¶

class pynlpl.evaluation.ClassEvaluation(goals=[], observations=[], missing={}, encoding='utf-8')¶

accuracy(cls=None)¶

append(goal, observation)¶

auc(cls=None, macro=False)¶

compute()¶

confusionmatrix(casesensitive=True)¶

fp_rate(cls=None, macro=False)¶

fscore(cls=None, beta=1, macro=False)¶

outputmetrics()¶

precision(cls=None, macro=False)¶

recall(cls=None, macro=False)¶

specificity(cls=None, macro=False)¶

tp_rate(cls=None, macro=False)¶

class pynlpl.evaluation.ConfusionMatrix(tokens=None, casesensitive=True, dovalidation=True)¶: Confusion Matrix

class pynlpl.evaluation.ExperimentPool(size)¶

append(experiment)¶

poll(haltonerror=True)¶

run(haltonerror=True)¶

start(experiment)¶

class pynlpl.evaluation.OrdinalEvaluation(goals=[], observations=[], missing={}, encoding='utf-8')¶

compute()¶

mae(cls=None)¶

rmse(cls=None)¶

class pynlpl.evaluation.ParamSearch(experimentclass, inputdata, parameterscope, poolsize=1, constraintfunc=None, delete=True)¶: A simpler version of ParamSearch without Wrapped Progressive Sampling

exception pynlpl.evaluation.ProcessFailed¶

class pynlpl.evaluation.WPSParamSearch(experimentclass, inputdata, size, parameterscope, poolsize=1, sizefunc=None, prunefunc=None, constraintfunc=None, delete=True)¶

ParamSearch with support for Wrapped Progressive Sampling

searchbest()¶

test(i=None)¶

pynlpl.evaluation.auc(x, y, reorder=False)¶

Compute Area Under the Curve (AUC) using the trapezoidal rule

This is a general fuction, given points on a curve. For computing the area under the ROC-curve, see auc_score().

Parameters:	x (array, shape = [n]) – x coordinates. y (array, shape = [n]) – y coordinates. reorder (boolean, optional (default=False)) – If True, assume that the curve is ascending in the case of ties, as for an ROC curve. If the curve is non-ascending, the result will be wrong.
Returns:	auc
Return type:	float

Examples

>>> import numpy as np
>>> from sklearn import metrics
>>> y = np.array([1, 1, 2, 2])
>>> pred = np.array([0.1, 0.4, 0.35, 0.8])
>>> fpr, tpr, thresholds = metrics.roc_curve(y, pred, pos_label=2)
>>> metrics.auc(fpr, tpr)
0.75