2
$\begingroup$

We currently have a tag with the description (written by myself)

For questions related to the concept of testing or evaluating machine learning models and algorithms.

In supervised learning, testing usually refers to computing some performance measure (e.g. accuracy) on some dataset that was not used for training or validation (i.e. early stopping, k-fold cross-validation, or hyper-parameter optimization). It seems to me that "evaluation" doesn't necessarily refer to this specific meaning of testing, but it's more general (or maybe I should say vague or not well-defined), and it may refer also to any type of measurement that quantifies some property of the model or algorithm. So, given this view, validation could also be some form of "evaluation" (for example, it would be a way to measure whether the model is over-fitting or not).

Let's take, for example, this question: How to evaluate an RL algorithm when used in a game?. The user is asking "how to evaluate this RL algorithm". I've tagged the question with , but, at this point, I'm not sure whether this is a good idea or not. So, should "evaluation" be a synonym for "testing"? If not, what should we use it for?

$\endgroup$
1
$\begingroup$

Hmmm I don't feel like they should necessarily be the same. In most machine learning contexts especially, as you say, I would view "evaluation" as something broader or more general than "testing". I guess I'd associate "testing" specifically with the classic train/validation/test data splits.

I could also see testing being used for an entirely different purpose though: testing whether an implementation is correct. Think of things like unit testing. A lot of that would probably just be about software engineering and be off-topic, but I could see questions about testing the correctness of specific things in AI being relevant. For example: How can I test that my gradient computations / gradient descent implementation is correct? (and an answer could get into things like numerically approximating gradients and checking that they're about the same as the analytically computed ones)

$\endgroup$
3
  • $\begingroup$ Ok, so what should we do? Should we reserve testing for the specific meaning of testing when we split the data into training/validation/testing datasets, or should we use it for a more general scope (as it seems to be the case now, according to the current description)? Should we have another tag evaluation for general "evaluations of algorithms and models"? That doesn't look necessary, given that "evaluation", as we seem to agree, is not a well-defined term. Maybe we should wait for a third person to give their perspective before taking action. $\endgroup$ – nbro Mod Feb 1 at 19:33
  • 1
    $\begingroup$ @nbro Keeping testing as is, i.e. a bit more general, seems fine to me. I don't think a distinct evaluation tag is necessary. It could be a synonym... except if people start regularly using it with the intention of referring to like a "heuristic evaluation function". But currently it's unused, so that doesn't seem to be happening? $\endgroup$ – Dennis Soemers Mod Feb 1 at 19:45
  • 1
    $\begingroup$ A few days/weeks ago, I had also introduced the tag evaluation-functions (which can be combined e.g. with the tag search), so I guess we can just make evaluation a synonym for testing, to avoid people using multiple tags when they basically refer to more or less the same thing. In case something else pops up that we didn't think about, we can remove the synonym. $\endgroup$ – nbro Mod Feb 2 at 1:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .