When talking about machine learning, you may encounter many terminologies such as such as “online learning,” “active learning,” and “human in the loop” methods. Here are some of my thoughts on the relationship between interactive machine learning and machine learning in general. This is an extract from my answers to my comprehensive exam.
Traditionally machine-learning has been classified into supervised and unsupervised learning families. In supervised learning the training data, , consists of N sets of input vectors along with the desired output provided by a teacher:
where, is a d-dimensional feature vector
and is the known label for it
The task is to learn a function, , which can be used on unseen data.
In case of unsupervised learning, our data consists vector values s but no target outputs s. Common tasks under this category include clustering, density estimation and discovering patterns. A combination of these two is called semi-supervised learning, which has a mixture of labeled and unlabeled data in the training set. The algorithm assigns labels for missing data points using certain similarity measures.
While researchers are actively looking at improving the unsupervised learning techniques, supervised machine learning has been the dominant form of learning till date. However, the traditional supervised algorithms assume that we have readily available training data along with the labels. They are not concerned with the process of obtaining the target values s in the training dataset. Often obtaining labelled data is one of the main bottlenecks in applying these techniques in domain specific applications. For example in biomedical informatics, we can have small training sets which are very skewed and contain rare events. This makes it challenging to build models for these domains. Further, they do not have easy ways to correct problems incase learning deviates from the desired concept. As a result there was a need for another class of algorithms called the “human-in-the-loop” algorithms that could leverage a human-expert’s capabilities during the learning process. These algorithms can optimize their learning behavior through their interactions with the human agents. Not all the human-in-the-loop algorithms are similar to the supervised machine learning algorithms in their problem formulation. They have been applied to various categories of learning and optimization problems where human expertise can help in reducing the complexity of solving them.
Interactive Machine Learning is defined as a process of building machine learning models iteratively by end-users. It allows the users to check the model outputs, make corrections by giving feedback to build revised models and verify changes. This feedback loop allows them to refine the models further with every iteration. Some early examples for this definition include applications in image segmentation, interactive document clustering, document retrieval, bug triaging and even music composition. You can read more about this in the article titled "Power to the People: The Role of Humans in Interactive Machine Learning" (Amershi et.al., 2014).
Interactive machine learning builds on a variety of styles of learning algorithms:
- Reinforcement Learning: In this class of learning we still want to learn but we see samples of but no target output . Instead of , we get a feedback from a critic about the goodness of the predicted output. The goal of the learner is to optimize for the reward function by selecting outputs that get best scores from the critics. The critic can be a human or any other agent. There need not be a human-in-the-loop for the algorithm to be classified under reinforcement learning. Several recent examples of this type include building systems that learn to play games such as Flappy Bird, Mario etc.
- Active Learning: Active learning algorithms try to optimize for the number of training examples. Such an algorithm would ask an oracle to give labels such that it can achieve higher accuracy with smallest number of queries. These queries contain a batch of examples to be labelled. For example, in SVMs, one could select training sets for labeling that are closest to the margin hyperplanes to reduce the number of queries.
- Online Algorithms: Online learning algorithms are used when training data is available in sequential order, say due to the nature of the problem or memory constraints, as opposed to a batch learning technique where all the training data is available at once. The algorithm must adapt to the continuous stream of data made available to it. Formulating the learning problem to handle this situation forms the core of designing algorithms under this class.
A commonly used example would be the online gradient descent method for linear regression: Suppose we are trying to learn the parameters for . We update the weights when we receive the th training example by taking the gradient of the defined error function:
. Where, is defined as the learning rate.
Interactive machine learning methods can include all or some of these learning techniques. The common property between all the interactive machine learning methods is the tight interaction loop between the human and the learning algorithm. Most of the effort in interactive machine learning has been about designing interactions for each step of this loop. My work on interactive clinical and legal text analysis also follows this pattern.