| Multivariate Pattern Analysis in Python

### Navigation

- index
- next |
- previous |
- ExpPsy Project |
- NeuroDebian Project |
- PyMVPA 0.4.x Home |
- PyMVPA Home |
- Documentation »

This content refers to the previous stable release of PyMVPA. Please visit www.pymvpa.org for the most recent version of PyMVPA and its documentation.

# Feature Selection

> This section has been contributed by James M. Hughes.

It is often the case in machine learning problems that we wish to reduce a feature space of high dimensionality into something more manageable by selecting only those features that contribute most to classification performance. Feature selection methods attempt to achieve this goal in an algorithmic fashion.

PyMVPA’s flexible framework allows various feature selection methods to take place within a small block of code. `FeatureSelectionClassifier` extends the basic classifier framework to allow for the use of arbitrary methods of feature selection according to whatever ranking metric, feature selection criteria, and stopping criterion the user chooses for a given application. Examples of the code/classification algorithms presented here can be found in mvpa/clfs/warehouse.py.

More formally, a `FeatureSelectionClassifier` is a meta-classifier. That is, it is not a classifier itself – it can take any slave `Classifier`, perform some feature selection in advance, select those features, and then train the provided slave `Classifier` on those features. Externally, however, it looks like a `Classifier`, in that it fulfills the specialization of the Classifier base class. The following are the relevant arguments to the constructor of such a `Classifier`:

clf: `Classifier`
  classifier based on which mask classifiers is created
feature_selection: FeatureSelection
  whatever feature selection is considered best
testdataset: `Dataset` (optional)
  dataset which would be given on call to feature_selection

Let us turn out attention to the second argument, FeatureSelection. As noted above, this feature selection can be arbitrary and should be chosen appropriately for the task at hand. For example, we could perform a one-way ANOVA statistic to select features, then keep only the most important 5% of them. It is crucial to note that, in PyMVPA, the way in which features are selected (in this example by keeping only 5% of them) is wholly independent of the way features are ranked (in this example, by using a one-way ANOVA). Feature selection using this method could be accomplished using the following code (from mvpa/clfs/warehouse.py):

    >>> from mvpa.suite import *
    >>> FeatureSelection = SensitivityBasedFeatureSelection(
    ...     OneWayAnova(),
    ...     FractionTailSelector(0.05, mode='select', tail='upper'))

A more interesting analysis is one in which we use the weights (hyperplane coefficients) to rank features. This allows us to use the same classifier to train the selected features as we used to select them:

It bears mentioning at this point that caution must be exercised when selecting features. The process of feature selection must be performed on an independent training dataset: it is not possible to select features using the entire dataset, re-train a classifier on a subset of the original data (but using only the selected features) and then test on a held-out testing dataset. This results in an obvious positive bias in classification performance. PyMVPA allows for easy dataset splitting, however, so creating independent training and testing datasets is easily accomplished, for instance using an `NFoldSplitter`, `OddEvenSplitter`, etc.

## Recursive Feature Elimination

Recursive feature elimination (RFE, applied to fMRI data in (Hanson et al., 2008)) is a technique that falls under the larger umbrella of feature selection. Recursive feature elimination specifically attempts to reduce the number of selected features used for classification in the following way:

- A classifier is trained on a subset of the data and features are ranked according to an arbitrary metric.
- Some amount of those features is either selected or discarded according to a pre-selected rule.
- The classifier is retrained and features are once again ranked; this process continues until some criterion determined textit{a priori} (such as classification error) is reached.
- One or more classifiers trained only on the final set of selected features are used on a generalization dataset and performance is calculated.

PyMVPA’s flexible framework allows each of these steps to take place within a small block of code. To actually perform recursive feature elimination, we consider two separate analysis scenarios that deal with a pre-selected training dataset:

- We split the training dataset into an arbitrary number of independent datasets and perform RFE on each of these; the sensitivity analysis of features is performed independently for each split and features are selected based on those independent measures.
- We split the training dataset into an arbitrary number of independent datasets (as before), but we average the feature sensitivities and select which features to prune/select based on that one average measure.

We will concentrate on the second approach. The following code can be used to perform such an analysis:

    >>> rfesvm_split = SplitClassifier(LinearCSVMC())
    >>> clf = \
    ...  FeatureSelectionClassifier(
    ...   clf = LinearCSVMC(),
    ...   # on features selected via RFE
    ...   feature_selection = RFE(
    ...       # based on sensitivity of a clf which does splitting internally
    ...       sensitivity_analyzer=rfesvm_split.getSensitivityAnalyzer(),
    ...       transfer_error=ConfusionBasedError(
    ...          rfesvm_split,
    ...          confusion_state="confusion"),
    ...          # and whose internal error we use
    ...       feature_selector=FractionTailSelector(
    ...                          0.2, mode='discard', tail='lower'),
    ...                          # remove 20% of features at each step
    ...       update_sensitivity=True),
    ...       # update sensitivity at each step
    ...   descr='LinSVM+RFE(splits_avg)' )

The code above introduces the `SplitClassifier`, which in this case is yet another meta-classifier that takes in a `Classifier` (in this case a LinearCSVMC) and an arbitrary `Splitter` object, so that the dataset can be split in whatever way the user desires. Prior to training, the `SplitClassifier` splits the training dataset, dedicates a separate classifier to each split, trains each on the training part of the split, and then computes transfer error on the testing part of the split. If a `SplitClassifier` instance is later on asked to predict some new data, it uses (by default) the MaximalVote strategy to derive an answer. A summary about the performance of a `SplitClassifier` internally on each split of the training dataset is available by accessing the confusion state variable.

To summarize somewhat, RFE is just one method of feature selection, so we use a `FeatureSelectionClassifier` to facilitate this. To parameterize the RFE process, we refer above to the following:

sensitivity_analyzer
  in this case just the default from a linear C-SVM (the SVM weights), taken as an average over all splits (in accordance with scenario 2 as above)
transfer_error
  confusion-based error that relies on the confusion matrices computed during splitting of the dataset by the `SplitClassifier`; this is used to provide a value that can be compared against a stopping criterion to stop eliminating features
feature_selector
  in this example we simply discard the 20% of features deemed least important
update_sensitivity
  true to retrain the classifiers each time we eliminate features; should be false if a non-classifier-based sensitivity measure (such as one-way ANOVA) is used

As has been shown, recursive feature elimination is an easy-to-implement, flexible, and powerful tool within the PyMVPA framework. Various ranking methods for selecting features have been discussed. Additionally, several analysis scenarios have been presented, along with enough requisite knowledge that the user can plug in whatever classifiers, error metrics, or sensitivity measures are most appropriate for the task at hand.

## Incremental Feature Search

IFS

(to be written)

### Navigation

- index
- next |
- previous |
- ExpPsy Project |
- NeuroDebian Project |
- PyMVPA 0.4.x Home |
- PyMVPA Home |
- Documentation »