path: root/report/paper.md

# K-means Codebook

A common technique for codebook generation involves utilising K-means clustering on a sample of the
image descriptors. In this way descriptors may be mapped to *visual* words which lend themselves to
binning and therefore the creation of bag-of-words histograms for the use of classification.

In this coursework 100-thousand random SIFT descriptors (with 128 dimenions) of the `Caltech_101` dataset are used to build the K-means visual vocabulary.

Both training and testing use 15 randomly selected images from the 10 available classes.

## Vocabulary size 

The number of clusters or the number of centroids determines the vocabulary size when creating the codebook with the K-means method. Each descriptor is mapped to the nearest centroid, and each descriptor belonging to that cluster is mapped to the same *visual word*. This allows similar descriptors to be mapped to the same word, allowing for comparison through bag-of-words techniques.

## Bag-of-words histogram of descriptor vectors

An example of histograms for training and testing images is shown on figure \ref{fig:histo_tr}, computed with a vocabulary size of 100. The histograms of the same class appear to have comparable magnitudes for their respective keywords, demonstrating they have a similar number of descriptors which map to each of the clusters. The effect of the vocabulary size (as determined by the number of K-means centroids) on the classification accuracy is shown in figure \ref{fig:km_vocsize}. We find that a small vocabulary size tends to misrepresent the information contained in the different patches, resulting in poor classification accuracy. Conversely a large vocabulary size (many K-mean centroids), may display overfitting. In our tests, we begin to observe a plateau after a cluster count of 60 on figure \ref{fig:km_vocsize}. The process of partitioning the input space into K distinct clusters is a form of **vector quantization**.

\begin{figure}
\begin{center}
\includegraphics[width=12em]{fig/kmeans_vocsize.pdf}
\includegraphics[width=12em]{fig/kmean_train_test_time.pdf}
\caption{Effect of vocabulary size; classification error (left) and time (right)}
\label{fig:km_vocsize}
\end{center}
\end{figure}


The time complexity of quantization with a K-means codebooks is $O(DNK)$, where N is the number of entities to be clustered (descriptors), D is the dimension (of the descriptors) and K is the cluster count [@km-complexity]. As the computation time is high, the tests use a subsample of descriptors to compute the centroids (a random selection of 100 thousand descriptors). An alternative method we tried is applying PCA to the descriptors vectors to improve time performance. However, the descriptor dimension of 128 is relatively small and as such we found PCA to be unnecessary.

K-means is a process that converges to local optima and heavily depends on the initialization values of the centroids[@kmean].
Initializing K-means is an expensive process, based on sequential attempts of centroids placement. Running for multiple instances significantly affects the computation process, leading to a linear increase in execution time. We did not observe increase in accuracy with more than one K-means clusters initializations, and therefore present results for accuracy and execution time with a single K-Means initialization.

\begin{figure}
\begin{center}
\includegraphics[width=12em]{fig/trainhist.pdf}
\includegraphics[width=12em]{fig/testhist.pdf}
\caption{Bag-of-words histograms; Training (left), Testing (right)}
\label{fig:histo_tr}
\end{center}
\end{figure}

# RF classifier 

We use a random forest classifier to label images based on the bag-of-words histograms. Random forests are an ensemble of randomly generated decision trees, whose performance depends on the ensemble size, tree depth, randomness and weak learner used.

## Hyperparameters tuning

Figure \ref{fig:km-tree-param} shows the effect of tree depth and number of trees, when classifying a bag-of-words created by K-means with 100 cluster centers.
Optimal values for tree depth and number of trees were found to be respectively 5 and 100 as shown in figure \ref{fig:km-tree-param}. Running for multiple seeds instances shows an average accuracy of 80% for these two parameters, peaking at 84% in very specific cases.
We expect a large tree depth to lead into overfitting. However for the data analysed it is only possible to observe a plateau in classification performance.

\begin{figure}
\begin{center}
\includegraphics[width=12em]{fig/error_depth_kmean100.pdf}
\includegraphics[width=12em]{fig/trees_kmean.pdf}
\caption{K-means Classification error varying tree depth (left) and forest size (right)}
\label{fig:km-tree-param}
\end{center}
\end{figure}

Random forests will select a random number of features on which to apply a weak learner (such as axis aligned split) and then choose the best feature of the sampled ones to perform the split on, based on a given criteria (our results use the *Gini index*). The fewer features that are compared for each split the quicker the trees are built and the more random they are. Therefore the randomness parameter can be considered as the number of features used when making splits. We evaluate accuracy given different randomness when using a K-means vocabulary of size 100 in figure \ref{fig:kmeanrandom}. The results in the figure \ref{fig:kmeanrandom} also use a forest size of 100 as we inferred that this is the estimator count for which performance gains tend to plateau (when selecting $\sqrt{n}$ random features).
This parameter also affects correlation between trees. We expect trees to be more correlated when using a large number of features for splits.

\begin{figure}
\begin{center}
\includegraphics[width=12em]{fig/new_kmean_random.pdf}
\includegraphics[width=12em]{fig/p3_rand.pdf}
\caption{Classification error for different number of random features; K-means left, RF codebooks right}
\label{fig:kmeanrandom}
\end{center}
\end{figure}

Changing the randomness parameter had no significant effect on execution time. This may be attributed to the increased required tree depth to purify the training set.

Effects of vocabulary size on accuracy and time performance are shown in section I, figure \ref{fig:km_vocsize}. Time increases linearly with vocabulary size. Optimal number of cluster centers was found to be around 100, giving a good trade-off between time and accuracy performance. As shown in figure \ref{fig:km_vocsize} the classification error in fact does no plateau completely, despite experiencing a significant decrease in gradient.

## Weak Learner comparison

In figure \ref{fig:2pt} it is possible to notice an improvement in recognition accuracy by 2%,
with the two pixels test, achieving better results than the axis-aligned counterpart. The two-pixels
test theoretically brings a slight deacrease in time performance due to complexity, since it adds one dimension to the computation. It is difficult to measure in our case since it should be less than a second.

\begin{figure}
\begin{center}
\includegraphics[width=14em]{fig/2pixels_kmean.pdf}
\caption{K-means classification accuracy changing the type of weak learners}
\label{fig:2pt}
\end{center}
\end{figure}

Figure \ref{fig:km_cm} shows a confusion matrix for RF Classification on K-means coded descriptors with 256 centroids, a forest size of 100 and trees depth of 5. The reported accuracy for this case is 82%. Figure \ref{fig:km_succ} reports examples of failure and success cases obtained from this test, with the top performing classes being `trilobite` and `windsor_chair`. `Water_lilly` was the one that on average performed worst.

\begin{figure}
\begin{center}
\includegraphics[width=14em]{fig/e100k256d5_cm.pdf}
\caption{Confusion Matrix: K=256, ClassifierForestSize=100, Depth=5}
\label{fig:km_cm}
\end{center}
\end{figure}

# RF codebook

An alternative to codebook creation via K-means involves using an ensemble of totally random trees. We code each descriptor according to which leaf of each tree in the ensemble it is sorted to. This effectively performs an unsupervised quantization of our descriptors. The vocabulary size is determined by the number of leaves in each random tree multiplied by the ensemble size. The returned leaf node IDs are then binned into a histogram to create a bag-of-words, much like the one created using K-Means. From comparing execution times of K-means in figure \ref{fig:km_vocsize} and the RF codebook in figure \ref{fig:p3_voc} we observe considerable speed gains from utilising the RF codebook. This may be attributed to the reduced complexity of RF Codebook creation, 
which is $O(\sqrt{D} N \log K)$ compared to $O(DNK)$ for K-means. Codebook mapping given a created vocabulary is also quicker than K-means, $O(\log K)$ (assuming a balanced tree) vs $O(KD)$.

The effect of vocabulary size on classification accuracy can be observed both in figure \ref{fig:p3_voc}, in which we independently vary number of leaves and ensemble size, and figure \ref{fig:p3_colormap}, in which both parameters are varied simultaneously. It is possible to notice that these two parameters make classification accuracy plateau for *leaves*$>80$ and *estimators*$>100$. The peaks of 82% accuracy visible on the heatmap in figure \ref{fig:p3_colormap} are highly dependent on the seed and indicate the range of *good* hyper-parameters.

\begin{figure}
\begin{center}
\includegraphics[width=12em]{fig/error_depth_p3.pdf}
\includegraphics[width=12em]{fig/trees_p3.pdf}
\caption{RF codebooks Classification error varying trees depth (left) and numbers of trees (right)}
\label{fig:p3_trees}
\end{center}
\end{figure}

Similarly to K-means codebook, we find that for the RF codebook the optimal tree depth and number of trees are around 5 and 100 as it can be seen in figure \ref{fig:p3_trees}. The classification accuracy on average is 1% to 2% lower (78% on average, peaking at 82%).

\begin{figure}
\begin{center}
\includegraphics[width=12em]{fig/p3_vocsize.pdf}
\includegraphics[width=12em]{fig/p3_train_test_time.pdf}
\caption{RF codebooks Effect of vocabulary size; classification error (left) and time (right)}
\label{fig:p3_voc}
\end{center}
\end{figure}

Varying the randomness parameter of the RF classifier (as seen in figure \ref{fig:kmeanrandom}) when using a RF codebook gives similar results to using the K-Means codebook.

Figure \ref{fig:p3_cm} shows the confusion matrix for results with Codebook Forest Size=256, Classifier Forest Size=100, Classifier Depth=5 (examples of success and failure in figure \ref{fig:p3_succ}). The classification accuracy for this case is 79%, with the top performing class being `windsor_chair`. In our tests, we observed poorest performance with the `water_lilly` class. The per class accuracy of classification with the RF codebook is similar to that of K-Means coded data, but we observe a significant speedup in training performance when building RF tree based vocabulary.

\begin{figure}
\begin{center}
\includegraphics[width=14em]{fig/256t1_e200D5_cm.pdf}
\caption{Confusion Matrix: CodeBookForestSize=256; ClassifierForestSize=200; Depth=5}
\label{fig:p3_cm}
\end{center}
\end{figure}

# Comparison of methods and conclusions

Overall we observe marginally higher accuracy when using a K-means codebook compared to RF codebook at the expense of a higher training execution time. Testing time is similar in both methods, with RF codebooks being slightly faster as explained in section III.

As discussed in section I, due to the initialization process for optimal centroids placements, K-means can be unpreferable for large 
descriptor counts (and in absence of methods for dimensionality reduction).
In many applications the increase in training time would not justify the small increase in classification performance. 

For the `Caltech_101` dataset, a RF codebook seems to be the most suitable method to perform RF classification.

The `water_lilly` is the most misclassified class, both for K-means and RF codebook (refer to figures \ref{fig:km_cm} and \ref{fig:p3_cm}). This indicates that the quantized descriptors obtained from the class do not provide for very discriminative splits, resulting in the prioritisation of other features in the first nodes of the decision trees.

# References

<div id="refs"></div>

\newpage

# Appendix 

The Appendix section includes additional pictures to support some of the points presented in the main report.

\begin{figure}
\begin{center}
\includegraphics[width=8em]{fig/success_km.pdf}
\includegraphics[width=8em]{fig/fail_km.pdf}
\caption{K-means + RF Classifier: Success (left); Failure (right)}
\label{fig:km_succ}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=14em]{fig/p3_colormap.pdf}
\caption{Varying leaves and estimators: effect on accuracy}
\label{fig:p3_colormap}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=8em]{fig/success_3.pdf}
\includegraphics[width=8em]{fig/fail_3.pdf}
\caption{RF Codebooks + RF Classifier: Success (left); Failure (right)}
\label{fig:p3_succ}
\end{center}
\end{figure}