Further modificaations to sections III and IV

1 files changed, 22 insertions, 25 deletions

diff --git a/report/paper.md b/report/paper.md
index 2a14059..3ea9b94 100644
--- a/report/paper.md
+++ b/report/paper.md
@@ -97,7 +97,7 @@ We evaluate permutations of the architecture involving:
 * Deep CGAN - 5 Dense-LeakyReLu-BN
 * Deep Convolutional GAN - DCGAN + conditional label input
 * One-Sided Label Smoothing (LS)
-* Various Dropout (DO)- Use 0.1 and 0.5 Dropout parameters
+* Various Dropout (DO)- Use 0.1, 0.3 and 0.5
 * Virtual Batch Normalisation (VBN)- Normalisation based on one batch [@improved]
 
 \begin{figure}
@@ -118,13 +118,31 @@ The best compromise is reached for 3 Dense-LeakyReLu-BN blocks as shown in figur
 and their value goes below 1, meaning the GAN is approaching the theoretical Nash Equilibrium of 0.5.
 The image quality is better than the two examples reported earlier, proving that this Medium-depth architecture is the best compromise.
 
+\begin{figure}
+\begin{center}
+\includegraphics[width=24em]{fig/med_cgan_ex.png}
+\includegraphics[width=24em]{fig/med_cgan.png}
+\caption{Medium CGAN}
+\label{fig:cmed}
+\end{center}
+\end{figure}
+
 The three levels of dropout rates attempted do not affect the performance significantly, and as we can see in figures \ref{fig:cg_drop1_1} (0.1), \ref{fig:cmed}(0.3) and \ref{fig:cg_drop2_1}(0.5), both
 image quality and G-D losses are comparable.
 
 The biggest improvement in performance is obtained through one-sided label smoothing, shifting the true labels form 1 to 0.9 to incentivize the discriminator.
-Using 0.1 instead of zero for the fake labels does not improve performance, as the discriminator loses incentive to do better. Performance results for
+Using 0.1 instead of zero for the fake labels does not improve performance, as the discriminator loses incentive to do better (generator behaviour is reinforced). Performance results for
 one-sided labels smoothing with true labels = 0.9 are shown in figure \ref{fig:smooth}.
 
+\begin{figure}
+\begin{center}
+\includegraphics[width=24em]{fig/smoothing_ex.png}
+\includegraphics[width=24em]{fig/smoothing.png}
+\caption{One sided label smoothing}
+\label{fig:smooth}
+\end{center}
+\end{figure}
+
 Virtual Batch normalization does not affect performance significantly. Applying this technique to both the CGAN architectures used keeps G-D losses
 mostly unchanged. The biggest change we expect to see is a lower correlation between images in the same batch. This aspect will mostly affect
 performance when training a classifier with the generated images from CGAN, as we will obtain more diverse images. Training with a larger batch size 
@@ -135,16 +153,6 @@ it is possible to achieve a better performance by finer tuning of the Convolutio
 of the generator loss, hence the image quality varies a lot at each training step. Attempting LS on this architecture achieved a similar outcome
 when compared to the non-convolutional counterpart.
 
-
-\begin{figure}
-\begin{center}
-\includegraphics[width=24em]{fig/med_cgan_ex.png}
-\includegraphics[width=24em]{fig/med_cgan.png}
-\caption{Medium CGAN}
-\label{fig:cmed}
-\end{center}
-\end{figure}
-
 # Inception Score
 
 Inception score is calculated as introduced by Tim Salimans et. al [@improved]. However as we are evaluating MNIST, we use LeNet-5 [@lenet]  as the basis of the inceptioen score.
@@ -154,9 +162,7 @@ $$ \textrm{IS}(x) = \exp(\mathbb{E}_x \left( \textrm{KL} ( p(y\mid x) \| p(y) )
 
 We further report the classification accuracy as found with LeNet.
 
-### Classifier Architecture Used
-
-\begin{table}[]
+\begin{table}[H]
 \begin{tabular}{llll}
                       & Accuracy & IS		 & GAN Tr. Time \\ \hline
 Shallow CGAN          & 0.645    & 3.57          & 8:14         \\
@@ -181,15 +187,6 @@ We observe increased accruacy as we increase the depth of the GAN arhitecture at
 
 ### One Side Label Smoothing
 
-\begin{figure}
-\begin{center}
-\includegraphics[width=24em]{fig/smoothing_ex.png}
-\includegraphics[width=24em]{fig/smoothing.png}
-\caption{One sided label smoothing}
-\label{fig:smooth}
-\end{center}
-\end{figure}
-
 One sided label smoothing involves relaxing our confidence on the labels in our data. This lowers the loss target to below 1. Tim Salimans et. al. [@improved] show smoothing of the positive labels reduces the vulnerability of the neural network to adversarial examples. We observe significant improvements to the Inception score and classification accuracy.
 
 ### Virtual Batch Normalisation
@@ -198,7 +195,7 @@ Virtual Batch Noramlisation is a further optimisation technique proposed by Tim
 
 ### Dropout
 
-The effect of dropout for the non-convolutional CGAN architecture does not affect performance as much as in DCGAN, nor does it seem to affect the quality of images produced, together with the G-D loss remain almost unchanged. Results are presented in figures \ref{fig:cg_drop1_1}, \ref{fig:cg_drop1_2}, \ref{fig:cg_drop2_1}, \ref{fig:cg_drop2_2}.
+The effect of dropout for the non-convolutional CGAN architecture does not affect performance as much as in DCGAN, nor does it seem to affect the quality of images produced, together with the G-D loss remain almost unchanged. Ultimately, judging from the inception scores, it is preferable to use a low dropout rate (in our case 0.1 seems to be the dropout rate that achieves the best results).
 
 
 # Re-training the handwritten digit classifier