The purpose of these experiments is to evaluate which of three WEKA clustering algorithms provides the best clustering performance for the provided malware dynamic analysis features.
Experiment 1– By using the WEKA machine learning package select 3 clustering algorithms and run the provided features with the selected clustering algorithms, evaluate the clustering metrics to determine which algorithm provides the best performance.
Experiment 2- Using the highest performing clustering algorithm from experiment 1; evaluate difference distance measures for best performance.
Experiment 3 – Using the highest performing clustering algorithm, evaluate the impact of reducing the dataset to correspond to only 50 malware samples.
.
Experiment 4– Using the highest performing clustering algorithm; evaluate the effect of reducing the features count to only 20 features.
Experimental Results
This section provides the results for the experiments given in the Experimental Design. These experiments use the WEKA machine learning package and client-supplied malware features. These experiments were performed with Binary API Histogram features. By running the experiment using the weka clustering algorithms we obtained the incorrectly clustered instances and their percentage. The correctly clustered instance percentage is calculated by subtracting the incorrectly clustered instances from 100.
Experiment 1 – The results of Experiment 1 are shown in Table 1
| Algorithms | Correctly clustered instances |
| Canopy | 70.4053% |
| EM | 80.4901% |
| SimpleKmeans | 33.3176% |
Experiment 2 – This experiment asked for three distance measures to be tested, however WEKA does not provide a user interface for this. It is noted that WEKA uses the Euclidean distance measure in the K-Means algorithm.
Experiment 3 – The purpose of this experiment is to test the effect of clustering small datasets; this experiment was run with the best performing algorithm from Experiment 1.
| Algorithm | Correctly clustered instances |
| EM | 70.9708% |
Experiment 4 – In this experiment we reduced the number of features from 172 to 20, to evaluate the effect on reducing the features.
| Algorithm | Correctly clustered instances |
| EM | 66.4939% |
Evaluation of Results
Experiment 1– On running the dataset of 172 features with the following three algorithms: canopy, EM, and SimpleKmeans. The best clustering result was obtained using the EM algorithm which correctly clustered 80.49% of malware samples.
Experiment 2– In this experiment, we were unable to select the distance metric in WEKA.
Experiment 3– This experiment was done using a smaller dataset corresponding to 50 malware samples, and was run with the best performing algorithm from experiment 1. The EM algorithm correctly clustered 70.97 % of samples. This experiment demonstrates that clustering algorithms perform better with large datasets.
Experiment 4 – This experiment aimed to evaluate the effect of reducing the feature count to 20 and run using the same algorithm used in Experiment 1. By performing this experiment the percentage of the correctly clustered instances degraded and gave a value of 66.49%. This demonstrates that clustering performs better with larger numbers of features.
