Due: March 2, 2004 (midnight)
For this assignment you will use WEKA to perform a statistical comparison
of several learning algorithms using several datasets.
- Using WEKA's experimenter environment, perform the following
- For the Results Destination section, select ARFF file and provide a
file name in which to store the experimental results.
- For Experiment Type, choose 10-fold cross-validation and
- For Iteration Control, choose the default settings: 10 iterations and
data sets first.
- Select the following five datasets that come with WEKA:
contact-lenses, iris, labor, soybean and weather.
- Select the following classifiers with default parameter settings:
ConjunctiveRule, NaiveBayes and J48.
- Run the experiment.
- Analyze the results by loading the ARFF results file, select
"Percent_incorrect" as the comparison field, set the significance level to
0.05, select ConjunctiveRule as the test base, check to show standard
deviations, and perform the test.
- Construct a table of classifiers vs. datasets, and in each entry,
enter the error and standard deviation of that classifier on that dataset
from the above experiment. Also, add an asterisk to the end of the entry
for each dataset/classifier pair for which the classifier outperforms
ConjunctiveRule at the 0.05 level.
- For the above experiment, what is the lowest significance level (to the
nearest 10th) at which the NaiveBayes classifier significantly outperforms
the J48 classifier on the soybean dataset?
- Next, we will use WEKA to generate ROC curves for NaiveBayes and J48
on the labor dataset. First, we need to generate and save ROC curve data.
Now, we need to load the data into Excel (or some other charting software)
to visualize the ROC curves for both classifiers at once. Here's an
outline of the process for Excel.
- Using the WEKA Explorer open the labor dataset under the Preprocess
- Under the Classify tab, choose the NaiveBayes classifier and click
Start to perform a 10-fold cross-validation test.
- In the Result list window, right-click on the NaiveBayes entry and
choose Visualize Threshold Curve and class "good". The visualization window
- Change the X axis to be False Positive Rate, and change the Y axis to
be True Positive Rate. You should now see the ROC curve.
- Click Save and store the results to a file in ARFF format.
- Exit the visualization window and repeat the above for the J48
classifier with default settings.
- Edit the two ARFF files containing the threshold curve results saved
above and remove everything above and including the "@data" line. Note that
the False Positive Rate and True Positive Rate values are the sixth and
seventh entries, respectively, in each line.
- Open Excel and choose Data -> Import External Data -> Import
Data. Browse to the first ARFF file and load it as a comma-delimited
file. Do the same for the second ARFF file.
- Insert a chart of type scatter line plot and put two lines on the
plot: one is TP vs. FP for NaiveBayes, and one is TP vs. FP for J48.
- This chart will now show the two ROC curves for NaiveBayes and J48 on
the labor dataset.
- Nicely format your chart with a title, correct axis titles, correct
legend titles, and proper ranges on X and Y axes.
- Discuss your conclusions about the performance of NaiveBayes and J48
on the labor dataset based on the appearance of the ROC curves.
- Email to me (firstname.lastname@example.org)
your nicely-formatted report (MSWord, PDF or PostScript) containing the
information referred to above. Specifically,
- Raw output of the first experiment above (result from 1g).
- Table summarizing results of first experiment.
- Minimum significance level at which NaiveBayes outperforms J48 on the
- Raw threshold curve data for NaiveBayes and J48 on the labor dataset
(the two files you saved in step 4e above).
- Nicely-formatted plot of the two ROC curves.
- Discussion of performance comparison based on the ROC curves.