For this assignment you will learn how to perform a statistical comparison of learning algorithms both by hand and using WEKA.

- Consider the following error rates made by the hypotheses learned by two
different learning algorithms L1 and L2 using a 10-fold cross-validated paired
t-test.
Trial L1 L2 1 0.36 0.25 2 0.36 0.24 3 0.26 0.20 4 0.23 0.20 5 0.25 0.21 6 0.28 0.22 7 0.33 0.25 8 0.33 0.24 9 0.28 0.21 10 0.30 0.20 - What is the 95% confidence interval around the true error for learner L1? Show all your work.
- What is the 95% confidence interval around the true error for learner L2? Show all your work.
- Can we conclude with 95% confidence that learner L2 is better than learner L1 on this domain? Show all work used to justify your answer.

- For this problem, we will use WEKA to generate ROC curves for J48 and
NaiveBayes on the labor dataset. First, we need to
generate and save ROC curve data.
- Using the WEKA Explorer open the labor dataset under the Preprocess tab.
- Under the Classify tab, choose the J48 classifier with default settings and click Start to perform the default 10-fold cross-validation test.
- In the Result list window, right-click on the J48 entry and choose Visualize Threshold Curve and class "good". The visualization window will appear.
- Verify the X axis to be False Positive Rate, and the Y axis to be True Positive Rate. You should now see the ROC curve.
- Click Save and store the results to a file in ARFF format.
- Exit the visualization window and repeat the above for the NaiveBayes classifier with default settings.

- Edit the two ARFF files containing the threshold curve results saved above and remove everything above and including the "@data" line. Note that the False Positive Rate and True Positive Rate values are the sixth and seventh entries, respectively, in each line.
- Open Excel and choose Data -> Get External Data -> From Text. Browse to the first ARFF file and load it as a Delimited file using comma as the delimiter. Do the same for the second ARFF file.
- Insert a chart of type Scatter with Straight Lines and put two lines on the plot: one is TP vs. FP for J48, and one is TP vs. FP for NaiveBayes.
- This chart will now show the two ROC curves for J48 and NaiveBayes on the labor dataset.
- Nicely format your chart with a title, correct axis titles, correct legend titles, and proper ranges on X and Y axes.

- Discuss your conclusions about the performance of J48 vs. NaiveBayes on the labor dataset based on the appearance of the ROC curves.
- Email to me (holder@eecs.wsu.edu)
a zip file containing the following:
- Raw threshold curve data for J48 and NaiveBayes on the labor dataset (the two files you saved in step 2e above).
- Nicely-formatted report (MSWord or PDF) containing:
- Answers from problem 1.
- Nicely-formatted plot of the two ROC curves (question 2).
- Discussion of performance comparison based on the ROC curves (question 3).