The results of the experiments using the 11 data sets and 5 model selection
methods are given in Table 1.
MML, MDL, CAICF, SRM and SC do converge into reasonably simple models. However,
there are two interesting phenomena worth mentioning.
First, amongst the five methods, SRM consistently underfits the data by
converging into the simplest models with the longest message length and
the worst predictive performance in all of the data sets.
On the other hand, SC consistently converges to the most complex models.
The suspicion that SC has overfitted the data sets can however not easily be
justified. Although in all of the data sets, the models have the longest
message length, they consistently show superior predictive power on
independent test data (i.e. which is not used in model development) than
all of the other models.
On the other hand, it performs much worse than the other methods on the set
that uses data from consecutive years. This suggests that SC might be more
sensitive to the presence of anomalous data items.
Second, the methods selected different final models from one another on
each data set. Although there are a few variables consistently selected by the
five methods on all of the hurricane data sets considered
(shown in Table 2), there are also variables chosen
by only one or some of the methods. This demonstrates how selection bias
introduced by using a non-exhaustive search strategy plays a part in
determining which path in the search space chosen in the search process.
One way to find out whether or not the methods can eventually home into a more
uniform model if the selection bias has been minimized is by performing an
exhaustive search on the variables that are chosen by at least one of the
methods. This research direction is being investigated.
The predictive performance of the models trained using the hurricane data from
consecutive years is inferior to that of the models trained using the
data sets built using random sampling method. This suggests that in order to
build models with good predictive performance for a problem domain that changes
overtime, it is important to make sure that the training and test data come
from the same probability distribution. One way to achieve this is by using
random sampling to build the different training and test data sets.
Table 2 shows some of the regressors found in the models
selected by MML, MDL, CAICF, SRM and SC in Table 5.
It is shown that regressor 10 (maximum potential cyclone intensity) and 6
(intensity change during the previous 12 hours) are found in all of the models
selected, hence should be concluded as the two most influential variables to
the target variable.
It is interesting to note that while the inclusion of variable (28,5) and 32 to
the model with variables 10 6 (36,36) results in a better model, the inclusion
of variable (28,5) alone results in a worse one. Hence, the model with
variables 10 6 (36,36) (28,5) 32 will never be found even using the more
exhaustive type of greedy search strategy implemented in this paper, since it
still has no capability to jump out of a bad local minimum. This suggests that
for the hurricane data sets, the implementation of adaptive search strategies
like simulated annealing or tabu search is worth considering.
Note that, the fact the costs shown by SC keeps decreasing as the model gets
more complex is not suprising since as shown in Table 1,
SC always prefers much more complex model on all of the data sets.
|
Data set |
Method |
Total Reg |
Msg Length |
Training |
Data |
Test |
Data |
|
RMSE |
R^2 |
RMSE |
R^2 |
||||
|
1 |
MML |
14 |
3568.36 |
21.53 |
0.45 |
21.94 |
0.46 |
|
|
MDL |
22 |
3571.48 |
21 |
0.47 |
21.72 |
0.47 |
|
|
CAICF |
14 |
3567.96 |
21.53 |
0.45 |
21.94 |
0.46 |
|
|
SRM |
7 |
3624.96 |
22.42 |
0.4 |
22.75 |
0.41 |
|
|
SC |
57 |
3612.07 |
19.25 |
0.56 |
20.95 |
0.52 |
|
|
SHIFOR' |
9 |
4146.16 |
26.37 |
0.17 |
27.1 |
0.17 |
|
|
SHIFOR94' |
9 |
3665.14 |
22.74 |
0.38 |
23.19 |
0.39 |
|
|
SHIFOR |
9 |
n/a |
24.64 |
n/a |
25.08 |
n/a |
|
|
SHIFOR94 |
9 |
n/a |
22.5 |
n/a |
22.97 |
n/a |
|
2 |
MML |
17 |
3470.35 |
21.13 |
0.49 |
22.1 |
0.4 |
|
|
MDL |
17 |
3493.82 |
21.2 |
0.48 |
22.22 |
0.39 |
|
|
CAICF |
17 |
3493.82 |
21.2 |
0.48 |
22.22 |
0.39 |
|
|
SRM |
16 |
3494.96 |
21.29 |
0.48 |
22.54 |
0.37 |
|
|
SC |
57 |
3550.43 |
19.28 |
0.58 |
21.11 |
0.47 |
|
|
SHIFOR' |
9 |
4121.89 |
26.69 |
0.18 |
26.3 |
0.14 |
|
|
SHIFOR94' |
9 |
3605.56 |
22.75 |
0.4 |
23.16 |
0.33 |
|
|
SHIFOR |
9 |
n/a |
25.15 |
n/a |
25.24 |
n/a |
|
|
SHIFOR94 |
9 |
n/a |
22.58 |
n/a |
22.78 |
n/a |
|
3 |
MML |
16 |
3501.82 |
21.3 |
0.47 |
21.78 |
0.43 |
|
|
MDL |
22 |
3525.14 |
21.01 |
0.49 |
21.93 |
0.42 |
|
|
CAICF |
22 |
3526.38 |
21.01 |
0.49 |
21.99 |
0.42 |
|
|
SRM |
14 |
3540.89 |
21.64 |
0.46 |
22.17 |
0.41 |
|
|
SC |
59 |
3590.3 |
19.32 |
0.57 |
20.91 |
0.49 |
|
|
SHIFOR' |
9 |
4113.04 |
26.49 |
0.19 |
26.73 |
0.13 |
|
|
SHIFOR94' |
9 |
3630.01 |
22.83 |
0.4 |
22.96 |
0.36 |
|
|
SHIFOR |
9 |
n/a |
24.78 |
n/a |
25.17 |
n/a |
|
|
SHIFOR94 |
9 |
n/a |
22.59 |
n/a |
22.76 |
n/a |
|
4 |
MML |
15 |
3544.67 |
21.55 |
0.46 |
21.3 |
0.46 |
|
|
MDL |
22 |
3526.55 |
20.95 |
0.49 |
21.44 |
0.45 |
|
|
CAICF |
22 |
3526.55 |
20.95 |
0.49 |
21.44 |
0.45 |
|
|
SRM |
12 |
3564.92 |
21.92 |
0.44 |
21.76 |
0.43 |
|
|
SC |
47 |
3550.58 |
19.63 |
0.56 |
20.62 |
0.51 |
|
|
SHIFOR' |
9 |
4160.53 |
26.84 |
0.16 |
25.98 |
0.19 |
|
|
SHIFOR94' |
9 |
3669.11 |
23.06 |
0.38 |
22.42 |
0.4 |
|
|
SHIFOR |
9 |
n/a |
24.95 |
n/a |
25.39 |
n/a |
|
|
SHIFOR94 |
9 |
n/a |
22.85 |
n/a |
22.13 |
n/a |
|
5 |
MML |
16 |
3517.58 |
21.09 |
0.47 |
22.41 |
0.43 |
|
|
MDL |
22 |
3525.92 |
20.71 |
0.49 |
22.32 |
0.44 |
|
|
CAICF |
21 |
3515.23 |
20.73 |
0.49 |
22.25 |
0.44 |
|
|
SRM |
18 |
3523.83 |
20.96 |
0.48 |
22.53 |
0.43 |
|
|
SC |
70 |
3640.86 |
18.79 |
0.59 |
20.96 |
0.52 |
|
|
SHIFOR' |
9 |
4137.05 |
26.32 |
0.17 |
27.11 |
0.16 |
|
|
SHIFOR94' |
9 |
3644.77 |
22.62 |
0.39 |
23.46 |
0.38 |
|
|
SHIFOR |
9 |
n/a |
24.41 |
n/a |
24.88 |
n/a |
|
|
SHIFOR94 |
9 |
n/a |
22.39 |
n/a |
23.2 |
n/a |
|
6 |
MML |
19 |
3541.81 |
21.16 |
0.47 |
21.9 |
0.45 |
|
|
MDL |
22 |
3550.68 |
21 |
0.48 |
21.99 |
0.44 |
|
|
CAICF |
22 |
3550.68 |
21 |
0.48 |
21.99 |
0.44 |
|
|
SRM |
16 |
3559.61 |
21.42 |
0.46 |
22.28 |
0.43 |
|
|
SC |
56 |
3606.12 |
19.4 |
0.56 |
20.86 |
0.51 |
|
|
SHIFOR' |
9 |
4144.59 |
26.53 |
0.17 |
26.65 |
0.17 |
|
|
SHIFOR94' |
9 |
3658.02 |
22.83 |
0.38 |
23 |
0.38 |
|
|
SHIFOR |
9 |
n/a |
25.6 |
n/a |
25.1 |
n/a |
|
|
SHIFOR94 |
9 |
n/a |
22.68 |
n/a |
22.53 |
n/a |
|
7 |
MML |
16 |
3550.54 |
21.42 |
0.46 |
21.66 |
0.46 |
|
|
MDL |
20 |
3555.25 |
21.15 |
0.47 |
21.47 |
0.47 |
|
|
CAICF |
20 |
3555.25 |
21.15 |
0.47 |
21.47 |
0.47 |
|
|
SRM |
3 |
3652.41 |
23.03 |
0.37 |
23.12 |
0.38 |
|
|
SC |
51 |
3616.02 |
19.71 |
0.55 |
20.54 |
0.53 |
|
|
SHIFOR' |
9 |
4129.82 |
26.36 |
0.18 |
27.08 |
0.15 |
|
|
SHIFOR94' |
9 |
3664.08 |
22.84 |
0.38 |
22.92 |
0.39 |
|
|
SHIFOR |
9 |
n/a |
25.1 |
n/a |
25.01 |
n/a |
|
|
SHIFOR94 |
9 |
n/a |
22.64 |
n/a |
22.64 |
n/a |
|
8 |
MML |
16 |
3547.12 |
21.45 |
0.46 |
21.32 |
0.47 |
|
|
MDL |
18 |
3556.98 |
21.34 |
0.47 |
21.31 |
0.47 |
|
|
CAICF |
18 |
3556.98 |
21.34 |
0.47 |
21.31 |
0.47 |
|
|
SRM |
7 |
3612.95 |
22.5 |
0.4 |
22.38 |
0.41 |
|
|
SC |
65 |
3649.64 |
19.23 |
0.57 |
20.84 |
0.51 |
|
|
SHIFOR' |
9 |
4150.87 |
26.64 |
0.16 |
26.38 |
0.18 |
|
|
SHIFOR94' |
9 |
3670.49 |
22.97 |
0.38 |
22.63 |
0.4 |
|
|
SHIFOR |
9 |
n/a |
25.31 |
n/a |
24.99 |
n/a |
|
|
SHIFOR94 |
9 |
n/a |
22.73 |
n/a |
22.42 |
n/a |
|
9 |
MML |
17 |
3516.67 |
21.36 |
0.47 |
21.34 |
0.44 |
|
|
MDL |
22 |
3521.57 |
21.05 |
0.49 |
21.04 |
0.46 |
|
|
CAICF |
17 |
3513.46 |
21.34 |
0.48 |
21.25 |
0.45 |
|
|
SRM |
16 |
3515.91 |
21.43 |
0.47 |
21.43 |
0.44 |
|
|
SC |
34 |
3558.53 |
20.54 |
0.52 |
20.69 |
0.48 |
|
|
SHIFOR' |
9 |
4122.91 |
26.64 |
0.18 |
26.44 |
0.14 |
|
|
SHIFOR94' |
9 |
3641.22 |
22.97 |
0.39 |
22.67 |
0.37 |
|
|
SHIFOR |
9 |
n/a |
25.37 |
n/a |
25.17 |
n/a |
|
|
SHIFOR94 |
9 |
n/a |
22.42 |
n/a |
22.8 |
n/a |
|
10 |
MML |
14 |
3545.93 |
21.72 |
0.45 |
21.72 |
0.43 |
|
|
MDL |
18 |
3542 |
21.4 |
0.47 |
21.31 |
0.46 |
|
|
CAICF |
18 |
3542 |
21.4 |
0.47 |
21.31 |
0.46 |
|
|
SRM |
4 |
3617.23 |
22.9 |
0.39 |
22.9 |
0.37 |
|
|
SC |
65 |
3649.65 |
19.33 |
0.57 |
20.14 |
0.53 |
|
|
SHIFOR' |
9 |
4155.62 |
26.82 |
0.16 |
26.01 |
0.18 |
|
|
SHIFOR94' |
9 |
3644 |
22.89 |
0.39 |
22.81 |
0.37 |
|
|
SHIFOR |
9 |
n/a |
25.97 |
n/a |
25.24 |
n/a |
|
|
SHIFOR94 |
9 |
n/a |
22.73 |
n/a |
22.42 |
n/a |
|
years: |
MML |
23 |
4131.98 |
20.15 |
0.52 |
26.65 |
0.24 |
|
1950-87 |
MDL |
26 |
4127.39 |
19.94 |
0.53 |
27.76 |
0.17 |
|
1988-94 |
CAICF |
26 |
4127.39 |
19.94 |
0.53 |
27.76 |
0.17 |
|
|
SRM |
22 |
4132.92 |
20.17 |
0.52 |
27.22 |
0.2 |
|
|
SC |
55 |
4166.37 |
18.82 |
0.58 |
35.24 |
0.27 |
|
|
SHIFOR' |
9 |
5036.07 |
26.65 |
0.15 |
26.33 |
0.24 |
|
|
SHIFOR94' |
9 |
4425.09 |
22.76 |
0.38 |
23.68 |
0.38 |
|
|
SHIFOR |
9 |
n/a |
25.18 |
n/a |
24.64 |
n/a |
|
|
SHIFOR94 |
9 |
n/a |
22.44 |
n/a |
23.74 |
n/a |
|
Model |
Commonly chosen regressors in models |
Freq.(of 50) |
|
1 |
10 6 |
50 |
|
2 |
10 6 (36,36) |
46 |
|
3 |
10 6 (36,36) (28,5) |
37 |
|
4 |
10 6 (36,36) (28,5) 32 |
27 |
|
5 |
10 6 (36,36) (28,5) 32 (33,32) |
24 |
|
6 |
10 6 (36,36) (28,5) 32 (33,32) 31 |
18 |
|
7 |
10 6 (36,36) (28,5) 32 (33,32) 31 (6,5) |
15 |
|
8 |
10 6 (36,36) (28,5) 32 (33,32) 31 (6,5) (35,29) |
9 |
|
9 |
10 6 (36,36) (28,5) 32 (33,32) 31 (6,5) (35,29) 29 (32,11) |
7 |
|
|
|
|
|
Regressors of benchmark models |
Name |
|
|
10 |
7 (3,1) (5,1) (6,1) (4,3) (5,3) (7,5) (5,5) (6,5) |
SHIFOR |
|
11 |
10 11 5 16 (16/4) 25 (10,10) (4,5) (6,3) |
SHIFOR94 |
|
Model |
MML |
MDL |
CAICF |
SRM |
SC |
RMSE |
R^2 |
|
1 |
5227.1275 |
5214.7275 |
5224.8145 |
0.6942 |
17250.3767 |
23.30 |
0.36 |
|
2 |
5213.8092 |
5195.0537 |
5207.4714 |
0.6935 |
17228.0916 |
23.16 |
0.37 |
|
3 |
5202.4689 |
5178.2392 |
5192.8259 |
0.6927 |
17208.5131 |
23.30 |
0.36 |
|
4 |
5206.1653 |
5180.0199 |
5196.6445 |
0.6971 |
17207.3841 |
23.01 |
0.38 |
|
5 |
5180.3789 |
5149.4671 |
5167.9434 |
0.6906 |
17173.7903 |
22.81 |
0.39 |
|
6 |
5169.7636 |
5137.5171 |
5157.7435 |
0.6896 |
17158.6810 |
22.71 |
0.39 |
|
7 |
5149.3784 |
5112.9015 |
5134.7360 |
0.6843 |
17130.7994 |
22.55 |
0.40 |
|
8 |
5142.2615 |
5101.8923 |
5125.2543 |
0.6829 |
17116.4272 |
22.45 |
0.41 |
|
9 |
5094.0638 |
5049.3720 |
5075.4189 |
0.6700 |
17056.9215 |
22.11 |
0.42 |
|
|
|
|
|
|
|
|
|
|
10 |
5871.9400 |
5825.7865 |
5850.8138 |
0.9528 |
17840.3214 |
26.52 |
0.17 |
|
11 |
5190.5200 |
5178.1915 |
5201.7290 |
0.7073 |
17192.7263 |
22.83 |
0.39 |
| 
|
|