|
Conclusions
|
||
|
Navigation:
Background Events Conclusions |
The first task of this thesis was a reverse engineering analysis of the ERA prototype model (ERA). This analysis included identification of knowledge engineering tasks that had not been undertaken by the model developers or that could be improved. The prototype model was evaluated and preliminary improvements were identified. These included the incorporation of temporal and improvements to the spatial representations of the model, which had a secondary advantage of an improved potential for learning from the data. It was also identified that elicitation and evaluation support tools would be useful in assisting the development tasks. In the first development phase (Phase 1) improvements identified during the reverse engineering of the model were investigated and presented at a stakeholder workshop held to evaluate the model. These included improvements to the spatial and temporal representation of the model. During the workshop, a range of improvements to the ontological, qualitative and quantitative components of the model were identified. Stakeholder suggestions were evaluated and implemented where possible. The temporal and spatial representation improvements were accepted and identified as powerful improvements to the representational power of the model. The workshop process highlighted the difficulties in the elicitation and evaluation of the quantitative component of the model. It was suggested that development of sensitivity analysis support tools would be useful for identifying where greater effort could be directed for improving the quantitative component of the BN. In the second development phase (Phase 2) various support tools were created and/or applied to evaluating the quantitative and qualitative components of the model. The end usability of the model was assessed and qualitative evaluation was conducted using structural evaluation program MATILDA. The development of the sensitivity analysis support tools took two directions. In order to identify which variables had dominating effects on the query variables, analysis on sensitivity to findings was conducted. Although support for this type of analysis was provided in the Netica environment, Netica did not enable a complete sensitivity analysis to be carried out efficiently. In order to improve upon the existing Netica tools, an automated program was written using Netica's programming interface. This enabled the generation of improved and informative reports. In order to identify which parameters had dominating effects on the query variables, analysis on sensitivity to parameters was conducted. No existing support tools for this task could be identified, although existing literature describing methods to do this efficiently were found. Again using Netica's programming interface, support tools where generated to guide the elicitation and evaluation of the quantitative components. The structural evaluation support tool MATILDA was used to improve the domain expert's understanding of the network structure. This was useful, as this improved knowledge would aid in directing future structural elicitation, evaluation and implementation. The end usability of the model was also assessed and suggestions were made for ontological improvements. The resulting tools and evaluation outcomes of this phase are to be incorporated with stakeholder evaluation in future development phases. In the third development phase (Phase 3) methods for evaluating and learning parameters and structure were investigated and applied. The existing data quality was assessed to determine its suitability to the learning tasks. Due to the poor quality of the data, methods were developed allowing the domain expert to supervise the automated learning tasks. Investigation into each of the learning methods revealed that the expectation maximization algorithm would be most appropriate to learn the parameters. Support tools to do EM learning were included in the Netica environment. In order to guide this learning task a program applying the Bhattacharyya distance measure was developed to provide feedback on which parameters were changed by learning and by how much. In addition to the learning of parameters an automated model-learning tool, CaMML, was used. Even though the resulting structure was not useful in improving the model, it did offer interesting insight into other properties of the data. The predictive accuracy was also assessed in this phase, using support tools in the Netica environment. Results from this evaluation showed that the model was not a very good predictor of the data. This was not very surprising as the knowledge of the domain system is poor and there is a suspected negative bias in parameters elicited from experts. |