|
Phase 2
|
||
|
Navigation:
Background |
In this development phase methods to assist in the evaluation and elicitation of the quantitative component with sensitivity analysis programs were investigated. Methods to assist in the evaluation of the qualitative component using the support tool MATILDA were investigated. The end usability of the model was also assessed. The methods identified in this development phase were to be combined with stakeholder evaluation and are to be implemented in future development phases, although preliminary application, testing applicability, was performed. Methods:Sensitivity To Findings:The properties of d-separation can be used to identify if a variable will be affected by evidence from another variable in the network. Sensitivity analysis can be used to identify where evidence of a variable will push the posterior probabilities of our query nodes to higher certainty levels, 0 or 1. This information could be used to identify the variables where further evidence entered would be most informative. Sensitivity to Findings Support Tool: Two text interface programs were generated to assist in performing sensitivity analysis to findings. Both of the programs used similar algorithms and differed mainly in giving the user control on the direction of the sensitivity analysis. In the first program the user was required to give a Netica network name and a query variable from this network on which to perform the sensitivity analysis. The second program required a Netica network, on which it did an exhaustive search. The Netica API (Norsys, Inc.) provides functions to compute the entropy of a node and the mutual information of one node given another. Sensitivity To Parameters:There is high amount of inherent inaccuracy in eliciting the quantitative component of a BN. This inaccuracy could be caused by incompleteness of data used to train the model or partial knowledge of domain experts, both of which are problems identified during the development of this model. It has been shown that a BN can be extremely sensitive to changes in certain parameter estimates. Analysis of these sensitive parameters allows attention to be directed on identifying and improving them. Sensitivity analysis can be performed using an empirical approach, by altering the parameters of query node and observing the related changes in the posterior probabilities of the target node. However, such a straightforward analysis can be extremely time consuming, especially on larger networks, such as the one developed in this project. Coupe and Van der Gaag [3] address this difficulty by identifying a sensitivity set of a variable given evidence. They also demonstrate that the probability of a state given evidence can be given a functional representation. Sensitivity to Parameters Support Tool: A text/graphical interface program was generated to assist in performing sensitivity analysis to parameters. The program required the user to give a Netica network name, an interest and test node and set of evidence nodes from this network on which to perform the sensitivity analysis. The Netica API (Norsys, Inc.) provides functions to compile a BN and find the belief of an event given evidence. A gnuplot API (Nicolas Devillard) was also used to generate and display plots to the screen. MATILDA Evaluation: MATILDA [4] is a support tool that was developed to aid in communicating and explaining the graphical component of a BN. To aid the domain expert developer in understanding the concepts of d-separation, and to trial this program, evaluation of the model was conducted using MATILDA. End User Evaluation: As well as ensuring that the model is a valid representation of the system, it is also important to evaluate the usability of the model. In order to evaluate usability of the model an external domain expert evaluation was needed, as experts involved in creating the model will tend to overlook these considerations if not identified at first. This evaluation was performed by conducting an elicitation review with the model end user, Sophie Martin, from Goulburn Murray Water. This evaluation was focused on the ontological component of the model. Results:Sensitivity Analysis Support Tool: The results of the sensitivity to findings investigations provide a useful extension to the user interface of the model. The sensitivity analysis tools developed in this study will be used to assist in making decisions regarding management interventions, and identifying where future studies will be most beneficial. The results of sensitivity analysis will also be used for further developing the model, prioritizing and assisting in the elicitation of conditional probabilities. MATILDA Evaluation: The MATILDA evaluation exercises were useful in aiding the domain expert to gain better understanding of the model. It also helped to identify where the implications of d-separation were not immediately obvious in the model. This was useful, not only for understanding the model, but also in guiding future structural elicitation sessions with experts who do not any experience with the technology. End User Evaluation: The end user evaluation session helped to evaluate the usability of the model. Suggestions were generated identifying aspects of the model that were unclear or were not represented. These suggestions are to be incorporated into future stakeholder evaluation sessions. |