Friedland, Noah S. and Allen, Paul G. and Witbrock, Michael and Mathews, Gavin and Salay, Nancy and Miraglia, Pierluigi and Angele, Jurgen and Staab, Stephen and Israel, David and Chaudhri, Vinay and Barker, Ken and Porter, Bruce and Clark E. Peter. Towards a Quantitative Platform Independent Analysis of Knowledge Systems, in Proceedings of the 9th International Conference on Knowledge Representation and Reasoning Systems, 2004.
Abstract
The Halo Pilot, a six-month effort to evaluate the state-of-the-art in applied Knowledge Representation and Reasoning (KRR) systems, collaboratively developed a taxonomy of failures with the goal of creating a common framework of metrics against which we could measure inter- and intra- system failure characteristics of each of the three Halo knowledge applications. This platform independent taxonomy was designed with the intent of maximizing its coverage of potential failure types; providing the necessary granularity and precision to enable clear categorization of failure types; and providing a productive framework for short and longer term corrective action.
Examining the failure analysis and initial empirical use of the taxonomy provides quantitative insights into the strengths and weaknesses of individual systems and raises some issues shared by all three. These results are particularly interesting when considered against the long history of assumed reasons for knowledge system failure. Our study has also uncovered some shortcomings in the taxonomy itself, implying the need to improve both its granularity and precision. It is the hope of Project Halo to eventually produce a failure taxonomy and associated methodology that will be of general use in the fine-grained analysis of knowledge systems.