KNOWLEDGE AND INNOVATION JOURNAL OF THE KMCI Estima

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIEstimating Benefits of KnowledgeManagement Initiatives: Concepts,Methodology, and ToolsJoseph M. Firestone, dge Management Benefits and Corporate GoalsHow are various claimed Knowledge Management (KM) benefits related tocorporate goals, business processes, and to IT applications? Most discussions ofKM benefits and, for that matter, of benefits of other alternatives to KM, are nottightly coupled to corporate goals and business processes [1]. In the KMliterature the discussion of benefits thus far has not approached a systematicanalysis of corporate goals, objectives and benefits in the context of d, in most analyses there is an ad hoc listing of envisioned outcomes oreffects of the introduction of KM initiatives and an assertion that these outcomesare unequivocal benefits. The approach is basically intuitive rather than analyticaland comprehensive. It doesn’t clarify the relationship of the claimed orenvisioned outcomes to corporate goals or business processes. And it oftendoesn’t distinguish the outcomes in terms of the degree of benefit they paper presents concepts, methodology and tools for producing improvedKM benefit estimates. My objective is to provide a framework for thinking aboutmore comprehensive estimation of KM benefits -- estimation that is tightlycoupled to corporate goals, and that distinguishes benefits according to theirrelative importance. I will not propose a specific methodology for estimation in allsituations, because, as we will see, no single methodology is appropriate forevery corporate situation. Comprehensive benefit estimation is not practical inmany situations. While, in others, varying degrees of comprehensiveness will d of a single methodology, I will define an abstract pattern ofComprehensive Benefit Estimation (CBE) that would, if implemented, achieve thegoal of tight coupling of benefits, goals, and KM initiatives and competingalternatives. Then I will point out how in different concrete situations one maytailor the pattern to achieve a feasible estimation procedure.A Framework for KM Benefit EstimatesTo improve KM benefit estimates both a broader conceptual perspective andmore substantial methods and tools are needed than those provided by ad hocanalytical approaches. The road from KM initiatives or programs to benefits leadsthrough business processes and corporate goals. So here is an introductoryVOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIconceptual framework that can lead us down this road. The first part of theframework relates business processes, corporate goals, and KM initiatives. Thesecond focuses on the relationship between corporate goals and benefits. Oncethe framework is developed, I will discuss issues related to applying it toestimating KM benefits tightly coupled to corporate ate Goals, Business Processes, and KM InitiativesCorporate Goals are one category of global property of corporations. Corporategoal-strivings are pre-dispositions to perform actions calculated to create ormaintain certain intrinsically valued states of the world, either internal or externalto a corporation. Corporate goals are no more than these valued states -- thetargets of goal-strivings. I distinguish between corporate goals and corporateobjectives by defining objectives as states that are valued instrumentally for thecontribution they make toward achieving corporate goals. So there is, in thisconception, a cause and effect relation between goals and objectives. Objectivescause an agent to move closer to its goal. Goals may or may not r One: Analytical, Structural,And Global Properties of CorporationsFor every multi-person corporate organization, we can distinguish analyticalproperties, structural properties, and global properties. [2] Analytical properties arederived by aggregating (summing, averaging, or performing other elementarymathematical operations on) data describing the members of the ural properties are derived by performing operations on data describingrelations of each member of the business to some or all of the others. Lastly, globalproperties are based on information about the corporation that is not derived frominformation about its members. Instead, such properties are produced by the intra-corporate interactions comprising the system they characterize. And, in that sense,may be said to "emerge" from these interactions. [3].This distinction between goals and objectives is conceptually precise, but actualstates of the world may be both goals and objectives. This is true because theycan be simultaneously valued in themselves, and for their instrumental ate goals can be highly abstract, or very concrete. They can also begeneral in their geographic or temporal focus, or very specific. Of course, highlyabstract goals also tend to be very general in scope, while highly concrete goalstend to be very specific. The same variations of abstractness and concretenessand generality and specificity apply to corporate goals and objectives are often expressed in generalized and vague form incorporate discussions of them. “Our goal is to be the most competitivecorporation in our industry.” “Our goal is to be an ethical and socially responsiblemember of the community.” “Our goal is bring the vision of the integrated desktopto all consumers.” These are three examples of vague statements of goals oneVOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCImight find in marketing literature. But, there are also precise ways to expresscorporate goals are states of the world, we can also look at them as sets of orderedattribute values describing the corporation or its environment. Imagine a row in adatabase table, or a row vector in an algebraic matrix, recording a set of valuesfor a corporate entity. This row might define the actual state of thecorporation at a particular time. Now imagine that this row was made up not ofactual values, but of desired values intrinsically valued by a corporation. The rownow defines a multi-attribute goal-state of the corporation at the particular conceptual “distance” between the goal-state and the actual state is the pre-decision descriptive instrumental behavior gap. It is the gap that must be closedfor the corporation to get to its goal. Figure One illustrates the ideas of the multi-attribute goal and actual states of a corporation through a geometricalinterpretation. The geometric space defined by the component attributes of thegoal and actual states I will call Corporate Reality goal and actual states are represented by line vectors drawn from the originto the points in corporate reality space defined by the attribute values of thecomponents of the vectors. The pre-decision, descriptive, instrumental behaviorgap is represented by the distance vector: “a.”A benefit is provided to a corporation when an instrumental action has the effectof moving it closer to its goal-state on one or more of the component attributes ofthe goal-state. A cost, in the general sense of the term, is levied to the samecorporation, if the effect of the action is to move it away from its goal-state on oneor more of the component attributes. A net benefit results when the sum of allbenefits is greater than the sum of all costs resulting from the action. A net costresults when the sum of all costs is greater than the sum of all benefits. In thegeometric interpretation, a net benefit reduces the distance between the actualstate and the goal-state. A net cost increases this One -- Corporate Reality SpaceVOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIThese statements raise the issue of measurement of the amount of benefit andcost resulting from a decision. While it is generally true that a reduction in thedistance “a” can be called a net benefit, the amount of distance reduction is notthe amount of net benefit. Nor is the amount of increase in “a” the amount of netcost increase. The conceptual distance between the descriptive goal and actualstates does not, alone, provide enough information to measure amount of benefitand cost; because corporate reality space and the component attributescomprising it are purely descriptive and not evaluative in say that there is a net benefit when we close the descriptive gap between theactual and goal-states in corporate reality space is to go beyond the purelydescriptive character of reality space and to place a value interpretation on suchmovement. But this value interpretation is still less than explicit and somewhat adhoc, because it assumes a correspondence between reality and benefit withoutclarifying exactly what this correspondence is. To make the correspondenceexplicit, we need to work with both a descriptive (corporate reality) representationof goal and actual states and with a valuational (benefit/cost) representation ofthese. And we need to define a value interpretation mapping corporate realityspace to corporate valuational space. I will return to this subject in the section oncorporate goals and KM benefits, ations try to achieve their goals and to produce benefits by performingbusiness processes. Business process activities may be viewed as sequentiallylinked and as governed by validated rule sets of agents, i.e. their knowledge. [4][5] [6] [7] A linked sequence of activities performed by one or more agentssharing at least one corporate objective or goal, is a Task. A linked sequence oftasks governed by validated rule sets of the agents performing them, andproducing results of measurable value to these agents is a Task Pattern. Acluster of task patterns, not necessarily performed sequentially, often performedBP1CTP1T1A1VOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIFigure Two -- The Activity to Business Process Hierarchyiteratively, and incrementally, is a Task Cluster. Finally, a hierarchical network ofinterrelated, purposive, activities of intelligent agents that transforms inputs intovalued outcomes, a cluster of task clusters, is a Business Process. This activityto business process hierarchy is illustrated in Figure ss processes in corporations may be evaluated in terms of their efficiency,quality, effectiveness, and net benefit or cost. Efficiency refers to the cycle timeof the business process compared to some norm. Quality refers to how well theactivities and tasks constituting a business process are performed relative tosome set of quality standards. Effectiveness refers to whether or not thebusiness process moves the corporation toward or away from its goals and byhow much. Net benefit and cost refer to how much a business process isbenefiting or costing a , like other business processes, helps or harms corporations in attaining goalsand producing benefits. In order to measure its impact, it is necessary to view itas one of a corporation’s business processes, making an impact on otherbusiness processes, and, through them, on movement toward or away fromcorporate goals and/or objectives. In attempting to measure, analyze, or forecastits likely benefits, we need to trace the impact or forecasted impact of theintroduction and operation of KM initiatives on knowledge processes. We thenneed to trace this impact through knowledge outcomes and other businessprocesses, to its further impact on corporate goals and benefits (see FigureThree). Assessments of this kind are not easy or straightforward. But they arenecessary if a claim about the likely benefits of a KM project is to amount to morethan nonsense or hyperbole.• A(1) . . . A(n) Attributes

• The Actual State is

are dependent on I(1)..composed of A(1) . . . A(N).• KM Solution composedof tasks T1, T2, and T3

T1I(1)A(1)ActualStateKMSolutionI(2)T2I(3)T3A(2)GoalStateA(3)• T3 causes I(1) . . .I(n) Business ProcessImpacts. The effects ofT2 and T3 are not shown.I(n)A(n)BenefitStateVOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIFigure Three – The Path from KM Introduction to BenefitKM Benefits and Corporate GoalsI pointed out earlier that to relate corporate goals to corporate benefits, weneeded both descriptive and valuational representations of actual and goal-statesand of the gap between them, and that we also needed a mapping between thetwo representations. Such a mapping is called a value interpretation. It is a rule(for example an if…then statement), or set of rules that establishes acorrespondence between the components of reality space and the componentsof the valuational space that is the target of the a geometric point of view a value interpretation of corporate reality space isdefined by a set of correspondence rules mapping the dimensions (coordinateaxes) of reality space onto the dimensions of valuational space. If the valuationalspace is one whose coordinate axes or attribute components are measured onan absolute benefit measurement scale, then we can call this valuational spacecorporate benefit the actual and goal-states will have corresponding vectors in corporatebenefit space. Let’s call these the actual benefit vector and the goal benefitvector. The distance between the actual benefit vector of a corporation and theorigin of corporate benefit space is the total net benefit enjoyed by a corporationat a point in time. The distance between the goal benefit vector of a corporationand the origin is the total net benefit desired by the corporation. The distancebetween its actual and goal benefit vectors is the instrumental behavior benefitgap. It is this gap, even more than the descriptive instrumental behavior gap thatcorporations seek to framework expresses the relationship between corporate goals and benefitsclearly. Corporate goals are expressed by the multi-attribute, descriptive, goal-state vector of corporate reality space. Corporate benefits are expressed by themuti-attribute, valuational goal benefit vector of corporate benefit space. Therelationship between the two is precisely defined by the set of correspondencerules defining the mapping between the two spaces. Figure Four illustrates

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCItateal

Sal

Beneaa’B1, B2, B3 = Benefit Sales Benefit, Customer

Bonding Benefit, Profit Benefit

GoX1, X2, X3 = Sales, Customer

Bonding, Profits

Gofit

(

(t)X1B1t)Actlua

Benitef

(t)Actu

Saltat(te

)X2a= The Pre-decision DescriptiveInstrumental Behavior Gapa’= The Pre-decision Instrumental

Behavior Benefit GapB2X3B3Figure Four -- The Relationship between Corporate Goals andCorporate BenefitsMethodology and Tools for Estimating the Benefits and Costs of KMIf we look at KM benefit assessments from the viewpoint of the conceptualframework, it is clear that a thoroughgoing KM benefit assessment would:§ explicitly postulate and measure goals, objectives and progress towardthem§ gauge the impact of KM introduction on business processes and theirsuccess in attaining goals and objectives, and finally,§ interpret these descriptive analyses of KM impact or projected impact ongoals in terms of corporate benefit so that descriptions of impact are notconfused with measurements of actual is a list of the steps involved in each of these phases of comprehensiveKM benefit estimation, along with some comments describing a little of the workinvolved, and tools that might be used to accomplish it. The list of steps is anoutline of a methodology. By comprehensive I mean it takes into account bothbenefits and costs, and also provides for measuring those benefits and costs thatcannot be expressed well in monetary

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIIn this technique of benefit estimation, attributes expressing value in monetaryterms are viewed as descriptive attributes. They only become benefit attributesduring the mapping from reality to benefit space and after the transformation thatmapping entails. So measures in benefit space transcend monetary value andincorporate it into the overall framework. In particular, monetary costs andbenefits are measured as a by-product of applying the framework and as a stepalong the road to more comprehensive ing Actual and Goal-StatesStep One: Perform Measurement Modeling [8][9]§ Conceptualize and select§ attributes to describe goals and objectives in reality space§ attributes hypothesized to cause changes in actual states moving themtoward or away from goal-states§ attributes describing possible unintended side effects of actionsactivating causal attributes§ other outcome attributes important for descriptionMany of the selected attributes will be abstractions. These are not defined bydata attributes, but are assumed to be computable from them. They aremeasurable attributes. Other attributes are directly defined by data, and aremeasured attributes. So we have a mix of measured and measurable attributesin each of the four categories.§ Organize attributes into measurement clustersThat is: group the abstract attribute (e.g., customer acquisition, customerretention, customer profitability, revenue growth) that is the target or focus ofmeasurement, with the set of already measured attributes that will provide valuesto be used to compute, measure, or derive the values of the abstract targetattribute. The outcome of this task is a categorization of measured attributes bythe measurable attributes that are the primary focus of the measurementmodeling effort.§ Construct measurement modelsThese are models made of rules expressing measured attribute values asantecedent conditions and target (measurable) attribute values as consequents,with no temporal priority specified between the antecedent and consequentvalues. They are not limited to goals and objectives, but include causes, sideeffects and other outcomes as well. The rules are called semantic rules or rulesof correspondence. They frequently create multi-attribute composites asmeasures of the target attributes. Such composites can often be complex andVOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIdemanding to construct. Measurement models are essential to modeling, andcan always be distinguished as a logical component in any systems model. Youcan’t formulate a testable model of an aspect of the world without using ameasurement model. The only question is whether the measurement model isexplicit or are at least four types of (“crisp logic-based”) measurement rules thatprovide the foundation for a composite in measurement models. [9] In addition,there are fuzzy logic-based rules that map crisp values to linguistic variablevalues through fuzzy membership functions. [8] Among other things, such rulesestablish priorities among the attributes entering the composite. The activity of prioritizing attributes for their relative importance to a criterionvariable is frequently part of measurement modeling. A measurement model isdifferent from a causal model, in that the latter requires temporal asymmetrybetween antecedent and consequent [10], while measurement models imply thatvalues of the measured and measurable attributes are being viewed in cross-section. I’ve provided detailed accounts of measurement modeling in otherplaces, and won’t review them here. [8] [9]§ Use ratio scaling methods where possibleRatio scaling methods should be used in constructing measurement models ofabstractions because they allow easy mathematical manipulation andcomposition of measured attributes into target attributes. That is, they’re easy towork with when you want to create an overall measure from a set of scaling should be used in doing priority assessments among attributes,because the resulting weights are defined on the same scale and facilitatecombining the measured attributes into the target attributes. It should also beused along with direct judgmental assessments of quantitative properties ofagents and corporations where attributes are not measured and you want toconvert them to measured attributes. In that case ratio scaling provides a furtherbasis for combining measured attributes into target scaling techniques are now well-known and easy to implement. Saaty’swork on the subject is particularly accessible, and his development of theAnalytical Hierarchy Process (AHP) over a period of more than thirty years hasfeatured an emphasis on practical ratio scaling methods and their application to awide variety of subjects. [11] I have also recently treated techniques for ratioscaling in the context of knowledge discovery in databases. [8]Step One can be greatly facilitated by the proper tools. While a wide variety oftools including a spreadsheet such as Excel with fuzzy logic add-ins, amathematics package such as MATHEMATICA or MATLAB (with Fuzzy logicadd-ons) can accomplish everything you need to do in this step, the bestVOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIcombination of ease of use and power will be found with an omnibusstatistical/modeling package such as SAS, SPSS, or Statsoft (my personalfavorite because of its great spreadsheet module, graphics and generalcombination of friendliness and power), supplemented by Inform Software’sFuzzytech. With these two tools you can accomplish all of the measurementmodeling and ratio scaling you need to do, and when you’re done you cancommunicate results to external packages and back-end Two: Gather Data or perform direct assessments to measureattributes that cannot be derived from measured attributes or that haveno dataOnce the measurement model is constructed, you can’t apply it without havingvalues for its measured variables. You get these by gathering data fromdocuments, surveys, or direct observation. Data from these sources is preferableto data gathered from direct assessments of properties of corporations andagents by “expert raters” because it is thought to have greater reliability andvalidity. But data gathered from direct judgmental assessments produced bypanels of expert raters is certainly better than missing data. And there is a gooddeal of evidence in both Saaty’s work [11] and some of my own [12] suggestingthat reliability and validity levels in models using direct assessment data arecomparable to those achieved by models using document or survey-based that they exceed the levels found in models based on opinion and Two uses the same tools as Step One for direct assessments. For moreconventional data gathering and data staging you may need data warehousingETL/data cleansing tools such as Informatica, Sagent, Informix’s Datastage, orEvolutionary Technologies” ETI-Extract, and, of course, a commercial Three: Determine Actual States by using measurement models tocompute attribute valuesOnce values are given to the measured attributes, the measurement model isused to compute the values of the target measurable attribute(s) to arrive at adescription of the actual Four: Determine Goal-States by specifying goal attribute valuesThe goal-state could be specified without first determining the actual state. But itis easier to do a complete job of specifying the goal-state once the actual state ismeasured and available for examination. Then one can begin by using the actualvalues of the attribute components of the goal-state as a baseline for estimatingthe goal-state values of the same attributes. A variety of methods can be used toperform these estimates, including pair comparison rating methods. The trick isto estimate goal-state values at different points in the future, so there is enoughVOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIdata to measure the logical consistency of the judgmental estimates. Once thesevalues are derived from the estimation procedures a consistency check can alsobe made on the fit between the computed future values of the abstract attributesand the judgmental estimates of those abstract values. The judgmental forecastsof measured attribute values may then be adjusted until they are consistent withthe forecast values of target Five: Compute the pre-decision instrumental behavior gapSubtract the goal-state vector from the actual state vector to get the distance orgap vector. Compute the length of this vector, which, in the most commonly usedmathematical interpretation, is the euclidean distance between the goal-statevector and the actual state 3, 4, and 5 require no additional ng the Impact of KM SolutionsStep One: Select abstract attributes that are the focus ofmeasurement models as target attributes for Impact ModelingClassify these attributes into exogenous attributes, mutually endogenousattributes, and endogenous attributes. The exogenous attributes are causes ofother attributes, but are “not caused “ by any other attributes included in theimpact model. Mutually endogenous attributes have effects on other attributes inthe model and are affected by these same attributes. Endogenous attributes areaffected by other attributes and only affect other attributes without being affectedby them. No additional tools are needed for this Two: Specify Impact ModelSpecify hypotheses expressing the values of:(1) mutually endogenous attributes as a function of other mutuallyendogenous attributes and exogenous attributes, or as a function ofexogenous attributes alone;(2) mutually endogenous attributes as a function of other mutuallyendogenous attributes; and(3) endogenous attributes as a function of mutually endogenous these hypotheses all determining attribute values must temporally precede alldetermined attribute result of this step has been variously called a cognitive map, a conceptualgraph, a causal model, an impact model, a semantic network and many otherterms. It is composed of nodes and connecting rules. The rules can beVOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIexpressed in terms of “crisp” logical rules, or in terms of fuzzy rules. If the latter istrue, it has been called a “Fuzzy Cognitive Map.” [13]A variety of modeling tools can be used for this task depending on one’s impactmodeling orientation. The statistical packages mentioned earlier support linearstructural equation modeling. In the system dynamics area good choices areVentana's Vensim, and High Performance Systems, ithink and Stella r leading product is Powersim. For Fuzzy Cognitive Mapping you can useFuzzytech. If you prefer a complex adaptive system (cas) approach, then SantaFe Institute’s agent-based Swarm simulation, is the indicated Three: Expand the impact model by adding hypothesescomparing the effects of KM and other software alternatives onmutually endogenous and endogenous variablesIt is better if this is done by expressing the software alternatives in terms ofcomponent attribute values that describe them and allow (1) direct comparisonsof features among alternatives, and (2) formulation of hypotheses relatingsoftware features to business process attributes. However, you can also addhypotheses specifying the relative magnitude of the impact of KM softwareversus alternative software options on each mutually endogenous or endogenousattribute in the model. Ratio scaling techniques can also be used here tomeasure these relative magnitudes and to check on the consistency of thejudgments. No additional software is needed for this Four: Implement empirical tests and simulations of competingimpact models and evaluate software alternativesThe tests will provide forecasts and analyses of the impact of KM vs. other typesof solutions in moving the corporation toward its goal-state. If you use a FuzzySystems approach to impact modeling you’ll need some neural networkestimation software for testing and validation. The statistical packages mentionedabove also provide such software, as does Fuzzytech, which specificallysupports neuro-fuzzy estimation. But if you’re willing to go beyond thesepackages to vendors more specialized in neural networking, Ward SystemsGroup and NeuroDimension offer excellent and versatile g from Reality to Benefit SpaceStep One: Define rules of correspondence between attributes ofreality space and attributes of benefit spaceThere are a number of things to keep in mind when doing this mapping. First,only some of the target attributes of reality space need be directly represented byattributes of benefit space. The determining factor is whether an attribute isintrinsically valued as a benefit. An important implication is that benefit spacemay be of much lower dimensionality than reality space. Second, an attribute ofbenefit space may be the result of a composite mapping from multiple attributesVOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIof reality space. This is another source of possible lower dimensionality in benefitspace. Third, even if an attribute in reality space is represented in benefit space,the mapping is unlikely to be a simple correspondence in values. Mappings canbe similarity transformations, linear transformations, non-linear transformations ofvarious kinds, and fuzzy membership functions of diverse form. [14] A mappingfrom an attribute in reality space to a corresponding attribute in benefit space iscalled a principle of correlation. [15] [9] Such a rule should be validated throughconsistency testing and graphical means [9].Step Two: Establish benefit priority weights among attributes ofbenefit spaceThis is done using the same type of ratio scaling techniques used in measuringactual and Three: Compute the instrumental behavior benefit gapThe same method can be used as in computing the descriptive instrumentalbehavior gap, but keep two differences in mind. First, the attributes used aremapped transformations of the descriptive attributes called benefit attributes. Andsecond, in computing the euclidean distance, priority weights determined in steptwo are used to weight the attributes of benefit space to arrive at an overallmeasure of benefit. No additional software is needed for these three enting EstimationThe estimation methodology I described has the advantages of beingcomprehensive, and of tying the analysis of benefits to corporate goals, but thedisadvantage of being expensive in effort and money. It is much more likely to beused to evaluate a KM initiative after the fact than it is to be used to forecastlikely impact during the planning stage. Not least because, if begun from scratch,it will take months to implement, an unacceptable time period for a KM planningstudy. To make it useful then, abbreviated versions of it are needed that willrepresent an improvement over ad hoc benefit analysis, but that can still beaccomplished in a few weeks of effort. The nature and extent of abbreviation willdepend on the corporate environment encountered. Here are three er they define the limits for abbreviating the methodology. Real worldsituations will fit some synthesis of the One: No prior work on development of an EnterprisePerformance Management (EPM), balanced scorecard, ERP, or datawarehousing systemThis situation is hard to imagine in any major corporate environment today. In it,the comprehensive methodology of benefit estimation cannot be applied withoutgoing through all of the steps outlined, because little prior work on measurementand impact modeling already exists. In a situation like this one, it won’t beVOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIpossible to accurately estimate the benefits of a KM initiative relative to anotherKM, or data warehousing, or balanced scorecard system, with any degree ofconfidence without months of the other hand, this is also the situation where a KM initiative, or any othercompeting alternative, is likely to have its highest ROI; because it introduces awhole system of measurement and performance analysis which was previouslynot available. So a decision selecting any data warehousing, EPM, ERP,balanced scorecard, or KM initiative can be made with reasonable confidence ofsubstantial this point is recognized, the question becomes not so much whether KMproduces enough benefit that it should be funded, but whether a specific KMinitiative should be funded in preference to one of the other alternative initiativesthat can improve knowledge production, and delivery to end users. If the questionis whether a KM initiative will bring greater benefits than other alternatives, ratherthan the broader one of providing an estimate of KM benefits relative to thoseprovided by other alternatives, then there is an inexpensive method of benefitassessment that can be used to project the impact of various programalternatives relative to one method is Saaty’s Analytical Hierarchy Process (AHP). It has been appliedby Fatimeh M. Zahedi to quantitative evaluation of expert systems [16], and canbe adapted to the problem of deciding which of a group of KM or other programalternatives will provide the greatest benefit relative to other members of nice thing about using the AHP when no prior work is available is that itneeds no measured data to work except data generated by the method fromjudgmental assessments. It takes judgmental assessments about decisionoptions generated at the lowest, most concrete level of a hierarchy, andcombines that data with ratio scaled attribute priority data also generated byjudgmental assessments. Let’s review the method in Analytic Hierarchy Process (AHP) and KM benefit estimationIt is easy to develop a set of criteria to use in comparing alternative of these can be cost. You can rate alternatives according to monetaryexpense. Find out which is the most expensive, which the least, and whichalternatives are in the middle. You can go on to compare alternatives on othercriteria of evaluation, presumably criteria relevant to non-monetary costs and tobenefits. But when your comparisons are all done, how do you assess thedegree of non-monetary cost? The degree of benefit? How do you combinedifferent non-monetary costs to arrive at summary measures? How do youcombine monetary and non-monetary costs? How do you combine differentbenefits to arrive at summary measures? How do you compare costs andbenefits to arrive at benefit/cost ratios? More generally, what is the relationshipVOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIbetween specific costs and the overall summative concept of cost? What is theanalogous relationship on the benefit side?None of these questions can be answered by an ad hoc comparative approach. Ifyou use one, the process you use to aggregate comparisons on individual criteriainto an overall assessment of which KM or another alternative program is best foryou and your organization, will of necessity have to be a subjective, implicitprocess. The justification for your choice will certainly be incomplete, probablyflawed, and subject to obvious criticism from those who quickly perceive othersubjective criteria you didn't consider.A better comparative evaluation is produced if you use a rigorous framework that(a) specifies the meaning of benefit and cost in terms that connect and tightlycouple the overall goals of your organization and the characteristics of the KMsolution and other alternatives within the context of a broad benefit/cost (b) provides a means of quantitatively comparing goals, characteristics andintermediate criteria comprising the evaluation scheme. In such a framework, thecriteria used to directly assess the alternative systems are themselves assessedby other criteria that are more directly related to cost or benefit. Then these areassessed relative to still other criteria more directly related to cost or benefit, andso on, until one reaches a set of criteria that may themselves be directlyevaluated in terms of overall cost or benefit. The last step in this progressionproduces a simplified mapping from reality to benefit space. The fact that all theassessments in the progression are quantitative in nature means that questionsof the sort posed above may all be answered by applying the addition, any criticisms involving formulation of additional evaluation criteria tobe applied to the alternatives would have to be related to the benefit/costframework before their validity could be asserted. If they were so related, further,they would not change the result of an evaluation unless their quantitativesignificance was great enough to have a major impact on overall scores. In otherwords, in contrast to subjective evaluation frameworks, a rigorous evaluationframework of the kind offered here produces cumulative results. Even if mistakesof omission are made, the results of a prior evaluation need not be scrapped butonly revised, and the overall result is much less likely to be AHP fits the specification for a rigorous comparative evaluation andassessment framework just described. The AHP has had almost 30 years ofdevelopment, since its inception in the early 1970's. The primary developer of theAHP is Professor Thomas L. Saaty of the University of Pittsburgh. Saaty beganwork on certain aspects of the AHP while he was with the U.S. Arms Control andDisarmament Agency (ACDA). He published the first studies applying the AHPduring the early 1970's, when he had moved to the University of Pennsylvania.[17] [18] There, and later at the University of Pittsburgh, Saaty, his colleaguesand students have applied the method to a wide range of practical problems,VOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIincluding planning, prioritizing, optimization, benefit/cost analysis, decisionmaking, the study of national influence, terrorism, international conflict,transportation, energy policy, and many other areas. [19] [20] [21] [22] These,represent only a few of a voluminous list of references available [11].The AHP was developed by Saaty to provide a rational basis for multi-criteriadecision making of the sort involved in evaluating and selecting programalternatives. The AHP has three aspects. First, it focuses on decomposition ofthe decision problem to identify various components of an attribute hierarchy --goals, objectives, criteria, sub-criteria, elements, actors, or characteristics --relevant for decision; and grouping and ordering these components into sets orclusters of attributes, comprising the levels of a hierarchy, or into clusters thatshare levels of a , it focuses on comparative judgments between pairs of attributes withineach cluster. These comparative judgments serve as the raw data forcomputations producing a priority rating of each component attribute of a clustercompared with all other components of that cluster in relation to some criterionattribute specified at the immediate higher level of the hierarchy. The priorityratings produced are defined on a ratio scale. One ratio scale is defined for eachcriterion attribute. Since the priority ratings are defined on a ratio scale, they canbe meaningfully multiplied or divided, thus providing a basis for later benefit/costcomputations. The ratings are also tested for logical consistency within the AHP,so the extent of departure from the classic ratio scale model is measured andused to evaluate the validity of AHP , the AHP focuses on the synthesis of priorities within the hierarchicalframework. This means that "local" priority ratings, those established for aparticular component attribute in relation to other components of its cluster and aparticular criterion variable, are adjusted or weighted according to priority ratingscomputed for that component in relation to other criterion variables. It also meansthey are weighted by the priority ratings computed for the criterion variable itselfwithin its hierarchical level and its adjustment process results in a "global" priority rating being determined foreach component of each cluster and level of the hierarchy relative to the focalconcept or goal of the hierarchy. If the focal concept is a goal attribute such ascustomer profitability or monetary cost, and the decision alternatives at thebottom of the hierarchy are KM and other program alternatives, then the analytichierarchy provides no mapping from reality to benefit space. It only provides akind of relative impact evaluation of each alternative on the goal attribute. If thefocal concept (or concepts in the case of more than one goal attribute) is abenefit attribute, then the priority weights defined at the goal level of thehierarchy (relative to the benefit level) will provide a simple mapping (a linearcomposite) from reality space to benefit space, as well as a relative assessmentof the impact of the program alternatives on goals and

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIThus, in a benefits hierarchy the adjustment process produces a rating ormeasure of the "global" or overall contribution to “benefit” of each member of thehierarchy. The analogous result also holds for a cost hierarchy. So, if globalratings of program (or any other decision) alternatives at the lowest level of anattribute hierarchy are developed through the synthesis of priorities, the AHPyields benefit and cost ratings that may then be divided to arrive at meaningfulbenefit/cost ratios for each KM or other program alternative being evaluated. Aswith local priority ratings, consistency tests for "global" priority ratings are alsoprovided by the AHP. If observed inconsistency is too great, the hierarchy maybe revised until consistent ratings are provided by decision re for implementing the AHP has been developed by Ernest Forman andTom Saaty at Expert Choice, Inc. [11] The Expert Choice software comes in anumber of versions for individuals and teams and enterprises. In all cases it isfriendly and it is much easier to implement the AHP with it then it would be usingthe computational tools recommended earlier for occasional prioritizations usingthe AHP Two: ERP and/or Data Warehousing Systems ExistThis situation is more favorable for implementing the comprehensive benefitestimation methodology. Much of the work of specifying attributes in thegoal/objective, cause, side effects, and outcome categories will have been addition, data will have been gathered on many of the attributes in the , comprehensive benefit estimation will remain difficult, because most of thesteps in the measurement, impact modeling, and mapping of reality to benefitspace categories will remain. The solution is to once again apply the AHP, but tosubstitute measured attribute values in the hierarchy where they are available,and to use the real data to enrich AHP judgments where Three: A Balanced Scorecard or Enterprise PerformanceManagement (EPM) System is already availableThis is the most favorable situation for implementing comprehensive benefitestimation prior to KM construction. Such systems contain measured attributes,measurable attributes, goals, objectives, causal and side effect attributes, andoutcome attributes. Balanced scorecards also include cause and effecthypotheses relating measurable attributes to one implement comprehensive benefit estimation on a balanced scorecard orEPM foundation, supplement the measurement and causal models alreadypresent with additional rules and hypotheses, particularly those relating programalternatives to mutually endogenous and endogenous attributes in the model. Inaddition, the presence of “missing measurements” will also require generatingdata from direct judgmental assessments. Nevertheless, most of the datagathering, measurement and causal modeling activities will have already beenVOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCIcompleted. The main remaining task is the mapping from reality to benefit mapping should follow the pattern for the general method I described yThis paper presented concepts, methodology and tools for producing improvedKM benefit estimates. It provided a corporate reality/corporate benefit spaceframework for thinking about more comprehensive estimation of KM benefits --estimation that is tightly coupled to corporate goals, and that distinguishesbenefits according to their relative importance. No single methodology isappropriate for every corporate situation. Comprehensive benefit estimation isnot practical in many situations. While, in others, varying degrees ofcomprehensiveness are d of a single methodology, the chapter defined an abstract pattern ofComprehensive Benefit Estimation (CBE) that would, if implemented, achieve thegoal of tight coupling of benefits, goals, and software alternatives. It then showedhow the general pattern could be abbreviated and tailored in three different “idealtype” situations to achieve a feasible estimation procedure. Actual situations willmix the characteristics of these ideal typesReferences[1] Thomas. H. Davenport, and Lawrence Prusak, Working Knowledge: HowOrganizations Manage what they Know (Boston, MA: Harvard Business SchoolPress, 1998).[2] Paul F. Lazarsfeld and Herbert Menzel, "On the Relation Between Individualand Collective Properties," in Amitai Etzioni (ed.), Complex Organizations (NewYork: Holt, Rinehart and Winston, 1961)[3] John H. Holland, Emergence (Reading, Mass.: Addison-Wesley, 1998)[4] Edward W. Swanstrom, "What is Knowledge Management?" available at/.[5] Joseph M. Firestone, "Enterprise Knowledge Management Modeling andDistributed Knowledge Management Systems," Executive Information Systems,Inc., Wilmington, DE, January 3, 1999, available at/White_.[6] Joseph M. Firestone," The Artificial Knowledge Manager Standard: AStrawman,” Knowledge Management Consortium, International, Gaithersburg,MD, January 25, 1999 available at /White_

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCI[7] Knowledge Management Consortium, “What is Knowledge Management? AComplex Adaptive Systems Approach," KMCI PowerPoint Presentation, Draft3.0, February, 1999[8] Joseph M. Firestone, "Knowledge Management Metrics Development: ATechnical Approach," at /White_.[9] Joseph M. Firestone and Richard W. Chadwick, "A New Procedure forConstructing Measurement Models of Ratio Scale Concepts," InternationalJournal of General Systems, 2 (1975), 35-53.[10] Joseph M. Firestone, "Remarks on Concept Formation: Theory Building andTheory Testing," Philosophy of Science, 38 (Dec. 1971), 570-604[11] [12] Joseph M. Firestone and Sidney Brounstein Strategic Evaluation andPlanning System (STEPS): The Needs Assessment Capability (NAC) - ADescription of Products (Co-authored with Sidney H. Brounstein). ProgramEvaluation Staff, Farmers Home Administration, USDA, September, 1981, Pp.127-151.[13] Bart Kosko, Neural Networks and Fuzzy Systems (Englewood Cliffs, NJ:Prentice-Hall, 1992)[14] Earl Cox, The Fuzzy Systems Handbook (Cambridge, MA: Academic Press,1994).[15] Brian Ellis, Basic Concepts of Measurement (Cambridge, England:Cambridge University Press, 1966), P. 41[16] Fatemeh M. Zahedi. Intelligent Systems for Business: Expert Systems withNeural Networks (Belmont, CA: Wadsworth, 1993). Chapter 12.[17] Thomas L. Saaty, "An Eigenvalue Allocation Model for Prioritization andPlanning," Energy Management Policy Center, University of Pennsylvania, 1972.[18] Thomas L. Saaty, "Measuring the Fuzziness of Sets," Journal ofCybernetics, 4, no. 4, (1974), 53-61[19] T. L. Saaty and Shubo Xu, "Recent Developments in the Analytic HierarchyProcess, " in Thomas L. Saaty, The Analytic Hierarchy Process: Panning, PrioritySetting, Resource Allocation, 2nd Edition (Pittsburgh, PA: RWS Publications,1990)[20] P. T. Harker (ed.) "Special Issue on the Analytic Hierarchy Process," Socio-economic Planning Sciences, 20, no.6 ( 1986)VOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.

KNOWLEDGE AND

INNOVATION: JOURNAL OF THE

KMCI[21] F. Zahedi, "The Analytic Hierarchy Process: A Survey of the Method and ItsApplications," Interfaces, 16, no. 4 (1986) 96-108.[22] Thomas L. Saaty. Decision Making for Leaders (Pittsburgh, PA: RWSPublications, 1990)VOLUME

ONE, NO. THREE, APRIL

15, 2001© 2001

KNOWLEDGE

MANAGEMENT

CONSORTIUM

INTERNATIONAL, INC.