Evaluation Matrices

An evaluation process takes us back to the customer requirements, and evaluates the options against these requirements. Ideally, the evaluation is informed by testing. The evaluation process itself is arbitrary, and there are many different evaluation rules that you can use - selecting the right procedure is important to get the best results for your end design.

Evaluation with multiple criteria

Choose the most appropriate evaluation method, given the requirements for your project:

• direct ranking of alternatives: using a pairwise to determine the ordered rank
• comparing alternatives against each other: choosing the best between alternatives
• comparing alternatives against a standard: the alternatives must meet a minimum standard
• comparing criteria across alternatives: do the alternatives meet the criteria by importance
• weighted evaluation*: such as the tabular additive method, which takes into consideration the relative importance of each criteria
• under risk: which compares the possible future or expected value of the alternative
• under uncertainty*: which allows for comparison where the results are uncertain * these methods are recommended, but are not necessarily the right choice for all evaluation processes

Steps

As with testing, an evaluation process should be transparent and repeatable. Each evaluation method requires tabulating alternatives against requirements.

• Construct a table with the relevant customer or design requirements in the first column, the relative importance of the requirement in the second column
• Place the alternative designs (usually at least three) in the following columns, and put the benchmark or TPM as a reference in the last column.
• Fill in the relevant performance data for the alternative designs against the requirements

If completing a direct comparison:

• choose the best alternative against the benchmark using the chosen decision rules

If completing a weighted evaluation:

• score the alternative on a defined scale (usually between 0-5) against the requirement
• multiply out and tally the alternatives, selecting the alternative that performs best against the requirements

If completing an evaluation under uncertainty:

• present the results in a graph with all possible values, and indicate the values that likely or suitable

Hints

• Make sure that you use the evaluation process to objectively show the best result, not show the result that you want.
• The design performs best against the criteria shouldn’t necessarily be the design to choose - use the evaluation to inform your decision, not as the decision

Core resources

• Evaluation Matrices worksheet. This is a good demonstration on how you can use your customer requirements to identify the best solution [PDF worksheet]
• Chris’ 2012 ENGN2226 Lecture Slides on evaluation of models, includes a demonstration of a number of design evaluation models [PDF slides]