This extract deals with Evaluation Criteria, one component of the second building block.

SCORING THE PROPOSALS

In most evaluation processes, scoring is performed as the second task, immediately following the determination of compliance.

Most proposals require that the financial information be provided separately. This is to ensure that the technical evaluators have no knowledge of the pricing proposal. This avoids the debate over whether knowledge of the pricing proposal influenced an evaluator's assessment of technical factors.

Upon receipt of the proposals, the financial section is removed and given to the analyst for evaluation. This person receives the information, establishes the costs to be used in the evaluation, and determines the score (if required). Costing is discussed later in this chapter.

Copies of the technical/management proposal are distributed to the Evaluation Team as required. For small proposals, each member of the Evaluation Team may read the entire proposal (except for cost) and perform the evaluation. On larger or more complex proposals, specific sections are usually assigned to individuals. For example, the communications expert on the Evaluation Team and the Project Manager might be the only people who evaluate the proposed network design

The Evaluators then meet to review each other=s evaluations, to resolve differences, and to ensure that they share the same understanding of each proposal. This process results in scores for each evaluation criterion. The purpose of the meeting, and the team effort for that matter, is to discuss, to understand and to resolve differences - not simply to average the scores. If two evaluators, both experts in the same area, score the same proposal as a '2' and a '6' respectively, there is some fundamental difference in each evaluator's interpretation of the proposal. It is not good enough to give the proposal a '4', the average of the two scores. Fairness dictates that the two evaluators discuss the issue, identify the differences in interpretation, and agree upon a score. (However, in some jurisdictions, evaluators do their scoring independent of one another and scores are then averaged.)

Three components are required to establish a numerical score for a proposal: a detailed set of evaluation criteria; an assignment of weights to reflect the importance of each factor; and a method for establishing a score. Each of these components is discussed in the remainder of this section.

Evaluation Criteria

Evaluation criteria are guidelines that aid procuring organizations in assessing responses to a Request for Proposal (RFP). These criteria serve two primary purposes. First, they enable project participants to standardize the project criteria to be considered during each reviewer’s evaluation of a proposal. Second, they provide potential responders with an understanding of how proposals will be reviewed, both individually and in comparison with other proposals.[i]

Evaluation criteria are as different as people. Some are very specific and easy to assess. Others are vague and highly subjective; and, some would argue, arbitrary.

There are three major families of criteria: Technical, Management and Cost.

Technical criteria usually include the following: understanding of the problem, soundness of the approach and solution, ability to satisfy the stated requirements, service and support capabilities, analysis of risks, and testing methodology.

Management criteria usually include the following: project plans, management approach, qualifications of key people, project timetable, and corporate experience.

Cost is often evaluated in terms of the following criteria: total life cycle costs, cost controls, and consistency with technical and management plans.

As the RFP is developed, the evaluation criteria are identified. There are many sources of details about evaluation factors: similar RFPs from other jurisdictions, your organization’s old RFPs and templates, and RFP Handbooks. The specific evaluation criteria to be employed are based on the specific requirements of the RFP. It is important that the RFP demand the information required to perform the evaluation. For this reason, the evaluation process must be finalized prior to issuing the RFP. You want to ensure that you have asked for all of the required data to perform the evaluation.

The quote which follows is one of the best which I have found that deals with the effectiveness of evaluation criteria:[ii]

In developing this book, hundreds of pages of documentation were reviewed. The best description of the critical role of evaluation criteria in the process was provided by Utah. It is an excellent discussion and is reproduced in full:[iii]

Evaluation Criteria

Evaluation criteria are the factors an agency uses to determine which of several competing proposals submitted in response to an RFP will best meet the agency's needs. In establishing effective evaluation criteria, an agency must clearly identify the factors relevant to its selection of a contractor and then prioritize or weight these factors according to their importance in satisfying the agency's needs in the procurement. Together, the proper identification and weighing of the evaluation criteria will form an evaluation plan which will provide the agency with a common standard by which to judge the merit of competing proposals. This allows the agency to rank the proposals received while simultaneously providing offerors with a fair basis for comparison. As importantly, when evaluation criteria are properly selected and weighted, the proposals received will accurately reflect the offeror's understanding of the solicitation and the offeror's ability to deliver what the agency needs.

The process of evaluating offers is unique to the RFP method of procurement. This method allows an agency to consider factors other than price in deciding to whom a contract should be awarded. Whenever the RFP method of procurement is used, evaluation criteria should be selected which will provide offerors with a clear idea of the factors that will be important in making award. By properly identifying and weighing evaluation criteria at the outset of the procurement process, an agency can later rely on the evaluation criteria to do the work of selecting and judging the proposals submitted.

Evaluation criteria should be individually tailored to each RFP. While the choice of criteria is within the agency's sound discretion, only those factors relevant to the acquisition should be included. Further, evaluation criteria should reflect the agency's minimum needs, and should not be so restrictive as to limit competition. Evaluation criteria often encompass such factors as price or cost, technical excellence, management capability, personnel qualifications, experience and past performance. While price or cost must be included in every procurement and will be the deciding factor in most, price or cost need not be the deciding factor in all acquisitions. This is especially true for cost‑reimbursement contracts, in which the contractor's ability to understand the procurement and produce a quality product may well override narrow cost concerns.

The establishment of meaningful evaluation criteria is a critical step in choosing the best contractor for a particular procurement. Since the goal of an effective evaluation scheme is to reflect an agency's program needs, an agency must determine what evaluation factors are relevant to the procurement before choosing an evaluation plan.

For example, an agency should select different evaluation criteria for a single task, data entry job than for a long term facilities management contract. In choosing the criteria for a data entry job, an agency would select factors reflecting its need for an experienced contractor with sufficient labor and equipment to complete timely performance. Management factors would not be stressed for this type of one shot job. In contrast, for a facilities management contract, an agency would identify factors stressing the management contract, an agency would identify factors stressing the management capabilities of prospective offerors, as well as their technical competence, since the differing circumstances of a long term management contract require proven managerial expertise.

The precise evaluation criteria chosen must reflect the particular requirements of the contract. For example, an agency may quickly realize that a contractor's technical capability will be decisive in meeting an agency's need to switch from mainframe to distributed computer processing. As part of technical capability, the agency might further identify a contractor's ability to convert the agency's current programs and data files in a timely manner as critical to filling the agency's mission. Thus, "technical approach" and "conversion plan" might then be broken down into subcriteria such as "delivery schedule," "prior conversion experience," and "conversion facilities."

As noted above, in addition to clearly stating what evaluation criteria will be considered in selecting an offeror, the RFP must identify the relative importance or weight of the criteria. Using the above example, an agency might then decide that "Technical approach" is twice as important as the "conversion plan" and thus should be assigned twice the weight in the evaluation plan. To establish the relative importance of evaluation criteria, the RFP may simply state that the evaluation criteria are listed in order of relative importance. Or, the RFP may state that the evaluation criteria listed are all of equal weight. If listed in order of importance, an agency must be sure that the first or second criterion is not assigned predominant importance, since this would not provide offerors with a realistic picture of the procurement. An agency may also assign numerical weight to each of the evaluation criteria listed.

Once evaluation criteria are issued, an agency must adhere to its evaluation plan. If the agency realizes in mid‑procurement that the evaluation plan does not accurately reflect the agency's needs, then the Purchasing Agent must issue a written amendment to all offerors stating the changed evaluation plan and requesting a new round of proposals.

The process of selecting and weighing the evaluation criteria will assist the agency in understanding and defining its own needs. Similarly, the proper choice of an evaluation plan will greatly assist contractors in understanding the agency needs. This will result in the receipt of better proposals from offerors. Moreover, by clearly identifying the evaluation criteria to be used together with the relative weight assigned to each factor, an agency will be able to ward off potential protests from disgruntled offerors who could otherwise claim that the evaluation plan was not properly disclosed.

The percentage weighting for the price criteria should not be less than 30%. Any lower percentage to be given for price must be justified in writing and will require prior approval by the Director of Purchasing.

Evaluation criteria are an integral and fundamental part of an RFP package and crucial to an orderly procurement. The evaluation plan must closely reflect the RFP Statement of Work and Specifications. When properly selected, weighted and drafted, evaluation criteria can tremendously assist an agency in its procurement of goods and services.

New Criteria

Once the evaluation criteria are published in the RFP, it is difficult to deal with factors that have not been included. How do you handle the proposal from a supplier that suggests an unexpected approach, a new piece of technology, or provides information which changes the way you think about the project? How do you build this unknown into the RFP? You certainly cannot include a factor labeled "unexpected information". If you did, you would be accused of being arbitrary. One approach which seems to be gaining in popularity is to include "Management of Risk" as a factor. This permits the Evaluators to take into account new, unexpected information. All new information influences the Evaluators' assessment of the risks of the project. In addition, other criteria such as Aunderstanding of our requirements@ can be used by the Evaluators to reward suppliers for innovation and unexpected approaches.

A Few Examples of the Amount of Information Provided in an RFP

Some RFPs provide the minimum amount of information. They satisfy the law, their policies and their own practices but don’t go out of their way to provide additional information. Other RFPs provide extensive descriptions of each evaluation factor. Throughout this text, we have endorsed the belief that “more is better” – the more information you provide in the RFP, the better the resulting proposals. Alternatively, based on detailed information, some vendors may decide not to submit a proposal. This self-selection process saves both the vendors and the evaluators time and money.

Example 1

This information about the evaluation criteria is inadequate. This 28-word description only provides broad categories with few details.

The evaluation criteria for this RFP are as follows:

Understanding of project and requirements                            25%

Ability to meet time frames                                                     15%

Skills and experience with required technology                     20%

Support ability                                                                         15%

Pricing                                                                                     25%

Example 2

This statement of evaluation criteria provides some direction but lacks important details in several areas.

Example 3

This example provides some direction to the proponents in crafting their proposals.

A. Proposal Evaluation Criteria[iv]

The Commissioner of DAS will establish an Evaluation Committee to evaluate the Proposals. The evaluation Committee will review the Proposals for format to ensure conformance with the requirements of this RFP. Failure to meet these requirements might result in rejection of your organization=s Proposal. The Evaluation Committee can waive minor irregularities if, in its judgment, to do so would be in the best interests of Connecticut.

Evaluations will be based on the Proposals, and additional information requested by Connecticut, applying the following criteria as to each Proposal:

(a) Proposer=s understanding of the project, its purpose and scope, and proposer=s plan for performing the IT services, as evidenced by the proposed solution

(b) Proposer=s ability to perform the scope of the IT services, as reflected by its experience in performing such services and by the qualifications and abilities of the key individuals proposed as proposer=s team

(c) Proposer=s demonstrated ability to make available the key personnel and facilities to perform the IT services at the time of contracting and to keep them on the project thereafter

(d) Proposer=s specific record of past performance of similar IT services

(e) Price competitiveness of proposed solution and cost savings demonstrated

(f) Proposer=s ability to provide IT services form off-site facilities in Connecticut and to foster job          retention and job creation in Connecticut

(g) Demonstration of commitment to affirmative action by full compliance with regulations of the         Connecticut Commission on Human Rights and Opportunities

(h) Previous experience and customer references in government-sector IT services

(i) Expertise in managing complex integrated systems and services and implementing and maintaining evolving leading-edge technologies

(j) Expertise in business process reengineering, for purposes of developing new system architectures and developing plans for changes in computing environments

(k) Expertise in consolidating mainframe environments and in migrating systems (in whole or in part) from mainframe environments to distributed-computing environments.

(l) Financial strength and depth necessary to sustain a long-term relationship and long-term growth as Connecticut=s IT services requirements change

(m) Readiness to assume full accountability to Connecticut, its Agencies and its citizens for performance including commitments to perform IT services at levels that meet acceptable performance criteria, and commitments to an open-book approach and financial-reporting requirements

(n) Proposer=s demonstrated ability to protect highly sensitive and confidential information of its customers

(o) A focus on delivering value-added services

Example 4

This approach is simple to use and thorough.[v]

7.01 Understanding of the Project (5 Percent)

Proposals will be evaluated against the questions set out below:

[a]   Has the offeror demonstrated a thorough understanding of the purpose and scope of the project?

[b]   How well has the offeror identified pertinent issues and potential problems related to the project?

[c]    Has the offeror demonstrated an understanding of the deliverables the State expects it to provide?

[d]   Has the offeror demonstrated an understanding of the State's time schedule and can meet it?

7.02 Methodology Used for the Project (5 Percent)

Proposals will be evaluated against the questions set out below:

[a]   Does the methodology depict a logical approach to fulfilling the requirements of the RFP?

[b]   Does the methodology match and achieve the objectives set out in the RFP?

[c]    Does the methodology interface with the time schedule in the RFP?

7.03 Management Plan for the Project (5 Percent)

Proposals will be evaluated against the questions set out below:

[a]   How well does the management plan support all of the project requirements and logically lead to the deliverables required in the RFP?

[b]   How well is accountability completely and clearly defined?

[c]    Is the organization of the project team clear?

[d]   How well does the management plan illustrate the lines of authority and communication?

[e]   To what extent does the offeror already have the hardware, software, equipment, and licenses necessary to perform the contract?

[f]    Does it appear that the offeror can meet the schedule set out in the RFP?

[g]   Has the offeror gone beyond the minimum tasks necessary to meet the objectives of the RFP?

[h]   Is the proposal practical, feasible, and within budget?

[i]    How well have any potential problems been identified?

[j]    Is the proposal submitted responsive to all material requirements in the RFP?

7.04 Experience and Qualifications (5 Percent)

Proposals will be evaluated against the questions set out below:

Questions regarding the personnel:

 [a] Do the individuals assigned to the project have experience on similar projects?

[b]   Are resumes complete and do they demonstrate backgrounds that would be desirable for individuals engaged in the work the project requires?

[c]    How extensive is the applicable education and experience of the personnel designated to work on the project?

Questions regarding the firm:

 [d] How well has the firm demonstrated experience in completing similar projects on time and within budget?

[e]   How successful is the general history of the firm regarding timely and successful completion of projects?

[f]    Has the firm provided letters of reference from previous clients?

[g]   If a subcontractor will perform work on the contract, how well do they measure up to the evaluation used for the offeror?