A real-life Actual Evaluators Guide (for proposals based on an RFP)

This document is the Evaluators Guide used by Orange County Sanitation District for five different RFPs for new technology. It was prepared by Michael Asner, a procurement consultant, who acted as a Fairness Officer for these RFPs. For copies of any of the RFPs, contact Michael.

www.rfpmentor.com;

michael@RFPMentor.com

This Guide is based on a similar guide developed by the Atlantic Lottery Corporation.

1 RFP Objective....................................................................................................................................................... 3

2 Evaluation Guide Goal/ Purpose............................................................................................................ 3

3 Evaluation Guide Deliverables.............................................................................................................. 3

4 Project Scope...................................................................................................................................................... 4

5 Project Strategy and Process Flow................................................................................................... 4

5.1 Strategy......................................................................................................................................................... 4

5.2 Project Process Flow................................................................................................................................ 6

6 Project Team Roles and Responsibilities........................................................................................ 8

6.1 Responsibilities for All Team Members............................................................................................ 8

6.2 Evaluation Team........................................................................................................................................ 9

7 RFP Evaluation, Criteria Percentage And Description......................................................... 10

7.1 RFP Evaluation.......................................................................................................................................... 10

7.1.1 Initial Review........................................................................................................................................ 10

7.1.2 Supplier Presentations........................................................................................................................ 10

7.1.3 Second Review..................................................................................................................................... 11

7.1.4 Recommendation Report.................................................................................................................... 12

7.2 Criteria Percentage Summary........................................................................................................... 13

8 Risk Management.......................................................................................................................................... 15

8.1 Critical Success Factors....................................................................................................................... 15

8.2 Risks................................................................................................................................................................. 15

9 Quality Management................................................................................................................................. 17

9.1 Project Quality Management............................................................................................................ 17

9.1.1 Lists of Pros, Cons, Issues, Questions............................................................................................ 17

9.1.2 RFP Evaluation Report........................................................................................................................ 17

10 Evaluation Worksheets...............................................................................................................................

10.1 mandatory proposal items............................................................................................................. 18

10.2 MINIMUM QUALIFICATION CRITERIA........................................................................................... 18

10.3 SCORING FOR PHASE 1......................................................................................................................... 18

10.4 SCORING FOR PHASE 2......................................................................................................................... 18

11 Evaluation Guide Approval................................................................................................................... 18

11.1 Approved by:.......................................................................................................................................... 18

11.2 Reviewed by:........................................................................................................................................... 18

Orange County Sanitation District (OCSD) or the District has recently completed its Long Range Biosolids Management Plan (LRBMP), which provided a number of recommendations for management of biosolids over the next ten to fifteen years.

A key recommendation of this plan is the diversification of biosolids products, markets and contractors. As part of the implementation of the LRBMP, OCSD has prepared four Requests for Proposals (RFPs). Each RFP solicits proposals for managing a portion of OCSD’s biosolids through a different technology. We are looking for one or more vendors to propose solutions which will comply with our policy and provide the needed processing capacities in timely and cost-effective manners.

The main purpose of this Guide is to ensure that the proposals received in response to these RFPs are evaluated fairly and completely, with input from all major corporate stake-holders, within the guidelines provided by Purchasing Department.

The resulting deliverable will be a Recommendation Report indicating the preferred Suppliers and the reasoning behind the selections. The results and recommendations from this evaluation will be presented to the Corporate Sponsor.

As a result of the evaluation procedures described in this Guide, the evaluation team will deliver the following:

1. Recommendations Report – This report will document the final recommendations that result from the evaluation process. It will summarize the content of the proposals, key points differentiating each, and the scoring each achieved related to the specifications outlined in the RFPs. It is anticipated that this document will identify the preferred Suppliers, the main reasons for selecting these Suppliers, the current state of negotiations, and will recommend that we complete negotiations.

2. Presentation to Corporate Sponsor– The project manager will present the RFP Evaluation Recommendations to the Corporate Sponsor.

· Suppliers have been requested to propose solutions in response to the issued RFPs.

· As proposals are evaluated, all team members will compile a list of Pros, Cons, Issues, and Questions related to the proposals.

· All team members will attend Supplier presentations and will attempt to answer all questions compiled during evaluation sessions.

· All team members will contribute to the final consensus scoring of each Supplier’s proposal.

5.1 The Overall Evaluation Process

Phase 1

There is a two-step selection process. First, we will evaluate the proposals for each of the four technologies separately and select the proposal which represents the “best value” for each technology.

We will now have identified the “best value” proposal for each technology. We will then review and evaluate each of these leading proposals based on the following factors:

This analysis will produce a score for each proposal. We will then select one or more of the highest scoring proposals for negotiation of appropriate contract(s). Proposals will be selected in order of decreasing scores.

5.2 Strategy

During this phase, we will evaluate each of the proposals submitted in response to this RFP. The evaluation will consist of the follow steps:

(i) Each proposal will be reviewed to ensure that all mandatory requirements have been met. Proposals failing to meet one or more mandatory requirements will be eliminated from further consideration. The mandatory proposal requirements are summarized in Attachment 2, Summary of Mandatory Proposal Items.

(ii) Each proposal will be reviewed to determine if it satisfies the Minimum Qualification Criteria identified in Table 1, page 25. Proposals failing to meet one or more minimum requirements will be eliminated from further consideration.

(iii) The evaluation committee will review each proposal and determine if any clarifications are required. If so, Proposers will be contacted by OCSD. The Evaluation Committee may require a visit to a Proposer’s site as part of this process. Discussions may be conducted with Proposers who submit proposals determined to be reasonably susceptible of being selected for award for the purpose of assuring full understanding of and responsiveness to solicitation requirements.

(iv) The evaluation committee will determine a preliminary score and rating for each proposal using the evaluation criteria, weights, and the scoring mechanisms described in Section 5.3 and 5.4 of the RFPs.

(v) Based on the scores, the evaluation committee may prepare a short-list of those proposals which are judged as being capable of providing an acceptable solution. Evaluation Committee may require a visit to a Proposer’s site as part of this process.

(vi) The Proposers on the short-list, or all Proposers, will be required to give a presentation explaining their proposal and addressing requested clarifications. The scores will then be finalized.

(vii) These Proposers may be asked to submit Best and Final Offers. (A brief explanation about Best and Final Offers is contained in Attachment 8.) In conducting discussions, there shall be no disclosure of any information derived from proposals submitted by a competing Proposer. Proposals submitted in response to Best and Final Offer will be re-assessed using the same evaluation criteria and scoring mechanisms that were used to score the original proposals.

We will now have identified the “best value” proposal for each technology. We will then review and evaluate each of these leading proposals based on the following factors:

This analysis will produce a score for each proposal. We may then select one or more of the highest scoring proposals for negotiation of appropriate contract(s). Proposals will be selected in order of decreasing scores.

5.3 Project Process Flow

Step 1

There is a two step selection process. First, we will evaluate the proposals for each of the four technologies separately and select the proposal which represents the “best value” for each technology.

The following list of activities outlines the process flow for this project. These activities will be completed separately for each of the 4 groups of proposals:

This is the initial planning session with all team members expected to attend. The objective of this planning session will be to ensure that all team members are aware of what is expected of them (their roles and responsibilities) and the schedule that must be adhered to for successful completion of this project. Of special interest will be the criteria that have been developed to assist team members in evaluating the proposals.

RFP responses that have been received by the proper closing date will be opened.

Copies of the proposals for each of the 4 RFPs will be delivered to all members of the Evaluation Team. Evaluation Team members should start their review of the documentation immediately. If team members require support from others in their departments to conduct the evaluation, it will be their responsibility to plan and coordinate that involvement.

Each proposal will be reviewed to ensure that all mandatory requirements have been met. Proposals failing to meet one or more mandatory requirements will be eliminated from further consideration. The mandatory proposal requirements are summarized in Attachment 3, Summary of Mandatory Proposal Requirements.

Each proposal will be reviewed to determine if it satisfies the Minimum Qualification Criteria identified in Table 1, page 28. Proposals failing to meet one or more minimum requirements will be eliminated from further consideration.

Meeting to review the proposals, with main emphasis on starting to evaluate, but also to start compilation of questions for suppliers. The evaluation committee will review each proposal and determine if any clarifications are required.

6. Compile Questions for Suppliers & Schedule Presentations Nov. 15 – 18, 2004

A list of questions for Suppliers will be developed and delivered to the Suppliers, in writing, for a response. The presentations (if held at this time) will be scheduled at the same time. Suppliers will be expected to respond to the questions in writing, and to include responses to the questions as part of their onsite presentation. Evaluation Committee may require a visit to a proposer’s site as part of this process.

The Suppliers will be requested to come to OCSD to meet with the RFP Evaluation Team. They will be asked to present their proposals and to answer questions that have been compiled during the initial reviews.

Once the presentations are delivered, and all outstanding questions answered, it is time for our second review. At this meeting, we will attempt to achieve consensus scoring to the proposals using the criteria that we have developed. The supplier costing will be combined with the solution scores at this point.

The evaluation committee determines a score for each proposal using the evaluation criteria, weights, and the scoring mechanisms described in Parts 1.4 and 1.5 of the RFP.

Based on the scores, the evaluation committee may prepare a short-list of those proposals which are judged as being capable of providing an acceptable solution. Evaluation Committee may require a visit to a proposer’s site as part of this process.

The proposers on the short-list, or all proposers, may be required to attend a presentation. If a presentation is held, each invited proposer will be given an agenda.

Based on all of the information provided in the proposals, through clarifications, presentations, and site visits, the Evaluation Team will finalize the scores and identify the Best Value proposal.

Discussions may be conducted with proposers who submit proposals determined to be reasonably susceptible of being selected for award for the purpose of assuring full understanding of and responsiveness to solicitation requirements. These proposers may be asked to submit Best and Final Offers. In conducting discussions, there shall be no disclosure of any information derived from proposals submitted by a competing proposer.

Proposals submitted in response to Best and Final Offer will be re-assessed using the same evaluation criteria and scoring mechanisms that were used to score the original proposals to determine the Best Value Proposal.

14. Determination of order of selection of proposals for negotiation Jan. 21, 2005

We will now have identified the “best value” proposal for each technology. We will then review and evaluate each of these leading proposals based on the following factors:

As a result of the evaluation process, a Recommendation Report will be developed by the Project Manager and Senior Buyer . The Recommendation Report will outline the results of the evaluation, and identify the preferred supplier(s). The Recommendation Report is the key deliverable of this project.

Before being presented to the Corporate Sponsor, the Recommendation Report will be presented to the RFP Evaluation Team. This review is intended to inform the team regarding what will be presented to the Corporate Sponsor. As a result of this meeting, additional information may be added to the Report for accuracy and/or clarity.

After being reviewed by the evaluation team the RFP Recommendations will be presented to the Corporate Sponsor. The RFP Recommendations will outline the results of the RFP process, the selection of preferred Supplier(s) and recommend a course of action.

6.1 Responsibilities for All Team Members

1. An evaluation committee member plays an active role in the selection of Suppliers.

2. An objective process will determine the best overall Supplier for a given proposal.

3. Evaluation criteria are items of importance that will be considered when reviewing the proposals of each Supplier. The evaluation criteria and weights have been included in the Request for Proposal (RFP) documents. The detailed breakdown of how points will be assigned must be set prior to the proposal opening.

4. During the RFP process, the designated representative from Purchasing is the ONLY person authorized to release results or communicate with the Suppliers. All requests for further information or clarification must be forwarded to the Purchasing representative.

5. All proposals and corresponding information must be kept under lock and key after working hours.

6. All proposals and corresponding information are CONFIDENTIAL. This information is not to be discussed with anyone outside the evaluation committee.

7. An evaluation committee member understands and will abide by the provisions of OCSD's "Conflict of Interest" policy.

8. While serving as an evaluation committee member, all possible conflicts of interest will be disclosed promptly to the other committee members.

6.2 Evaluation Team

The Evaluation Team is comprised of five evaluators who will contribute to the consensus scoring. Anne Marie Feery of Purchasing Dept., and Michael Asner, an external consultant, overlook the process to ensure fairness within the process. Ch2MHILL, an external consulting firm, has provided the Project Team with technical data and input related to biosolids management for development of the RFPs but is not involved in their evaluation. OCSD’s outside legal advisors, Woodruff, Spradlin & Smart, have contributed to the RFP and will be involved in the contracting process.

Each team member is responsible to review and evaluate each of the proposals, compile lists of questions/issues for submission to the suppliers, to attend Supplier presentations, and to deliver input into the process of selection.

7.1 RFP Evaluation

7.1.1 Initial Review (Tasks 3 – 6)

Purchasing will first examine each proposal to ensure its compliance with the mandatory administrative requirements. Firms failing to do so will be identified to the Project Manager.

The Proposals will be delivered to each team member by Nov. 1, 2004. Each team member should work diligently to review all proposal content, to ensure that they are ready for the Initial Review scheduled for Nov. 15-18, 2004.

Each member of the Evaluation Team is requested to organize a list of Pros, Cons, Issues, and Questions, broken down by each phase. Evaluators are requested to assign a score to each proposal that will allow us to produce a shortlist of suppliers. To facilitate an easy comparison of evaluation notes, a document will be provided to each team by Purchasing. An Excel spreadsheet(s) will also be prepared for evaluators to record their scores. Your preliminary scores and list of Pros, Cons, Issues, and Questions should be prepared prior to the initial review.

During the initial review meeting, the PM/Purchasing will review each proposal. As we go through each of the pros, cons, and issues, questions will be reviewed. Evaluators should be prepared to discuss the rationale behind their comments. As a result, the following will be determined:

3. What questions should be forwarded to each supplier, prior to their presentation?

As well as determining those points, the purpose of the initial review is to go over everyone’s lists, ensuring that all team members are aware of all other team members’ viewpoints.

7.1.2 Supplier Presentations (Tasks 7 -10)

OCSD will decide which Suppliers are requested to present their proposals, and will schedule the presentations at the same time that the compiled lists of questions are delivered to them for consideration and response.

2. If required, Suppliers respond to the questions that were delivered at time of scheduling. Additional questions may also arise during the presentation themselves.

All team members are requested to attend the supplier presentations, as additional information may be delivered at that time which may be considered significant. You may not get an answer for a question, but it will be an indicator of an issue.

As a result of the suppliers’ presentations, each evaluator will be requested to update their list of Pros/Cons/Issues/Questions and assign rankings based on the evaluations spreadsheet and the corresponding definitions (see below). The updated list of Pros/Cons/Issues and Scoresheets will be prepared during the meetings.

7.1.3 Second Review (Task 11)

A second review will be held on xxxxxx, 2004. This review will be similar to the first, as it will review the outstanding pros/cons/issues lists of all team members. This review will have one major difference, we will attempt to document consensus scores.

A grid which reflects the individual team rankings for each criterion and Supplier will be prepared in advance of the meeting. It will provide us with a snapshot of the rankings and should help us to identify areas of agreement and difference between team members. Where a wide discrepancy exists between team rankings, a discussion will take place to determine why the difference exists. In order to reach a consensus score, teams should be willing to adjust their scores if a compelling argument can be made in one direction or another. The other significant event that will occur, either as part of or immediately following the second review, will be the Financial Scoring. A method of scoring value (based on life cycle total costs over the life of the contract) has been established. We will use the consensus scores as a base, combined with the costs of the proposed solutions. The objective is to determine which proposal will deliver the highest value for dollars being spent. The combined solution and cost scores will represent the overall consensus score for each Supplier. The teams have been setup to allow each team to indicate points of view, track issues, and record preference by special areas of interest. Our objective is to achieve consensus scores.

7.1.3.1 Ranking Definitions

7.1.3.1.1 Exceptional (90 to 100) –

Ø The response covers areas not originally addressed within the RFP category and includes additional information and recommendations that would prove both valuable and beneficial to XXX.

Ø This response is considered an excellent approach demonstrating the Supplier’s authoritative knowledge and understanding of the project.

7.1.3.1.2 Exceeds Objectives (70 to 89) –

Ø A very good response that shows experience and knowledge within the category. The Supplier provides insight into their expertise, knowledge and understanding of the subject matter.

Ø The collective approach yields a qualitative benefit to the project that is beyond the minimum objectives.

Ø The approach being proposed or the majority of the features are acceptable or above the objectives set out in the RFP.

Ø The Supplier can easily implement additional features or options. A minimal number of resources will be required from XXX.

1 RFP Objective

2 Evaluation Guide Goal/ Purpose

3 Evaluation Guide Deliverables

4 Project Scope

5 Project Strategy and Process Flow

5.1 The Overall Evaluation Process

Phase 1

5.2 Strategy

5.3 Project Process Flow

Step 1

6 Project Team Roles and Responsibilities

6.1 Responsibilities for All Team Members

6.2 Evaluation Team

7 RFP Evaluation, Criteria Percentage And Description

7.1 RFP Evaluation

7.1.1 Initial Review (Tasks 3 – 6)

7.1.2 Supplier Presentations (Tasks 7 -10)

7.1.3 Second Review (Task 11)

7.1.3.1 Ranking Definitions

7.1.3.1.1 Exceptional (90 to 100) –

7.1.3.1.2 Exceeds Objectives (70 to 89) –

7.1.3.1.3 Meets Objectives (50 to 69) –