CHECKING REFERENCES

There are two common approaches to incorporating references into the evaluation process: award points, or simply use the references to confirm the winner=s capabilities.

Using References to Confirm a Selection

References are usually used to confirm the selection rather than as an evaluation criterion.  In many cases, only the references of the winning proponent are checked.  If several proponents are close in the final evaluation, references may be used to help choose between them.  The interview should be conducted using predetermined, consistent questions, in order to receive reliable and objective information about service, support, reliability, etc. It is a good idea to interview more than one person within the organization. Often senior management and the technical staff have different views of a vendor’s performance.

References should only be checked if the RFP requested them.

Most selection procedures require the bidders to identify reference accounts. The wording in many RFPs is similar:

 "Proposals must include a list of references including names and telephone numbers for whom the bidder has performed similar work. These references might be contacted during the proposal evaluation phase to determine their satisfaction with the work carried out."

"Provide three references, for work performed within the last two years, which you judge to be of similar scope and complexity."

Massachusetts describes the ground rules for using references:[i]

The Commonwealth of Massachusetts

PROCUREMENT POLICIES AND PROCEDURES HANDBOOK

PART TWO - THE DETAILS

CHAPTER II. HOW TO DO A COMPETITIVE PROCUREMENT

Pursuant to 801 CMR 21.06(9) a Procurement Management Team may verify any references included in a Bidder=s Response and to conduct any other reference or credit checks as the Procuring Department deems appropriate. The Procuring Department may consider any documented references, including documented performance records of a Bidder on file at the Procuring Department or solicited from other Departments or entities.

The PMT may conduct reference checks in a manner that it deems most appropriate and efficient, however all reference checks must be documented in writing by the PMT. Departments should use the same "script" or format of questions when conducting reference checks so that the results are consistent and fair to all Bidders. A sample Reference Check form is attached to this Handbook that Departments can modify as needed. The Procurement Management Team may decide whether or not to accept unsolicited references. Reference checks may be made at any time during the procurement process or during the life of the contract. 

Invariably, someone from the selection team makes the calls. If this is your job, what do you say? What questions do you ask? What is the purpose of the exercise? Clearly, you want to verify that the supplier did, in fact, do a very good job at that company.

But only the most naive evaluator would assume that a supplier will submit the name of a bad reference. The existence of a few solid references only demonstrates that the bidder has the potential for excellent work. It seems obvious that vendors will only submit the names of companies which they know will provide them with wonderful references. This is not always true. Occasionally, you will contact a reference that endorses the competition.

There is another, more aggressive tactic. Phone the reference accounts, not only to learn about the vendor, but to learn the names of other purchasers. (Alternatively, ask for an extensive list of customers in the RFP.) Once this has been accomplished, call all of the purchasers who weren't listed as references.

The purpose of these calls to "non reference" accounts is to learn about the supplier from a broader range of customers. These customers will relate both good and bad stories about the bidder, and its products or services.

The purpose in contacting "non reference" accounts is not simply to identify problem situations (which may have been caused by the supplier or its products, the purchaser, third parties, etc.). The purpose is to learn what the supplier did when difficulties were encountered? What did the supplier do when its project leader left? What happened when the key user became ill? What did the supplier do when the customer required more support on short-notice? Did the supplier act in the customer's best interest? Did the supplier simply disavow responsibility? Did the supplier and purchaser solve the problem by working as partners?

If these conversations cause you to be concerned about the risks in dealing with a particular supplier, then discuss the information with the supplier. You may want to listen to the supplier's side before accepting the information.

If you do decide to incorporate this approach to reference checking into your evaluation procedure, I suggest that you modify your RFP so that the vendors are aware of this new process. Here's how one RFP informed the bidders that the reference checking would include a broad range of customers, not just the names they provided: "Our evaluation will be made primarily via checks with the bidder provided references and other industry sources and users known to the evaluation team."

In one of its RFPs, Nebraska identified the areas of concern that would be discussed with the references:[ii]

5.7 Reference Checks

A vendor=s references will be contacted and interviewed with a standard questionnaire. Areas of discussion will include:

C           quality and performance of installed equipment

C           installation of hardware and software

C           quality of training and training programs

C           responsiveness to support and maintenance requirements

C           problems (poor quality deliverables, contract disputes, work stoppages)

C           problem handling, and problem resolution

C           functional and technical abilities

C           timeliness in meeting project deadlines

C           the performance of the vendor=s project team staff

C           the overall opinion of the vendor=s performance

C           whether or not the reference would rehire the firm.

Responses to reference checks will be evaluated individually and as a whole. Negative comments may be reviewed with the vendor at our option. Consistent unfavorable responses may cause disqualification of the proposal.

The State reserves the right to check any other reference(s) that might be indicated through the explicitly specified contacts or that result from communication with other entities involved with similar projects.

Assigning Points to References

Some organizations assign a score to the references and include the points in the overall evaluation. Sometimes, there is a minimum acceptable score, say 12 out of 20. Firms receiving 12 or fewer points would be eliminated from further consideration.

Typically, references are worth between 5% and 25 % of the total points. The references are contacted for the Awinner@ or the Afinalists@, the information is obtained using a checklist or questionnaire, and a score is assigned. Here is the manner is which a Washington State agency described the process in its RFP:[iii] The good news is that they provided the bidders with details of the reference information sought and the weight or score that was being assigned to references. The bad news is that this section is legalistic and difficult to understand. I expect that more than one of the bidders read this section two or three times and still didn’t understand it fully.

4.8 REFERENCES - EQUIPMENT, SOFTWARE, AND INSTALLATION

VALUE = 50%

Provide at lease three appropriate references of customers which use equipment/software combinations which are the same as or substantially similar to the system being proposed and where you managed the installation and conversion. For each reference, list:

a) Company Name

b) Individual contact name, and title if available

c) Telephone number

d) If available, a brief description of the system(s) installed at the reference and the application for        which it is used.

The references will be contacted during the evaluation of the management proposal, and the results of the contacts will be used to derive the score for this section. It is recognized that some bidders may be dealers of systems manufactured by other entities, and in that case customers of other dealers may be referenced if they acquired and installed a system which is substantially the same as that being proposed. For such references also provide the name and telephone number of the dealer for whom the system was acquired.

If the result of the foregoing is that there are fewer than three references which are direct customers of the bidder, then provide additional other references so that there are at least three which are customers of the bidder, although they need not necessarily be purchasers of the system being proposed.

References to entities which are owned by the bidder, or are owned by an entity which also owns a significant interest in the bidder, are not acceptable and do not comply with the requirement of this subsection.

A Concrete Example[iv]

In this example from the state of Washington references are used as an evaluation criterion. This example is more complex and more formalized than that found in many RFPs. Each proponent is required to submit at least five references. Each of the references must complete a questionnaire contained in the RFP. The questionnaire itself focuses on twenty measures of performance excellence.

To be considered responsive, the proposal must contain five references and the scoring of past performance must be at least 560 out of a possible 700 points indicating that the overall performance was “above expectations”.

References 700 maximum points, 560 minimum category points required to be considered responsive. 

Proposer should provide five (5) references who have had experience with the proposer under existing and/or prior contracts of a scope similar to this RFP and for each of the categories listed in this section. Any proposals which fail to meet this level of excellence, as identified above, may not be allowed to progress to step 3.  Proposer is only required to submit references for the categories the Proposer is bidding.  The scoring is based off the five (5) references provided. If more than five (5) references are provided for the same category then the state will randomly select only five (5) references.  If Proposer fails to provide five (5) references then that Proposer will lose 140 points per reference not submitted.   N/A or no response to a question = 0 points for that questions.  The Lead State reserves the right to lower these requirement in order to ensure sufficient contract coverage

Qualified Proposers will be evaluated on performance under existing and prior contracts of a scope similar to this RFP.  Performance information will be used for both responsibility determinations and as an evaluation factor against which Proposers relative rankings will be compared to assure best service value to the states.  The state will focus on information that demonstrates the Proposer’s performance relative to the size and complexity of the procurement under consideration.  References identified by the Proposer may be contacted by the state to validate the score they received.  The state reserves the right to adjust the reference score to reflect all information received.  

Past Performance was scored by each reference. The RFP provided a detailed “how to” section which included a definition of each of seven different performance levels to be assigned to each factor.

Past performance will receive relative consideration as designated in each segment, as identified below.  All subfactors are of equal importance.  The Proposer(s) is responsible for providing a copy of the “Past Performance Questionnaire” to no less than 5 references for completion and delivery of completed sealed surveys with their RFP proposal response, and with a signed copy of Attachment “B” for each reference.  The references will be evaluated and scored on the following categories.  If Proposer(s) fails to provide references at the time the RFP is submitted the RFP may be deemed non-responsive.

The state will make additional copies of the completed sealed evaluations for its use in scoring the proposal.

Past Performance segments:

Product and Service – compliance with contract requirements

Timeliness of Performance – met interim milestones, responsive to technical direction, completed on time, including wrap-up and contract administration, met repair response times, etc.

Cost Control – within budget – current accurate and complete billings – relationship of bid costs to actual costs – cost efficiencies…

Business Relations – effective management, reasonable/cooperative behavior – flexible effective contractor recommended solutions – business like concern for customer’s interests.

Community Relations – Citizen like concern for community safety.

Performance for non-cost factor (past performance) will be scored as raw points from 1(lowest) to 7 (highest) using the following definitions: (N/A responses = 0) The points will be totaled.

Performance Level 7:  Performance indicates excellent capability and support of the contract.  Performance stands above all others.  There are no critical shortfalls.

Performance Level 6:  Performance is above expectation, far exceeds desired quality, and stands out.  May have shortfalls in a few non-critical areas.

Performance Level 5:  Performance is slightly above expectations and for the most areas exceeds desired quality.  Has exhibited some shortfalls in a few non-critical areas.

Performance Level 4:  Performance meets minimum expectations and is generally adequate.  Has exhibited shortfalls in performance in non-critical areas and does not stand out.

Performance Level 3:  Performance is seldom complete, deficiencies exist in critical areas and limited shortfalls exist in non-critical areas.

Performance Level 2:  Performance is not complete and serious shortfalls in capability exist.

Performance Level 1:  Performance is non existent in critical and non-critical areas.

Assessment of the Proposer’s past performance will be one means of evaluating the credibility of the Proposer’s proposal, and relative capability to meet performance requirements.

Information utilized may be obtained from the references listed in the proposal, other customers known to the state, consumer protection organizations, and other who may have useful and relevant information Award may be made from the initial offers without discussion.  However, if discussions are held, Proposer(s) may be given an opportunity to address unfavorable reports of past performance.  Recent contracts will be examined to ensure that corrective measures have been implemented.  Prompt corrective action in isolated instances may not outweigh overall negative trends.

The questionnaire contained 20 items and required the reference to assign a performance score of between 1 (“non existent”) to 7 (“excellent capability”) for each.

Question 1:  Past Performance

Question 2:  Delivery As Promised?

Question 3: Equipment Provided Meets Operational requirements

Question 4:  Equipment Provided Meets Specifications

Question 5:  Provided Necessary Documentation (e.g. warranties, instructional materials, directions)

Question 6:  Interconnects Appropriately with Existing Equipment

Question 7:  Maintenance Was Performed as Required?

Question 8: Reliability – Did Equipment Perform as Expected?

Question 9:  Would you buy from this Manufacturer again?

Question 10:  Did the equipment you purchased from this Manufacturer require interoperability with any other manufacturer?  YES__________ NO__________

If YES, then please rate the equipment’s Interoperability with the other manufacturer

Question 11: Warranty/Customer Service Support

Question 12: Did your contract or purchase agreement with this manufacturer require interaction with agency (ies) or other vendors when solving a equipment problems? 

                                                YES_____________ NO____________

                        If YES, then please rate this manufacturers ability to interact with others to solve problems

Question 13: Use of  Current Technology in this manufacturer’s products

Question 14:  Fitness for use of company products

Question 15:  Effectiveness of company services (e.g. support and repair)

Question 16:  Company’s efforts to develop business relationship with you as a customer

Question 17:   Execution/Efficiency of Manufacturer during  performance of your agreement:

Question 18:  Value of Company products to your system(s)

Question 19:  Value of Company services to your system(s)

Question 20:  Rate your overall satisfaction with this Manufacturer.