Working with market research firms

One important decision is whether you want to outsource some or all of your market research. Market research is specialized knowledge, and it often makes sense to work with a skilled vendor of market research services. Most utilities outsource the grunt work – that is, the actual data collection (e.g., surveying). Some outsource other components as well, such as preparation of the survey instrument or data analysis. Because outsourcing is so common, we provide detailed instructions on how to select a market research vendor, and how to evaluate their work.

What to ask for in the RFP

A utility soliciting market research vendors should always get at least 3 bids on any project to assure a competitive price. For a list of market research vendors, the American Marketing Association (AMA) website [link to www.marketingpower.com] can provide the names of several good companies.

1. The vendor should respond to the client’s Request for Proposal (RFP) with a proposal that is complete, on point, on time, professionally done, and cost competitive.
2. The vendor should provide examples of previous reports, for an evaluation of the level and quality of their analysis. [When the client is shown examples of previous work, that client’s name should not be apparent on the report. This is an indication the vendor is likely not to respect the privacy of its clients.]
3. The vendor should provide 3-5 relevant references.
4. The vendor should provide the client with a project outline, including a timeline with key milestones; information regarding the staff responsible for the study and their relevant experience; and a breakdown sheet of costs by task and an average cost for each survey method used, as well as options for task and survey methods at alternative costs levels.

What to look for in vendor staff and experience

1. The vendor should have experience in:
   • conducting market research studies with the customer group the client is interested in surveying;
   • developing questionnaires and conducting analyses on issues related to the client’s industry;
   • standard data collection methodologies - phone, mail, fax, and online, and mixed methods such as phone-mail-phone – and should be able to make recommendations for when to use each approach.
2. The company should be established and active in the market research field for over 5 years.
3. The client should be provided with a single point of contact – a project manager with a manageable workload (the rule of thumb is no more than 3-4 projects) and at least five years experience in market research.
4. The vendor should have a statistician and energy industry experts on staff.

What to look for in a vendor’s standards and practices

1. It is best if a vendor manages their data collection activities in-house. This promotes better quality control and communication between the project team. If they do sub-contract to other data collection firms, then it is important that they have had a long-standing partnership with the third party
2. The vendor should subscribe to the market research code of ethics, as described by such organizations as CASRO (Council of American Survey Research Organizations)
3. The vendor should have stringent quality assurance standards in place for conducting interviews, verifying completed surveys, coding open-ended questions, etc. If desired, the client should be able to monitor telephone interviews remotely
4. The vendor should be willing to sign a contract or statement of work that includes final estimates of costs, timeline and deliverables.

What the vendor should do before the study: Preparation

1. Study objectives should be identified up front with the project team and communicated clearly to this team. A written agreement that includes the project objectives, and approved by all principal internal and external clients, is a good idea to avoiding misunderstandings later
2. The vendor should provide the client with a project timeline including key milestones
3. Before data collection can begin, the survey sample must be properly identified along these dimensions:
   • Sample Frame: to identify the relevant population, sample groups, and the criteria for classifying respondents into groups.
   • Randomness: to ensure there is a proper representation of the overall population, samples should be obtained in a random fashion, enabling everyone an equal chance of being selected.
   • Size: samples need to be of statistically significant size. Sample sizes of 400 are utilized most often among market researchers. Assuming the sample has been randomly selected, results based upon 400 completed interviews have a error range of +/- 5 percentage points with a 95 percent confidence level. When a sample is stratified in advance, it is important that each group is large enough to conduct individual analysis and meaningful comparisons. Therefore, sub-groups, or quota groups, should ideally have at least 100 in each mutually exclusive cell.
4. The client should receive a draft version of the survey instrument for comment and input.
5. The survey instrument(s) should be designed to accurately measure or test the research issue in question. Measurement scales need to be consistent and appropriate for the issue under investigation. If data will be combined from more than one source, there needs to be consistency in sample and time frames and survey instrument measurement scales.
6. 6. An analysis plan should be drafted prior to survey design and collection. This plan will detail the analyses the vendor intends to conduct and may include hypotheses and anticipated relationships in the data.

What the vendor should do during the study

1. The vendor should pretest the survey – with the client’s participation, if desired - prior to launching it.
2. The vendor should provide daily or weekly status updates on the project regarding completion rates or problem groups.
3. If it is a telephone survey, the client should be able to monitor interviews while the study is being conducted.
4. If the study was conducted via the mail or over the Internet (where there are typically lower response rates) the vendor should consider non-respondent bias and determine whether or not non-responders need to be evaluated to assure the integrity of the data.
5. If the sample was stratified into sub-groups before data collection, weight factors need to be calculated and applied before the data is rolled together for final reporting.
   • During data collection respondents should be:
   • treated with respect and thanked for their time;
   • offered incentives when appropriate;
   • offered a mechanism to document complaints and/or requests to be taken off the research "list" for future studies;
   • guaranteed confidentiality of individual responses when requested.

What the vendor should do after the study: Deliverables

1. The client should help determine the specifications for data tables and the outline of the report.
2. The client should receive a draft version of the report for comment and input before distribution.
   • Included in the final set of deliverables, the client should receive:
   • the raw datasets which have been cleaned, a data map or codebook and the data tables/tabs;
   • a final report which includes key findings, recommendations, detailed analyses and any findings based upon statistical differences in the data.
   • a formal presentation of results, either over the phone or in-person.
3. Results of the market research should be defendable and make sense to those who deal with the issues being investigated. Results need to be incorporated with real world data and information to be most effective.
4. The project should come in on time and on budget. The vendor should not require more money for the project at any mid-point in the study, unless there have been major changes in project scope or in estimated response rates or incidence levels.
   
   
   Return to top