IMPACTSCAN

A policy intelligence tool
for regional innovation policy
 

USERS GUIDE

Contact
 

The regions
 

www.Impactscan.net
Home
Downloads
Table of Contents
Introduction
The Tool
Matrix System Overview
Matrix 1
Matrix 2
Matrix 3
Context Setting
Interpretation of results
Survey Example



search function requires internet connection

Methodology: data gathering and scoring mechanism

Two elements are needed to implement a scoring system for Matrix 3:
1. An information gathering tool to capture companies’ views: this consists of exploitation of existing surveys, interviews, new surveys, working groups, use of intermediaries such as industry federations, etc. In addition, segmentation of companies (to account for different needs of different types of companies) might also be included.
2. A scoring system: development of a metrics system to translate results of information gathering into scores. The scoring system needs to generate comparable data across regions, with the view of inter-regional benchmarking.

Information collection procedures
The most appropriate method to collect data for the purpose of creating the scores for M3, is to launch a new firm survey focused on the question of the influence of regional services on firms’ “innovation enablers”, using the harmonised IMPACTSCAN categories of services and innovation enablers. The ideal would be to gather data for individual services in a first step and then aggregate them according the IMPACTSCAN categories in a second step. Gathering primary data directly from the beneficiaries of regional services appears as a necessary stage in the context of the IMPACTSCAN objectives, as testified by the pilot phase experiments.

Experience shows that written surveys generally generate a very low response rate, due to “survey fatigue” and to the complexity of questions linked to innovation. Given the presence of such complexity also in the two typologies used in IMPACTSCAN (services and innovation enablers), an alternative is to carry out phone surveys. This can be implemented by using specialised service companies, making sure that enquirers are well briefed on the scope and key concepts of the survey.

The best way to gather the information- but also the most time consuming one – is to carry out in-depth case study interviews. Some limitations of this method need to be kept in mind: 1) a properly run phone survey would probably already exhaust available time and budget for this phase of the work; 2) those interviews need to be done by very qualified people who are not always available; 3) results gathered are qualitative in nature and do not easily translate into a standardised score. When possible, regions are encouraged to carry out such in-depth interviews, which would mainly be used for interpreting the results of the main phone survey.

The content of the survey will need to be adapted to the context of each region; in Annex one example of a survey structure is provided.

The sample of firms to be enquired needs to be representative of the regional fabric, but limited to firms with innovation potential. Use should be made of existing firms databases such as the ones held by regional authorities, chambers of commerce, industry federations, funding organizations, etc. The danger to avoid would be to limit the sample to firms involved in R&D activities or highly innovative companies, as the role of intermediaries and the nature of services often target those firms with innovation potential but not yet active in innovation on a systematic basis. Often, database on R&D-intensive firms do exist, but they need to be complemented by other sources to cover non R&D-active firms

A categorisation of the firms should be developed in order to account for the various needs of different types of companies. The following parameters may be taken into consideration:
• size: number of employees with classes <10; <50; <250;>= 250
further segmentation according regional requirements optional
• age: new firm (up to 2 to 5 years depending on the region) – mature
• Sector: based on NACE codes
• R&D intensity (if possible): expenditures to turnover.
These parameters can be used as filters for Matrix 3 in order to gain an overall picture of the impact assessment of services on all type of firms or to gain a focussed picture of the assessment on a specific target group. Differences by applying the filters will allow the identification of different impact of single service types on different categories of firms.

Example : Postal survey and face-to-face interviews for M3 in Brittany

A postal survey  was launched in Brittany to more than 6500 firms. 580 responses were received, i.e. a response rate of 7.21% .

Two lists of contacts were used :

  • the database of the Regional advisers’ network managed by Brittany Innovation (5000 contacts “informed” through advisers’ visits in firms)

  • the database of the COFACE (export insurance company).

The responses came mostly from the contacts taken from the database managed by Brittany Innovation, filled in by more than 100 advisers from 40 structures.

So it is not a totally representative sample of the regional business fabric but it is a sample based on a large set of SMEs potentially beneficiary of the innovation support offer. All important sectors for Brittany are represented, for some sectors we took only SMEs with more than 5 employees.

 The services of a call center were used for follow-up calls to 400 firms.

 Face-to-face interviews were carried out with 81 SMEs

·           38 interviews from contact lists sent by 25 intermediaries

·           43 interviews from contacts coming from the postal survey.

The sample had the following characteristics :

·           31% ICT, 25% services to business, 15% food-processing. This is not strictly representative of the regional economic fabric but key regional sectors are represented.

·          50% have less than 10 employees = very small enterprises (this is good because 99% of the regional fabric is made up of such firms)

·           very large range in terms of turnover (from 100K€ to 5000K€ in similar proportion)

·           78% said to have had a turnover in progress over the last 3 years

·           1/3 are less than 3 years old.

The representativeness of the innovation support offer was one of our main objectives: all structures analysed in Matrix 2 have been mentioned during the meetings but not at the same proportion. We decided to elaborate radar diagrams only for the structures mentioned by at least 20 firms but we did do the data processing for the whole data collected.

 We led open semi-directive discussion with firms about innovation process, needs, the received support and impacts of this support. Then the results of the interviews have been codified into the IMPACTSCAN typology and matrices.

 

Example: M3 - Madrid data gathering survey
Empirical research for M3 in Madrid has been based on the analysis of replies to 103 questionnaires administered through a survey to a stratified sample of 600 Madrid-based innovative companies. The results achieved have a high reliability according to the sample selection.

With regard to the method of capturing responses, questionnaires were followed up by telephone calls to remind and advise people in charge on filling the questionnaire. This effort has allowed the collection of significant number of answers in terms of statistical sampling.

Such monitoring, and subsequent tabulation of the questionnaires, provided the opportunity to verify that the survey had been completed properly, in accordance with the objectives previously set in the project. This question is particularly relevant in a study as conceptually complex as this one, since it was possible that the respondents provide their answers without having a clear awareness of the meaning of each word, and distort the results achieved. Thus, devoting personal attention to a significant number of respondents increases the validity of the survey.

In addition, to further strengthen the statistical validity of the research, part of the sample (11 company directors) have been interviewed personally by the research team. The findings from these interviews were an extremely useful input to draw conclusions from the exercise.

Stratification has been done on the basis of three variables:
• size in terms of staff numbers,
• industry sector depending on the areas of activity identified by interviewees and reclassified by reference to the CNAE national economic activity classification.
• innovative character given by firms investment in R & D
Data Exploitation has been implemented through a check of consistency using various recoding models (percentage of maximum response; percentage of minimum response; assigning numerical values to each category of response). Once the questionnaire has been recoded an exploration of data has generated the final results of the analysis.

 

Example : Data collection for M3 in Slovenia

A firm survey was launched in Slovenia. Two questionnaires were prepared. A first questionnaire was sent by e-mail to 2.500 addresses has 4 tables with the following contents:
T1: enablers/services
T2: intermediaries/enablers
T3: Any positive changes with regard to enablers in the firms in the last 2 years? (Y/N)
T4: services/changes with regard to enablers (in ref. to the T3)
M3: The response was 77 completed questionnaires by 15 May 2007. These data were processed: the average estimation of impact of services to enablers is given. In M3 only those companies’ responses were taken into consideration that used at least a (1) service.
Individual interviews were also carried out. Questions for each enabler were prepared in such a form that could be well understood by companies. We found out that companies with high-tech products and with longer period of operation were able to respond to the majority of questions. Some of them described their way of operation in details; some limited themselves to yes/no. Based on the answers we conclude that companies have also been learning from our questions, i.e. sometimes they said they would think about introducing and including additional innovation actions to their daily practice. They also defined their needs and desires. Some companies could not answer a whole spectre of questions relating to an enabler. This indicates a definitive need for new services which will close the gap in their knowledge in the field of innovation.
Our conclusions: the in depth questionnaire based on the set of ten enablers was found as a very good tool, not only to assess the existing innovation capacity of companies, but also to find out the missing knowledge and to plan introduction of new services to close the gap in their knowledge of innovation management.

Apart from new surveys, some existing tools need to be screened for their contribution to the exercise. Existing information gathering tools may include:
• Existing surveys: with a few exceptions, most existing surveys in EU regions are not oriented towards perception of the system of regional services by companies. Some inform about the perception of services by one intermediary only or one type of service (R&D grants e.g.). The results of these surveys need to be taken into account when gathering information for M3, provided that the results can be customised according to the standard services and innovation enablers categories;
• Working groups or panels of companies: this method, often used with the support of industry federations or chambers of commerce, is relatively easy and cheap to implement. However its important drawback is that it is unavoidable to introduce a bias in the sample of companies, since the types of companies participating to such exercises are likely to be already well acquainted with the regional system, and to present a higher than average innovation profile than the population of innovative firms in general. It seems more appropriate to use such tool as a complement to a survey, to discuss results;
• Use of company representatives: indirect sources of information on companies’ perception can also be gathered through the use of firm representatives such as industry federations. However, even more than the previous tool, this method suffers from the introduction of bias in the representation of companies (e.g. often larger firms are better represented than smaller ones). Furthermore, such representatives do sometimes play the role of an intermediary and therefore can rather be classified on the supply-side.

[top]

Scoring mechanism

Each cell of Matrix 3 reflects the influence of a type of service on a firms’ enabler, as a result of a comprehensive survey (see possible structure in Annex).
The results of the survey exploitation are transformed into the common 6 scale scoring system for Matrix 3:
0: no influence
1: very low influence
2: low influence
3: acceptable
4: high influence
5: very high influence
N.A: not applicable (service not used; impact unknown; missing, etc)
Note: it is important to differentiate between a score of 0 (no influence) and missing values, which should be treated as such.

Since the score measures the influence of a service on an innovation enabler, and not its visibility, IMPACTSCAN partners choose to compute average scores based on a representative sample of the beneficiaries of services.

Example : Data analysis for M3 in Lower Austria
For data analysis of the interviews among companies (which started end of March 2007) an expert organisation used the software “Matlab” to run various correlations of the 440 feedbacks on single services as well as the answers on current situation and company figures. But also comparisons of the impact perception of both sides (companies and intermediaries) were possible. Finally, two workshops took place on 19th and 20th of April 2007 where further brainstorming results of regional entrepreneurs who had not been interviewed within the face-to-face-in-depth interviews (control group) and the regional intermediary organisations as well as other experts were included in the analysis of the survey results.
Additionally, this analysis was matched with results of further data sources like:
-statistical analysis of the RISNÖ++ questionnaire survey 2002/2003
-analyses of other existing evaluations.

 

An alternative: using penetration rates in Limburg
When the total target population for a service is known, it would be interesting to weigh the scores by the rate of penetration of the service, defined as the ratio between number of firms beneficiaries of the service/total target population. This calculation has been carried out in Limburg on a project-by-project basis.