1、中文 2287 字 Comparative evaluation of performance of national R&D programs with heterogeneous objectives: A DEA approach1 Hakyeon Lee a, Yongtae Park a,*, Hoogon Choi Abstract The strategic importance of performance evaluation of national R&D programs is highlighted as the resource allocation draws mo
2、re attention in R&D policy agenda. Due to the heterogeneity of national R&D programs objectives, however, it is intractably difficult to relatively evaluate multiple programs and, consequently, few studies have been conducted on the performance comparison of the R&D programs. This study measures and
3、 compares the performance of national R&D programs using data envelopment analysis (DEA). Since DEA allows each DMU to choose the optimal weights of inputs and outputs which maximize its efficiency, it can mirror R&D programs unique characteristics by assigning relatively high weights to the variabl
4、es in which each program has strength. Every project in every R&D program is evaluated together based on the DEA model for comparison of efficiency among different systems. Kruskal Wallis test with a post hoc Mann Whitney U test is then run to compare performance of R&D programs. Two alternative app
5、roaches to incorporating the importance of variables, the AR model and output integration, are also introduced. The results are expected to provide policy implications for effectively formulating and implementing national R&D programs. 1. Introduction As R&D has been considered as a driving force fo
6、r national competitive advantage, many countries have been raising R&D investments through various national R&D programs (Lee et al., 1996). Since R&D investment is one of the most decisive elements in promoting scientific and technological progress (Wang and Huang, 2007), the effective use of the l
7、imited R&D resources can be regarded as a prerequisite for benefiting from formulation and implementation of national R&D programs. Thus, performance evaluations of R&D programs need to be made so that the limited resources are allocated to promising R&D programs and poor R&D programs can be improve
8、d or terminated. Although a number of studies have been conducted to measure R&D performance at various levels, few attempts have been made at the national program-level. This is due to the heterogeneity of R&D programs in terms of policy purpose. Since each R&D program has its own primary objective
9、 such as publishing academic papers for basic research, issuing patents and developing prototypes for applied research, and providing funds with researchers for R&D human resource development, it is intractably difficult to relatively compare the performance of various national R&D programs at the s
10、ame time and in the same context. 1 来源： European Journal of Operational Research 196 (2009) 847 855 Two conventional approaches to assessing R&D performance, peer review and bibliometric method do not work well for the relative evaluation of heterogeneous R&D programs. The peer review method, which
11、is based on perceptions of well-informed experts about various quality dimensions of R&D, is inherently subjective and likely to be biased depending on interests, experience, and knowledge of the evaluators (Nederhof and van Raan, 1987； Brinnet al., 1996). The bibliometric method is considered relat
12、ively objective, but the results highly depend on the measurement method (Nederhof and van Raan, 1993). The tenet of this paper is data envelopment analysis (DEA) can overcome these limitations. DEA is a linear programming model for measuring the relative efficiency of decision making units (DMUs) w
13、ith multiple inputs and outputs (Cooper et al., 2000). Since it can not only handle multiple outputs, but also allow each DMU to choose the optimal weights of inputs and outputs which maximize its efficiency (Cherchye et al., 2007), it is capable of mirroring R&D programs unique characteristics by a
14、ssigning high weights to the variables in which each project has strength. This study measures and compares the performance of six national R&D programs in Korea using the DEA efficiency. DEA is a non-parametric approach that does not require any assumptions about the functional form of a production
15、 function and a priori information on importance of inputs and outputs. The relative efficiency of a DMU is measured by estimating the ratio of weighted outputs to weighted inputs and comparing it with other DMUs. DEA allows each DMU to choose the weights of inputs and outputs which maximize its eff
16、iciency. The DMUs that achieve 100% efficiency are considered efficient while the other DMUs with efficiency scores below 100% are inefficient. The first DEA model proposed by Charnes et al. (1978) is the CCR model that assumes that production exhibits constant returns to scale. Banker et al. (1984)
17、 extended it to the BCC model for the case of variable returns to scale. When it comes to R&D returns to scale, findings from previous studies are somewhat mixed (Graves and Langowitz, 1996). It was found that R&D activity can exhibit increasing or decreasing returns to scale as well as constant ret
18、urns to scale (Bound et al., 1984； Scherer, 1983)； thus, the BCC model is employed in this study. DEA models are also distinguished by the objective of a model: maximize outputs (output-oriented) or minimize inputs (input-oriented). It is implicitly assumed that the objective of R&D lies in increasi
19、ng outputs rather than decreasing inputs. Therefore, this study adopts the output-oriented model. 2. Conclusions We measured and compared the performance of the six national R&D programs with heterogeneous objectives using DEA. Every project in every program was evaluated together, and Kruskal Walli
20、s test with a post hoc Mann Whitney U test was then run to compare performance of R&D programs. The two alternative approaches to incorporating the importance of variables in reality, the AR model and output integration, were also considered. Due to the heterogeneity of national R&D programs objecti
21、ves, few studies have been conducted on the performance comparison among programs. This study contributes to the field in that it filled the void by applying DEA to the national R&D programs. DEA, particularly the model for comparison of efficiency between different systems, was proved to be effecti
22、ve for performance comparison among R&D programs with heterogeneous objectives. The DEA results are expected to provide practical implications for policy making on national R&D programs. The limited resources can be effectively allocated to several R&D programs based on their performance rankings. R
23、&D programs doing well (e.g. Program C and D) deserve more investments； on the other hand, poor programs (e.g. Program A and F) have to be terminated or funds given to them should be cut down unless their performance is improved. Basically, DEA offers the way of improving efficiency for inefficient
24、DMUs although it is not explicitly dealt with in this study. Each inefficient project is provided with the reference set consisting of efficient projects for benchmarking, which in turn results in performance improvement of programs. However, what DEA tells us is the way of improving efficiency, tha
25、t is, how many outputs should be increased to achieve 100% efficiency, not the way of increasing actual outputs at the current setting. To seek the way of enhancing performance, the reasons for poor performance should be uncovered by examining the context in which poor programs are formulated and im
26、plemented, such as project selection procedure, operational regulation, funding systems, etc. It is obvious that the prerequisite for this is to be able to measure and compare the performance of various R&D programs, which is the primary contribution of this study. Nevertheless, this study is subjec
27、t to some limitations. Firstly, since the projects that have not been finished at the time of data collection were not included, program performance was measured without them. Secondly, despite the fact that it takes several years for R&D outputs to be achieved, the outputs produced only for two yea
28、rs after termination of projects were considered. These limitations will be overcome if the analysis is conducted again at some time in future. Thirdly, it may occur that a R&D program is considered as a high-performer, even though they failed to achieve its own objectives, but accomplished excellen
29、t outcomes in another area. Although it was not found in this study, when this is the case, judgment could be controversial. These issues should be dealt with in future research. Another fruitful avenue for future research is to employ various types of extended DEA models and compare the results. Considering another model is expected to lead us to seek a better way of evaluating and comparing the performance of national R&D programs with heterogeneous objectives.
