Incomplete block design as an alternative for project evaluation in governmental and non-governmental organizations

Osval Antonio Montesinos López, Francisco Javier Luna Vázquez, Emeterio Franco Pérez, Alberto Pastrana Palma, Raymundo Buenrostro Mariscal

Abstract


Introduction: The transparent evaluation of projects proposals for scientific research with financial support through calls made by governmental institutions or private organizations becomes extremely important, along with increasing confidence in the researchers who participate in the calls for the allocation of financial resources. Therefore, this research presents an alternative to the process of evaluating projects with financial resources. The current method of evaluating funded projects is to obtain an average of the scores assigned to each project by a group of reviewers. However, this method does not discount the effect of the reviewer and produces a bias in the final evaluation of each project, since projects are likely to be favored when reviewers are relaxed and affected when reviewers are demanding and very strict.               

Method: This research proposes the use of two experimental design statistical methods for project evaluation: (a) incomplete balanced block design and (b) Alpha block design. In addition, we provide codes in the statistical software R, for both statistical designs, which allows us to implement project evaluation quickly and accurately.                  

Results: Based on the proposed methods, it was found that the traditional method presents a very important bias due to the heterogeneity among the reviewers.                     

Discussion and Conclusion: The use of experimental design statistical methods (balanced incomplete block design or Alpha design) can reduce this bias considerably, obtaining a higher accuracy for evaluations.


Keywords


incomplete block design; projects evaluation; projects for scientific research; experimental design statistical; calls for the allocation of financial resources

References


Campos-Arceiz, A., Primack, R., & Koh, L. P. (2015). Reviewer recommendations and editors’ decisions for a conservation journal: Is it just a crapshoot? And do Chinese authors get a fair shot? Biological Conservation, 186, 22-27. doi:http://dx.doi.org/10.1016/j.biocon.2015.02.025

Bates, D., Máchler, M., Bolker, B., & Walker, S. (2015). Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67, 1--48. doi:10.18637/jss.v067.i01

CONACYT. (2015). Convocatoria 2016 - Proces de evaluación de las propuestas y selección para apoyo. Proceso de evaluación de las propuestas y selección para apoyo, Consejo Nacional de Ciencia y Tecnologia. Obtenido de https://www.conacyt.gob.mx/index.php/sni/convocatorias-conacyt/convocatorias-programa-de-estimulos-a-la-innovacion/convocatoria-2016/11036-proceso-de-evaluacion-2016/file

CONACYT. (2016). Evaluación 2016 y Preguntas frecuentes. Correo, Consejo Nacional de Ciencia y Tecnologia. Obtenido de https://www.conacyt.gob.mx/index.php/sni/convocatorias-conacyt/convocatorias-programa-de-estimulos-a-la-innovacion/convocatoria-2016/10586-evaluacion-2016-preguntas-frecuentes/file

De Mendiburu, F. (2017). agricolae: Statistical Procedures for Agricultural Research. Obtenido de https://CRAN.R-project.org/package=agricolae

ELSEVIER. (2009). Home. Recuperado el 08 de octubre de 2018, de https://www.elsevier.com/reviewers/what-is-peer-review

ELSEVIER. (2018). https://www.elsevier.com/reviewers/what-is-peer-review. Recuperado el 19 de octubre de 2018

González- Hernández, I. J. (2006). Diseños experimentales de bloques incompletos y aplicaciones en la industria. Pachuca de Soto, Hidalgo, México: Instituto de ciencias básicas e ingeniería. Recuperado el 11 de Octubre de 2017

Maj Gen A.K, .. (April de 2016). Peer review' for scientific manuscripts: Emerging issues, potential threats, and possible remedies. Medical Journal Armed Forces India, 72(2), 172-174. doi:https://doi.org/10.1016/j.mjafi.2016.02.014

Mead, R. (2010). The design of experiments. New York, USA: Cambridge University Press. doi:ISBN 9780521287623

Mendoza, H., & López, L. A. (2001). Estimación de datos faltantes en bloques incompletos balanceados conectados. Revista Colombiana de Estadística, 24(2), 73-89. Obtenido de http://www.redalyc.org/articulo.oa?id=89924201

Muhammad , A., & Wahida, S. (2018). The Application of Algebraic Methods in Balanced. Journal of Physics:, 1028(1), 012123. doi:10.1088/1742-6596/1028/1/012123

Publishing Research Consortium. (2016). Publishing Research Consortium Peer review survey 2015. 58. Obtenido de https://www.elsevier.com/__data/assets/pdf_file/0007/655756/PRC-peer-review-survey-report-Final-2016-05-19.pdf

Sailer, M. O. (2013). crossdes: Construction of Crossover Designs. Obtenido de https://CRAN.R-project.org/package=crossdes

SINECYT. (2012). Código de ética del sistema nacional de evaluación científica y tecnológica. Obtenido de CONACYT: 2006-2012.conacyt.gob.mx/registros/sinecyt/Documents/CODIGO_ETICA.pdf

Spier, R. (2002). The history of the peer-review process. TRENDS in Biotechnology, 20(8), 357-358. doi:doi.org/10.1016/S0167-7799(02)01985-6

Wickham, H., & Henry, L. (2017). tidyr: Easily Tidy Data with 'spread()' and 'gather()' Functions. Obtenido de https://CRAN.R-project.org/package=tidyr

Wickham, H., Francois, R., Henry, L., & Müller, K. (2017). dplyr: A Grammar of Data Manipulation. Obtenido de https://CRAN.R-project.org/package=dplyr

Zuckerman, H., & Merton, R. (1972). Patterns of Evaluation in Science:Institutionalisation, Structure and Functions of the Referee System. Minerva, 9, 66-100. doi:https://doi.org/10.1007/BF01553188]




DOI: https://doi.org/10.21640/ns.v11i22.1867

Refbacks

  • There are currently no refbacks.


Copyright (c) 2019 Nova Scientia

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Scope

Nova Scientia is a multidisciplinary, electronic publication that publishes biannually in the months of May and November; it is published by the Universidad De La Salle Bajío and aims to distribute unpublished and original papers from the different scientific disciplines written by national and international researchers and academics. It does not publish reviews, bibliographical revisions, or professional applications.

Nova Scientia, year 11, issue 22, May – October 2019, is a biannual journal printed by the Universidad De La Salle Bajío, with its address: Av. Universidad 602, Col. Lomas del Campestre, C. P. 37150, León, Gto. México. Phone: (52) 477 214 3900, http://novascientia.delasalle.edu.mx/. Chief editor: Ph.D. Ramiro Rico Martínez. ISSN 2007 - 0705. Copyright for exclusive use No. 04-2008-092518225500/102, Diffusion rights via computer net 04 - 2008 – 121011584800-203 both granted by the Instituto Nacional del Derecho de Autor.

Editor responsible for updating this issue: Direction of Research Department of the Universidad De La Salle Bajío, last updated on May 28th, 2019.