Moderating Role of Schema and Behavioral Training in the Relationship between Assessment Center Results with Performance Appraisal of Self, Colleagues and Supervisors

Zamanpour Boroujeni, Fatemeh; Oreyzi Samani, Hamid Reza; Zamanpour, Enayatollah

doi:10.22055/jiops.2024.45730.1394

Document Type : Research Article

Authors

¹ MA of Industrial Psychology, Department of Psychology, Faculty of Educational Sciences and Psychology, Isfahan University, Isfahan, Iran

² Professor, Department of Psychology, Faculty of Educational Sciences and Psychology, Isfahan University, Isfahan, Iran

³ Assistant Professor, Department of Assessment and Education, Faculty of Psychology and Education, Allameh Tabataba’i University, Tehran, Iran

https://doi.org/10.22055/jiops.2024.45730.1394

Abstract

Introduction
The role of assessors and their training in order to increase accuracy and reduce errors in the results of the assessment center is undeniable. Two types of schema and frame of reference (FoR) training are commonly used in rater training. Also, there have always been differences between the results of the assessment center and the results of self, peer, and observer evaluation. Therefore, the purpose of this research was to identify the moderating role of two types of schema training and the behavioral frame of reference (FOR) of the assessor in the relationship between the results of the assessment center and the performance evaluation of self, peers and supervisors.

Method
For this purpose, in the first phase of the research, assessment skills were trained to two groups of PhD and senior psychology students with different orientations using a schema and behavioral of frame of reference, and the third group was not provided with any training as a control group that asked them to assess two groups of 28 managers of the industrial complex located in Isfahan city. It is necessary to explain that the evaluated competencies included judgment, leadership, planning, interpersonal relations and information processing. These competencies were in fact finding simulation exercises, group discussion, role playing and oral presentation. Then, after 6 months in the second phase of the study, managers, peers and supervisors were asked to complete the performance appraisal form (Nouri .et al, 2007) to be used as a criteria variable in the analysis. Pearson product-moment correlation coefficient and Fisher Z-Transformation, the relationship between results of assessment center and performance appraisal (self, peer and supervisor) and the correlation difference were investigated.

Results
One of the most important results has been the effectiveness of training of assessors. In other words, the training of assessors of the schema and behavioral frame of reference (FoR) has been effective. Although the correlation between performance evaluation and assessment center’s results with assessors who received schema training is slightly higher than assessors who received behavioral training, this difference was not statistically significant.

Discussion
Training the assessors of the reference framework, either behaviorally or schematically, is one of the basic aspects of the assessment center (AC) and providing accurate results along with error reduction. Although there are differences between these types of training, but based on the results of this research, it can be concluded that the appraisal of self, peer, and supervisor are more similar to evaluators who have been trained in a schematic way. This result is due to the fact that in this type of training, the generality of behavior is evaluate, while in behavioral training, individual behaviors are observed, recorded, classified, and evaluated. In addition, assessor training methods as well as the direction of future research and practical and research suggestions are discussed.

Keywords

References

Armoneit, C., Schuler, H., & Hell, B. (2020). Nutzung, validität, praktikabilität und akzeptanz psychologischer personalauswahlverfahren in deutschland 1985, 1993, 2007, 2020: Fortführung einer trendstudie. Zeitschrift Für Arbeits-Und Organisationspsychologie, 64(2), 67-82. https://doi.org/10.1026/0932-4089/a000311

Badali, Z., & Amirkhani, A. (2013). Determining the key competencies of evaluation center evaluators. Organizational Behavior Studies Quarterly, 1(4), 61-81. (Persian) https://sid.ir/paper/472425/fa

Balantain, Y., & Poe, N. (2006). Managers evaluation and development centers (Translated by Masoud Soltani). Tehran: Asia. (Persian) https://www.gisoom.com/book/1407759/

Baret, N. (2018). How does frame-of-reference training increase rater accuracy? A test of potential explanatory mechanisms. school of management studies. https://open.uct.ac.za/bitstreams/43cc09af-b6eb-4e51-89b3-f51f19117558/

Bastian, P. K., Leonore Titze, J., & Blickle, G. (2022). Improving assessment center criterion validity for salesperson selection: A socioanalytic approach. Journal of Personal Selling & Sales Management, 42(2), 1-16. https://doi.org/10.1080.08853134.2022.2037436

Bernardin, H. J., & Buckley, M. R. (1981). Strategies in rater training. Academy of Management Review, 6, 205-212. https://doi.org/137.158.158.60

Breil, S. M., Lievens, F., Forthmann, B., & Back, M. D. (2022). Interpersonal behavior in assessment center role‐play exercises: Investigating structure, consistency, and effectiveness. Personnel Psychology, 76(3), 759–795. https://doi.org/10.1111/peps.12507

De Kock, F. S., Lievens, F., & Born, M. (2018). The profile of the ‘good judge’ in HRM: A systematic review and agenda for future research. Human Resource Management Review, 30(2), 100667 https://doi.org/10.1016.j.hrmr.2018.09.003

Fiske, S. T., & Taylor, S. E. (2013). Social cognition: From brains to culture. London: SAGE. https://books.google.com/books/about/Social_Cognition.html?id=FYth8i4Zg78C

Gibbons, A. M., & Rupp, D. E. (2009). Dimension consistency as an individual difference: A new (old) perspective on the assessment center construct validity debate. Journal of Management, 35(5), 1154-1180. https://doi.org/10.1177/0149206308328504

Gorman, C. A. (2008). Evaluating frame-of-reference rater training effectiveness via performance schema accuracy. PhD diss, University of Tennessee. https://doi.org/10.1037/a0016476

Gorman, C. A., Jackson, D. J., Meriac, J. P., Himmler, J. R., & Contreras, T. F. (2023). Beyond rating accuracy: Unpacking frame-of-reference assessor training effectiveness. Industrial and Organizational Psychology, 1(2), 1-14. https://doi.org/10.1017/iop.2024.6

Hauenstein, N. M. A., & McCusker, M. E. (2017). Rater training: Understanding effects of training content, practice ratings, and feedback. International Journal of Selection and Assessment, 25, 253-266. https://doi.org/10.1111/ijsa.12177

Hosseini, Y., Arman, M., & Mohamadizade, Z. (2017). The role of the evaluation center in staff improvement programs using artificial neural networks. Organizational Resource Management Studies, 7(3), 1-11. (Persian) http://ormr.modares.ac.ir/article-28-8427-fa.html

International Taskforce on Assessment Center Guidelines. (2015). Guidelines and ethical considerations for assessment center operations. Journal of Management, 41(4), 1244-1273. https://doi.org/10.1177/0149206314567780

Jamshidi, H., Badali, Z., & Hosseini, S. M. (2017). Key success factors in evaluation and development centers. Tehran: Industrial Management Organization. (Persian) https://www.gisoom.com/book/11331641/

Jansen, P. G. W., & Vinkenburg, C. J. (2006). Predicting management career success from assessment center data: A longitudinal study. Journal of Vocational Behavior ,68(2), 253-266. https://doi.org/10.1016/j.jvb.2005.07.004

Kark, R., Blatt, R., & Wiesel, V. (2023). A woman’s got to be what a woman’s got to be? How managerial assessment centers perpetuate gender inequality. Human Relations, 0(0), 1-12. https://doi.org/10.1177/00187267231161426

Khoshouei, M. S., Oreizy, H., & Nouri, A. (2014). Design and implementation of a manager promotion evaluation center. Quarterly Journal of Human Resource Management Research, Imam Hossein University, 6(3), 27-47. (Persian) https://hrmj.ihu.ac.ir/article_11748.html

Kogan, J. R., Dine, C. J., Conforti, L. N., & Holmboe, E. S. (2022). Can rater training improve the quality and accuracy of workplace-based assessment narrative comments and entrustment ratings? A randomized controlled trial. Academic Medicine, 98(2), 237-247. https://doi.org/10.1097/ACM.0000000000004819

Krause, D, E., & Thornton, G. C. (2009). A Cross-cultural look at assessment center practices: Survey results from western europe and north america. Applied Psychology: An International Review, 58(4), 557-585. https://doi.org/10.1111/j.1464-0597.2008.00371.x

Krause, D. E., Rossberger, R. J., Dowdeswell, K., Venter, N., & Joubert, T. (2011). Assessment center practices in South Africa. International Journal of Selection and Assessment, 19(3), 262-275. https://doi.org/10.1111/j.1468-2389.2011.00555.x

Landau, M. J., Meier, B. P., & Keefer, L. A. (2010). A metaphor-enriched social cognition. Psychological Bulletin, 136(6), 1045-1067. https://doi.org/10.1037/a0020970

Levin, K. (2023). Understanding the impact of age and gender demographic similarity in assessment center and individual assessment ratings. Doctoral dissertation, Illinois Institute of Technology. https://www.proquest.com/openview/

Lievens, F., & Sanchez, J. I. (2007). Can training improve the quality of inferences made by raters in competency modeling? A quasi‐experiment. Journal of Applied Psychology, 92(3), 812-819. https://doi.org/10.1037.0021‐9010.92.3.812

Lievens. F. (2001). Assessor Training Strategies and Their Effects on Accuracy, Interrater Reliability, and Discriminant Validity. Journal of Applied Psychology, 86, No. 2, 255-264. https://psycnet.apa.org/doi/10.1037/0021-9010.86.2.255

Maciejczyk, M. (1991). The relationship of self-ratings to assessor ratings of assessment center performance. Dissertation Abstracts International, 52, 4505. https://doi.org/10.31274/RTD-180813-9506

Macrae, C. N., & Bodenhausen, G. V. (2000). Social cognition: Thinking categorically about others. Annual Review of Psychology, 51(1), 93-120. https://doi.org/10.1146/annurev.psych.51.1.93

Meriac, J. P., Hoffman, B. J., & Woehr, D. J. (2014). A conceptual and empirical review of the structure of assessment center dimensions. Journal of Management, 40(5), 1269-1296. https://doi.org/10.1177/0149206314522299

Nouri, A., Labaf, H., Oreyzi, H., Rumi, H., Gorgani, H., … & Janatian, S. (2007). Psychological and comprehensive analysis of the performance appraisal system and career path of the managers of the National Iranian Gas Company and the province's gas companies. Isfahan. (Persian) Research project of Isfahan University

O’Neill, G., Travaglione, A., McShane, S., Hancock, J., & Chang, J. (2017). Converting values awareness to values enactment through frame-of-reference training. International Journal of Organizational Analysis, 25, 536-558. https://doi.org/10.1108/IJOA-02-2016-0975

Oreyzi, H., & Asadi, A. (2017). Design and implementation of Strategic Thinking Competency Assessment Center for managers of Iran Petroleum Products Refining and Distribution Company. Two Quarterly Journal of Contemporary Psychology, 12(1), 53-62. (Persian) https://bjcp.ir/article-1-1181-fa.html

Powell, D. M., & Bourdage, J. S. (2016). The detection of personality traits in employment interviews: Can “good judges” be trained? Personality and Individual Differences, 94, 194-199. https://doi.org/10.1016.j.paid.2016.01.009

Powell, D. M., & Goffin, R. D. (2009). Assessing personality in the employment interview: The impact of training on rater accuracy. Human Performance, 22, 450-465. https://doi.org/10.1080.08959280903248450

Pulakos, E. D. (1986). The development of training programs to increase accuracy with different rating tasks. Organizational Behavior and Human Decision Processes, 38, 76-91. https://doi.org/10.1016/0749-5978(86)90027-0

Radi Afsouran, N., Thornton, G., & Charkhabi, M. (2022). Leadership development assessment center: A review on advantages and disadvantages for developing leadership behavioral competencies. International Journal of Organizational Leadership, 11(1), 71-88. https://ijol.cikd.ca/article_60622_en.html

Roch, S. G., & O’Sullivan, B. J. (2003). Frame–of–reference rater training issues: Recall, time and behavior observation training. International Journal of Training and Development, 7, 93-107. https://doi.org/10.1111/1468-2419.00174

Roch, S. J., Woehr, D. J., Mishra,V., & Kieszczynska, U. (2012). Rater training revisited: An updated meta-analytic review of frame-of-reference training. Journal of Occupational and Organizational Psychology, 85, 370-395. https://doi.org/10.1111/j.2044-8325.2011.02045.x

Sackett, P. R., Shewach, O. R., & Keiser, H. N. (2017). Assessment centers versus cognitive ability tests: Challenging the conventional wisdom on criterion-related validity. J. Appl. Psychol, 102, 1435-1447. https://doi.org/10.1037.ap1000 0236

Schleicher, D. J., Day, D. V., Mayes, B. T., & Riggio, R. E. (2002). A new frame for frame‐of‐reference training: Enhancing the construct validity of assessment centers. Journal of Applied Psychology, 87(4), 735-746. https://doi.org/10.1037.0021‐9010.87.4.735

Shakeri, A., Mousavi, Z., Noori, A., & Oreyzi, H. (2018). 360-degree evaluation validity through the evaluation center in the National Iranian Gas Company. Quarterly Journal of Human Resource Management in the Oil Industry, 11(41), 243-262. (Persian) https://www.sid.ir/paper/196594/fa

Shakery, I. (2016). Validating the assessment of traditional performance evaluation and competency-based assessment through the assessment center. Industrial and Organizational Psychology Studies, 3(1), 57-70. (Persian) https://doi.org/10.22055/jiops.2018.21906.1049

Shakery, I., Mehdad, A., & Farhadi, A. (2023). Designing and validating the test of situational judgment mental toughness of managers in the Iran Alloy Steel Company. Industrial and Organizational Psychology Studies, 10(1), 43-56. (Persian) https://doi.org/10.22055/jiops.2023.43449.1347

Taylor, I. (2007). A practical guide to assessment centres and selection method: Measuring competency for recruitment and development. London: Kogan Page. https://books.google.com/books/about/

Thornton, G. C., & Lievens, F. (2018). Theoretical principles relevant to assessment center design and implementation. Assessment centres: Unlocking potential for growth. 1-36. Research Collection Lee Kong Chian School of Business. Available at: https:ink.library.smu.edu.sg.lkcsb_research.6222.

Thornton, G. C., & Rupp, D. E. (2006). Assessment centers in human resource management: Strategies for prediction, diagnosis and developmant. Lawrence Erblaum Associates, 1(2). 3-23. https://doi.org/10.4324/9781410617170

Thornton, G. C., Rupp, D., & Mueller-Hanson, R. A. (2004). Developing organizational simulations: A guide for practitioners and students. Lawrence Erlbaum Associat. https://doi.org/10.4324/9781410609458

Thornton, J. (2007). Centers for evaluating human resource management(Translation: Saeed Jafari, Saeed Shahbazi, Majid Salimi). Tehran: University of Tehran, Faculty of Entrepreneurship. (Persian) https://www.gisoom.com/book/1442483/

Tsai, M. H., Wee, S., & Koh, B. (2019). Restructured frame‐of‐reference training improves rating accuracy. Journal of Organizational Behavior, 40(6), 740-757. https://doi.org/10.1002/job.2368

Vanhove, A. J., Graham, B. Z., & Thornton, G. C. (2023). Moderators of sex‐and race‐based subgroup differences in assessment center ratings: A meta‐analysis. International Journal of Selection and Assessment, 31(1), 63-91. https://doi.org/10.1111/ijsa.12411

Wirz, A., Melchers, K. G., Kleinmann, M., Lievens, F., Annen, H., & Blum, U. (2020). Do overall dimension ratings from assessment centers show external construct-related validity? European Journal of Work and Organizational Psychology, 29(3), 405-420. https://doi.org/10.1080/1359432X.2020.1714593

Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67, 189-205. https://doi.org/10.1111/j.2044-8325.1994.tb00562.x

Zamanpour, E. (2017). Validation of the competency model of oil industry managers in evaluation and development centers: Application of procedural theory. Doctoral dissertation, Allameh Tabatabai University, Tehran. (Persian) https://ganj.irandoc.ac.ir/viewer/0bace92075ee5dc46fcfeefb63636e6b?sample=1

Moderating Role of Schema and Behavioral Training in the Relationship between Assessment Center Results with Performance Appraisal of Self, Colleagues and Supervisors

References

References

Volume 11, Issue 1April 2024Pages 17-32

Volume 11, Issue 1
April 2024
Pages 17-32