Modification of CSE-UCLA and Discrepancy as a New Evaluation Model in Education

The absence of an evaluation model that can be used to evaluate the effectiveness of educational activities at vocational universities in the view of six domains causes the need to find innovations in an evaluation model. The six domains, included: context, input, socialization process, implementation process, results, and imbalances that occur in the learning implementation. An evaluation model innovation was initiated through this research in the form of a modification of two educational evaluation models. Those models are CSE-UCLA and Discrepancy. Based on those conditions, the main objective of this research was to demonstrate the existence of a new evaluation model design in education which was a modification of the CSE-UCLA evaluation model and the Discrepancy evaluation model. Specifically, this evaluation model design can be used to determine the effectiveness of the one education activity namely the synchronous learning implementation at vocational universities. The research method used to achieve the objectives of this research was the development method by the Borg and Gall model development design. Five stages of development become the focus of this research, included: research and field data collection, research planning, design development, initial trials, and main product revision. The location of this research was conducted in several vocational universities in the North Bali area (Indonesia). Subjects involved in the initial trials toward this evaluation model design were 20 people. The tools used to conduct the initial trials were questionnaires. The technique used to analyze the data of initial trial results was descriptive quantitative by comparing the percentage of the initial trial results with the standard percentage of the effectiveness level of modified model design. This research results indicate that the evaluation model design was categorized as effective by an effectiveness level was 81.93%. The implication of this research results was the existence of a new design of the educational evaluation model. This design combines two evaluation models to produce good functionality integration in determining the effectiveness level of the synchronous learning implementation in all universities generally and in vocational universities in Bali particularly.

The reason for making modifications to those two evaluation models is because they have evaluation components that have similar functionality and complement each other. If those two models are combined, it can be used to completely and thoroughly evaluate the synchronous learning implementation in the viewed of the context domain, input domain, socialization process domain, implementation process domain, results domain, and imbalances domain. The context domain can be evaluated  [24] showed the use of the Discrepancy evaluation model to measure the effectiveness level of synchronous e-learning. The research results of Sudarwati & Rukminingsih only were able to show the imbalances that occur in the implementation of synchronous e-learning but weren't able to show the details of the mechanisms and processes of socializing the existence of synchronous elearning to the schools' community. Divayana et al.'s research [25] showed the use of the CSE-UCLA model to evaluate the learning effectiveness of expert system subjects based on mobile technology. The results of Divayana et al.'s research indicated the effectiveness of the learning implementation of expert system subjects from the context domain, input domain, process domain, results domain, and also the socialization process domain. However, the results of Divayana et al.'s research weren't able to show imbalances in the learning process. The efforts to facilitate understanding toward some of the results of previous research that background this research, it is necessary to show through a research roadmap. The roadmap for this research can be seen in Figure 1.

Figure 1. Research Roadmap.
Based on the visualization of Figure 1, it appears that the results of this research were in the form of a modified CSE-UCLA evaluation model design with the Discrepancy model. The presence of this model design is an answer to several obstacles from previous studies starting from 2018-2020. The evaluation model design resulted from this research can be used as a description of the process carried out in evaluation activities toward the effectiveness of the synchronous learning implementation at vocational universities.

2-1-Research Goal, Research Approach, and Research Method
The goal of this research was to determine the modified form of the CSE-UCLA model and the Discrepancy model used to determine the effectiveness level of synchronous learning. The effort was made to achieve that goal was to conduct research using a development approach. The development method utilized in this research was based on the development design of the Borg and Gall model which consists of ten stages [29,30]. Those ten stages of development can be seen in Figure 2.

2-2-Research Location, Sample, and Technique of Data Collection
This research was conducted in several vocational universities in Bali Province, especially in North Bali. The reason for this research location in the North Bali area was because this area is the center of education in Bali, with the characteristics of students who came from various regions in Indonesia. Besides, access to information technology in North Bali is not superior compare with South Bali. However, this precisely makes it easier to determine the effectiveness of synchronous learning. If the area that has limited information technology can effectively carry out synchronous learning, so logically, the area that already has superior facilities and access to information technology, certain can carry out synchronous learning more effectively.
Subjects in this research were determined using a purposive sampling technique. Purposive sampling was conducted by determining the parties directly related to the implementation of synchronous learning at vocational universities in North Bali. The reason for using this purposive sampling technique was because this technique was very appropriate to be used to obtain complete information from parties who had knowledge and experience of synchronous learning [31,32]. Subjects involved in this research were three research members and three field workers, especially at the stage of research and field data collection, research planning, and design development. The subjects involved in the initial trials of the evaluation model design were ten evaluators and ten lecturers in the field of vocational from several universities in North Bali. The subjects involved in the main product revision were three members of the research team in this research.
The data collection tools of the initial trial results toward the model design were questionnaires. The questionnaires consist of 15 questions. Each question was assessed using five types of assessment scores that follow the Likert scale. A score of 5 means excellent, a score of 4 means good, a score of 3 means enough, a score of 2 means less, and a score of 1 means poor [33][34][35]. The content validity tests of the questionnaire questions were carried out by two educational evaluation experts. The formula used to content validity tests of the questionnaire questions was the Gregory formula. The Gregory formula can be seen in Equation 1 [36][37][38].
Notes: = cell indicating disagreement between the two judges; and = cells indicating the difference in views between the judges; = cell indicating valid agreement between the two judges.

2-3-Analyzing of Data
The analysis technique of the questionnaire questions was carried out by comparing the results of the content validity tests with the content validity score standards which refer to the Guilford category. The categorization of content validity refers to Guilford [39,40], included: a validity score of 0.80 to 1.00 means very high validity, 0.60 to 0.80 means high validity, 0.40 to 0.60 means sufficient validity, 0.20 to 0.40 means low validity, 0.00 to 0.20 means very low validity, a validity score < 0.00 means invalid.
The analysis technique used in analyzing the trial results of the modified evaluation model design was quantitative descriptive. This technique was done by comparing the percentage of trial results with the percentage of the effectiveness standards of the modified evaluation model design. The formula was used to calculate the percentage of trial results follows Equation 2 [41], while the percentage of the effectiveness standards of the modified evaluation model design follows the percentage range of effectiveness scores. The percentage score in the range of 0% to 39% means poor, the percentage score in the range of 40% to 64% which means less, the percentage score in the range of 65% to 79% means enough, the percentage score in the range 80% to 89% means effective, and the percentage score in the range of 90% to 100% means excellent [42].
Notes: = total of acquisition values; = total of maximum scores.

3-Results and Discussion
The results obtained at the stage of research and field data collection were data related to components and aspects of the CSE-UCLA evaluation model and the Discrepancy evaluation model which were used as references in evaluating the implementation of synchronous learning. The components and aspects of the CSE-UCLA evaluation model completely can be seen in Table 1, while the components and aspects of the Discrepancy evaluation model can be seen in Table 2.

Table 1. Components and Aspects of the CSE-UCLA Evaluation Model as References for Evaluating the Implementation of Synchronous
Learning at Vocational Universities in North Bali.

AP1
The purpose of synchronous learning implementation AP2 Support from the academic community in each vocational university in North Bali

AP3
Regulations that support the implementation of synchronous learning

AP4
The readiness of students in providing internet data packages

AP5
The readiness of students in providing computer hardware to support synchronous learning

AP6
The student's ability to operate the platform used in implementing the synchronous learning

AP7
The readiness of lecturers in providing internet data packages

AP8
The lecturers' ability to provide interesting teaching materials and suitable for use in synchronous learning

AP9
The readiness of lecturers in providing computer hardware to support synchronous learning AP10 The lecturers' ability to operate the platform used in implementing the synchronous learning Program Implementation

AP11
Socialization of the existence of synchronous learning by lecturers through distributing platform links to students AP12 Socialization of links of teaching materials to students by lecturers before or after synchronous learning implementation AP13 Socialization of the guideline of synchronous learning implementation to students by the lecturers

AP14
The mechanism for making interesting teaching materials in digital format

AP15
The mechanism for creating an account in the platform that is used to support synchronous learning AP16 Mechanism of synchronous learning implementation

AP17
Students' satisfaction in the ease to operate the platform that is used to support the synchronous learning implementation AP18 Lecturers' satisfaction in the ease to operate the platform that is used to support the synchronous learning implementation

AP19
The safety of teaching materials that are distributed to students in synchronous learning AP20 The satisfaction of students and lecturers in the interactions and communications that occur through the platform which used to support the synchronous learning implementation Procedures for creating an account in the platform for lecturers and students so that they can access the platform which is used for the synchronous learning process AD9 Procedures for implementing synchronous learning so that it runs effectively

AD10
The satisfaction of students and lecturers toward the ease to operate a platform for synchronous learning AD11 The satisfaction of students and lecturers toward the access speed into the platform is used in synchronous learning

AD12
The security level of digital format teaching materials that are distributed by lecturers to students

AD13
The satisfaction of students and lecturers in communicating and interacting through the platform that supports synchronous learning AD14 Imbalances score of synchronous learning implementation Table 1 shows the existence of 20 evaluation aspects of the CSE-UCLA model. Three aspects of the system assessment component are used to evaluate things related to objectives, support, and regulations in implementing synchronous learning. Seven aspects of the program planning component are used to evaluate things related to infrastructure readiness and the readiness of lecturers' ability and students' ability in implementing synchronous learning. Three aspects of the program implementation component are used to evaluate things related to the socialization of synchronous learning implementation. Three aspects of the program improvement component are used to evaluate things related to the mechanism and process of implementing synchronous learning. Four aspects of the program certification component are used to evaluate things related to the results of implementing synchronous learning is viewed from the satisfaction of students and lecturers. Table 2 shows the existence of 14 evaluation aspects of the Discrepancy model. Three aspects of the definition component are used to evaluate things related to the vision, mission, goals, support, and legality of implementing synchronous learning. Three aspects of the installation component are used to evaluate things related to infrastructure readiness, the readiness of lecturers and students, and the readiness of the systems/platforms management team used in implementing synchronous learning. Three aspects of the process component are used to evaluate things related to the implementation of synchronous learning. Four aspects of the product component are used to evaluate things related to satisfaction results and the inequality score of the synchronous learning implementation.
The results obtained at the planning stage of this research were data about the description of activities, the number of personnel who were involved, and the completion time of the CSE-UCLA model design modified with the Discrepancy model. The data of research planning completely can be seen in Table 3. Three research members and three field workers Initial trials of the evaluation model design 14 20 Ten evaluators and ten lecturers at vocational universities in North Bali Revision of the initial trial results of the evaluation model design 7 3 Three research members Table 3 shows five important activities planned to finalize the design of the CSE-UCLA model that modification with the Discrepancy model. The total time planned to complete the design was 30 days. The total numbers of personnel who were planned to be involved in completing the design were 26 people, included: three researchers, three field staff, ten lecturers, and ten evaluators.
The results obtained at the stage of design development were the modified design of the CSE-UCLA model and the Discrepancy model. The design of this model showed an innovative evaluation model that can be used to determine the synchronous learning effectiveness in vocational universities. The model design intended completely can be seen in Figure 3.  The implementation of synchronous learning can be said to be effective if the effectiveness level in the determinants of effectiveness (context, input, socialization process, creation/implementation process, results, and imbalances) is categorized as effective. The effort that was made to determine the effectiveness level of synchronous learning was determined using a modification of the CSE-UCLA model and the Discrepancy model. The CSE-UCLA model has five evaluation components, included: 1) system assessment, 2) program planning, 3) program implementation, 4) program improvement, and 5) program certification.
System assessment is an evaluation component that provides information about the state or position of the system/program being evaluated. Program planning is an evaluation component that helps select certain programs that are likely to be successful in fulfilling program needs. Program implementation is an evaluation component that provides information on whether the program has been introduced to certain groups as planned. Program improvement is an evaluation component that provides information about how the program is functioning or working, whether it leads to certain achievements. Program certification is an evaluation that provides information about the value or results and benefits of the program [19].
The CSE-UCLA model has 20 evaluation aspects. Three aspects (AP1 to AP3) are used to evaluate the context domain, seven aspects (AP4 to AP10) are used to evaluate the input domain, three aspects (AP11 to AP13) are used to evaluate the socialization process domain, three aspects (AP14 to AP16) are used to evaluate the domain of the creation/implementation process, four aspects (AP17 to AP20) are used to evaluate the outcome domain. A description of all evaluation aspects of the CSE-UCLA model had been shown previously in Table 1.
The Discrepancy model has four evaluation components, included: 1) definition, 2) installation, 3) process, and 4) product. The definition is an evaluation component to assess the completeness and consistency of the program design. The installation is an evaluation component to assess the suitability of program installations. The process is an evaluation component to assess the extent to which the program is implemented. The product is an evaluation component to assess the results of program implementation [21,24].
The Discrepancy model has 14 evaluation aspects. Three aspects (AD1 to AD3) are used to evaluate the context domain, three aspects (AD4 to AD6) are used to evaluate the input domain, three aspects (AD7 to AD9) are used to evaluate the creation/implementation process domain, four aspects (AD10 to AD13) are used to evaluate the product domain, and one aspect (AD14) is used to evaluate the imbalances domain. A description of all evaluation aspects of the Discrepancy model had been shown previously in Table 2. The CSE-UCLA model does not have an aspect that is used to evaluate the imbalances domain, but it has been assisted by the AD14 aspect of the Discrepancy model to evaluate the imbalances domain. The Discrepancy model does not have an aspect that is used to evaluate the domain of the socialization process, but it has been assisted through the AP11 to AP13 aspects of the CSE-UCLA model to evaluate the domain of socialization process. Recommendations for the improvement of the synchronous learning process are given based on the constraints found in all the determinants of the effectiveness of synchronous learning implementation.
The results obtained at the stage of initial trials were used as the basis for making improvements to the modified design of the CSE-UCLA and Discrepancy models. The results of the initial trials were in the form of quantitative and qualitative data. The quantitative data showed the effectiveness level percentage of the evaluation model design modified, while the qualitative data showed several arguments given by the respondents as the basis for improvements to the modified design. The details regarding the quantitative data from the initial trial results can be seen in Table 4, while the qualitative data from the initial trial results can be seen in Table 5.  Table 4 shows the average percentage of effectiveness obtained from the initial trials conducted by 20 respondents toward the modified design of the CSE-UCLA and Discrepancy models. Fifteen question items were used as a basis for evaluating that design. Fifteen questions were used to obtain data from the initial trial results. Item-1 is a question about the availability of evaluation aspects  Table 5 shows six important suggestions given by respondent-2, respondent-10, respondent-14, respondent-16, respondent-18, and respondent-20. All of those suggestions were very well used as a basis for improvements to the design of the CSE-UCLA model that modification with the Discrepancy model. Based on several arguments given by respondents as shown in Table 5, a revision was made to the modified design of the CSE-UCLA and Discrepancy models. The revised design can be seen in Figure 4.

Respondents Arguments
Respondent-2 It was necessary to put a mark or lines that distinguish between the components and aspects of the evaluation in both the CSE-UCLA model and the Discrepancy model

Respondent-18
It was necessary to provide dividing lines to distinguish between the components and aspects of the evaluation between the two evaluation models.
Respondent-20 It was necessary to show evidence of the recommendations given for improving the synchronous learning process   Evidence of improvement according to the suggestions from respondent-20 is indicated by one word of "RECOMMENDATIONS" in blue and using a capital letter which goes to the box "Synchronous Learning".

CSE-UCLA Evaluation Model Discrepancy Evaluation Model
The percentage of effectiveness of 81.93% which was shown in Table 4 showed that the modified design of the CSE-UCLA and Discrepancy models included in the effective category. This is because that percentage is in the range of 80% to 89% which refers to the scores of effectiveness percentage. Generally, the modified results of the CSE-UCLA and Discrepancy models are effectively used as an evaluation model to determine the effectiveness of the synchronous learning implementation if viewed from that effective categorization. Based on that categorization, it can also be stated that the modified model is ready for use and does not need to be revised again. However, according to the positive and constructive arguments that had been given by respondents during the initial trials, it was very good and reasonable making the revisions to improve the evaluation model design.
The key parameters needed to determine the success for realizing this modification model design are the program implementation component owned by the CSE-UCLA model and the product component owned by the Discrepancy model. The reason for used those two components as important parameters is because functionally those components complement each other. The limitation of the Discrepancy model in its inability to evaluate the socialization process domain can be complemented by the program implementation component. Likewise, the limitation of the CSE-UCLA model in its inability to evaluate the imbalances domain can be complemented by the product component.
The novelty of this research is the combination of two different evaluation models that can be used to evaluate the effectiveness of the synchronous learning implementation. The integration of evaluation components of two models can be used to measure the synchronous learning effectiveness from the context domain, input domain, socialization process domain, implementation process domain, results domain, and imbalances domain. If this research compared with the research was conducted by the authors previously in 2018 about the use of the CSE-UCLA model to evaluate the blended learning program at SMA Negeri 1 Ubud [26], so it has certainly a clear difference. The difference is that research conducted in 2018 had not shown specifically the evaluation component that was able to be used to measure the imbalances in the blended learning implementation.  [43] showed the use of the CIPP model to evaluate the learning quality in health universities. The limitations of Toosi et al.'s research were that it had not shown evaluation components that measure the learning quality from the socialization process domain and the imbalances domain that occurs in the learning implementation. Lee et al.'s research [44] demonstrated the use of the CIPP model to evaluate online learning in health education. The limitations of Lee et al.'s research were that it had not shown evaluation components that measure the online learning effectiveness in the viewed from socialization form of the online learning existence and the imbalances that occur in its implementation. Iqbal et al.'s research [45] showed the use of three models in evaluating the impact of educational program implementation. Limitations of Iqbal et al.'s research were that it had not shown evaluation components that measure the impact of the education programs implementation because of six domains. Those domains included: the program origin, input of resources to support the program running, the socialization process, the program implementation process, program results, and imbalances in program implementation. Finney's research [46] showed a modification of the CIPP model by adding a confirmative evaluation stage which is used to measure the effectiveness of the student learning process. The limitations of Finney's research were that it had not shown evaluation components that measure the imbalances that occur in the student learning process. Generally, from four studies, it is clear that all the results of those studies had not shown the evaluation components that measure the socialization form of the existence of a program and the imbalance that occurs in the implementation of the evaluated program.
The limitations shown from those four studies also strengthen the sensitivity of the components of the CSE-UCLA and Discrepancy models in supporting the accuracy of the evaluation results. This is because all evaluation components of the CSE-UCLA and Discrepancy models can measure the effectiveness of synchronous learning implementation accurately and completely in viewed from six domains. Those domains included: 1) context, 2) input, 3) socialization process, 4) implementation process, 5) results, and 6) imbalances in the learning implementation.
Besides, the limitations shown in those four studies can also affect the evaluation results of the learning process's effectiveness. This is because the evaluation components were used in previous studies had not been complete. Several components are not owned or not raised in the evaluation process, namely components to measure the socialization process domain and the imbalances domain. Those two evaluation components had been used in this research particularly, so certain that the evaluation results become more optimal.
Principally, this study had been able to answer the limitations of Mohmmed et al.'s research [23], Sudarwati and Rukminingsih's research [24], and Divayana et al.'s research [25] by showing the socialization process for the existence of synchronous learning and showing imbalances in the synchronous learning implementation. This research results had also shown that there was a combination of evaluation models that can be used to determine the effectiveness level of the learning implementation. This is reinforced by the results of previous research which also showed that there was a combination of several evaluation models in determining the effectiveness of a learning process implementation [47,48]. Besides some of the advantages shown from this research, there are also several limitations. The limitations of this research are that it has not shown yet aspects are priority improvements in the synchronous learning implementation.

4-Conclusion
Generally, the design which was a modified result of the CSE-UCLA model and the Discrepancy model had good categories and was effective to be used as an innovative evaluation model design. That design can be used to determine the effectiveness level of synchronous learning. That evaluation model design was formed has a good mix of functionality. This is shown from the evaluation aspect that is not available in the CSE-UCLA model to evaluate the imbalances domain, which is represented by the evaluation aspect of the Discrepancy model. Likewise, the evaluation aspect that is not present in the Discrepancy model to evaluate the socialization process domain has been represented by the evaluation aspects of the CSE-UCLA model. The modification between CSE-UCLA and Discrepancy models is not only used as an evaluation model design to determine the effectiveness level of synchronous learning implementation. However, this modification model design can also show the measurement process of the effectiveness level of all learning programs, education services, and education policies. This modification model design can show the evaluation process of programs, services, and policies in the educational field from the dimensions of initial conditions, inputs, promotion mechanisms, processes, outputs, and imbalances. Recommendations for educational evaluators regarding the existence of an evaluation model design which is the modification results of the two evaluation models are to encourage educational evaluators to use this evaluation model design in their evaluation practice toward the implementation of ICT-based learning, specifically in synchronous learning. Besides, it is also recommended can use this modification model as basic to evaluate learning programs, education services, and education policies in a wider scope. Future work that is recommended to be done in overcoming the limitations of this research is to develop an innovative evaluation model based on artificial intelligence that can determine priority aspects of improvement in the synchronous learning implementation.

5-1-Author Contributions
P.W.A.S. Conceptualization, methodology, data curation, formal analysis, investigation, writing-original draft, and editing; and D.G.H.D. Conceptualization, methodology, data curation, formal analysis, investigation, writing-original draft, writing-reviewing and editing. All authors have read and agreed to the published version of the manuscript.

5-2-Data Availability Statement
The data presented in this study are available in the article.

5-3-Funding
The authors received financial support based on a Research Grant with Contract Number: 461/UN48.16/LT/2021 from the Chair of the Institute for Research and Community Services, Universitas Pendidikan Ganesha for the research, authorship, and/or publication of this article.

5-4-Acknowledgements
The authors would like to express their deepest gratitude to the Chancellor and Chair of the Institute for Research and Community Services, Universitas Pendidikan Ganesha that had given the opportunity and provide the funding for the completion of this research.

5-5-Conflicts of Interest
The authors declare that there is no conflict of interests regarding the publication of this manuscript. In addition, the ethical issues, including plagiarism, informed consent, misconduct, data fabrication and/or falsification, double publication and/or submission, and redundancies have been completely observed by the authors.