Utilization of Alkin - WP -Based Digital Library Evaluation Software as Evaluation Tool of Digital Library Effectiveness

One source of learning in universities is a digital library. In the era of industry 4.0, most universities have implemented digital libraries in supporting the learning process. However, the reality shows that digital library management is still ineffective. Therefore, the implementation of digital libraries needs to be evaluated for determining the digital library effectiveness used as learning resources in supporting the learning process in universities. Many evaluation tools are used to evaluate the effectiveness of digital libraries but have not provided accurate recommendation results to support decision-making. This research presents an innovation in the form of an evaluation tool that can be used to evaluate the digital library effectiveness in universities. That evaluation tool is called the Alkin - WP -based digital library evaluation software. This software is a desktop platform that contains aspects of measuring the digital library effectiveness by referring to the components of the Alkin evaluation model and the WP ( Weighted Product ) method. This research aimed to show the effectiveness level of the utilization of Alkin - WP -based digital library evaluation software. This research method was R & D (Research & Development) which refers to the ten development stages of the Borg and Gall model. In this research, development was focused only on a few stages, included: usage trials, final product revision, dissemination, and implementation. The subjects involved in assessing the implementation/utilization of the Alkin-WP -based digital library evaluation software were 35 people, in the usage trials were six people, in product revision were three people, and at the stage of dissemination were 15 people. The tools used to collect data were questionnaires and interview guidelines. The data analysis technique used was descriptive quantitative. The effectiveness level of utilizing the Alkin-WP -based digital library evaluation software was 88.34%. It showed that the evaluation software had effective. The impact of this research results on the scientific field of educational evaluation is being able to show the existence of a new evaluation tool based on educational evaluation and artificial intelligence. That evaluation tool can easier for library heads to make policies for revamping digital library services based on accurate recommendations.

need fast services from a librarian through digital-based service facilities. This statement is following the opinion of Deja et al. [3] which stated that the management and maintenance of traditional library collections are expensive. The high cost of managing traditional libraries is caused by the need for access to complex knowledge sources.
Digital library services make it easy for library service users to register, search, and order library collection documents online in a short time without must go to the library building. This statement is following the opinion of Muthanna and Sang [4] which stated that digital libraries can be easily accessed anytime and anywhere by users without going to the library building physically. Besides, digital library services also make it easier for the librarian to enter new collections in digital format, make it easier to control transactions online for borrowing and returning collections owned by the library. In addition to the ease of access and obtaining digital reading resources, the presence of a digital library also has other impacts. Those impacts, included: the appearance of knowledge resource networks, the appearance of creation of digital format knowledge sources, and the culture of utilizing electronic reading resources. This statement is following the opinion of Sherriff et al. [5] which stated that many things can be realized after the existence of digital libraries in universities, included: publishing digital collections, digital literacy network communities, habits of using digital literature to support the learning process.
Generally, it is shown that the presence of digital libraries has been able to provide adequate literacy services. However, the facts show that digital library services are not run optimally. Evidence that shows the low perception of the presence of a digital library in a university can be seen from the poor management procedures. Some universities still think that the digital library implementation is only to complete the fulfillment of the accreditation component [6]. In addition, the digital library's existence is only used as a promotional tool for university facilities to attract prospective new students. Seeing that reality, it is necessary to take real action to improve digital library services in universities. This improvement can be done by providing appropriate recommendations to policy-makers (head of the library) so that it is easier for policy-makers to make accurate decisions to obtain a high level of effectiveness in digital library services.
The effectiveness level of digital library services is very important to be considered by the head of the library as a measure of success and responsibility in managing the library. This statement is following the opinion of Rosa [7] which revealed that the effectiveness of library services is the fundamental responsibility of the library management team, starting from the highest level of leadership to the librarian. Good and appropriate recommendations for policymakers can be obtained through evaluation activities using evaluation tools that are in accordance with the object being evaluated. One evaluation tool that is suitable to be used to evaluate the effectiveness of digital library services in universities is the Alkin-WP-based evaluation software. This tool is an innovation from the combination results between the educational evaluation model ("Alkin model") and the decision support system method ("WP/Weighted Product"). This tool can use to evaluate all components that determine the effectiveness of digital library services by showing the results of the effectiveness percentage. It's from the highest to the lowest percentage scores for each component through an accurate and fast calculation process.
Based on the findings of the evaluation tool innovation, this research question is how the effectiveness level of Alkin-WP-based evaluation software that applied in libraries? The main objective of this research was to determine the effectiveness level of the utilization of evaluation software based on modifications of the WP method and the Alkin evaluation model. Referring to the problems and main objective of research that had been explained, thus the researchers were interested in a deeper observation for determining the effectiveness level of the evaluation software that applied in the learning resource services in universities (i.e., digital library). The WP method is used to conducting the effectiveness scores calculation, while the aspect of the Alkin model is a basis for determining the measurement indicators of the effectiveness level from digital library services.
The background of this research was the previous research conducted by Divayana et al. [8] explained in detail the use of the CSE-UCLA model and the Weighted Product method as a basis for making evaluation applications for digital library services. The CSE-UCLA model in principle the same as the Alkin model which was used in this study because the CSE-UCLA (Center for the Study of Evaluation-University of California in Los Angeles) model is an evaluation model in the field of education that was initiated by Alkin. Niqresh's research [9] showed the functions, elements, and development stage of a digital library.  [11] showed the use of artificial intelligence in determining the effectiveness of digital library services. The constraint of Asemi et al.'s research was it had not shown the existence of the effectiveness score of digital library services starting from the lowest score to the highest score. Stiller and Petras's research [12] showed evaluation criteria to determine the quality or effectiveness of digital library services. The constraint of Stiller and Petras's research was it had not shown calculation process quantitatively in the quality or effectiveness determination of digital library services. Muglia [14] showed an evaluation activity on user experience in using digital libraries. The constraint of Barifah et al.'s research was it had not shown the calculation process to determine the effectiveness level of digital library services. Yu et al.'s research [15] showed the application of AI (Artificial Intelligence) concepts to digital libraries, so it becomes smarter and more optimal in serving visitor requests. The constraint of Yu et al.'s research was it had not shown the physical form of software that applies the AI concepts in the evaluation process of digital library effectiveness.

2-Material and Methods
This research used the R&D method by development design of Borg and Gall. It consists of ten development stages [16][17][18]. The development stage of Borg and Gall design completely can be seen in Figure 1. The research subjects involved in the usage trials stage were six people consist of two evaluators, two education experts, and two users. The research subjects involved in the final product revision stage were three people consist of one lead researcher and two members of the researcher. The research subjects involved in the dissemination stage were 15 people consist of six evaluators and nine users. Subjects involved in the implementation stage were 35 people consist of two education experts, 16 evaluators, and 17 users. The reasons for selecting the number of research subjects mentioned in the stage of usage trials, stage of final product revision, stage of dissemination and implementation were based on the purposive sampling method. The purposive sampling method was used to select research samples that refer to those who understand and involve directly in the use of the program being evaluated. The instrument forms used to retrieved data at the stage of usage trials were in the form of questionnaires and interview guidelines. The instrument forms used as the basis for revising the final product revision stage were the interview guidelines. The instrument forms used in data collection at the stage of dissemination and implementation were questionnaires.
The formula for the content validity test of the instruments used in usage trials and assessing the implementation of the Alkin-WP-based digital library evaluation software is the Gregory formula. The complete Gregory formula can be seen in Equation 1 [19][20][21].
Notes: A = cell indicating disagreement between the two raters; B and C = cells indicating the difference in views between the raters; D = cell indicating valid agreement between the two raters.
The score categorization of the instruments' content validity results refers to the Guilford classification of validity. That Guilford classification can be seen in Table 1 [22][23][24].

Research and Field
Data Collection  The data that had been collected were analyzed using descriptive statistics. The results data of filling out questionnaires that had been collected were analyzed using the calculation of the descriptive percentages in the form of scores. The formula for calculating descriptive percentages can be seen in Equation 2 [25][26][27].
Notes: f = number of scores; N = maximum number of scores.
The percentage of results needs converting to easier form in the interpretation and take of decisions about the evaluation software effectiveness. That conversion refers to the scale of effectiveness level that can be seen in Table 2 [28][29][30]. The data of interview results that had been collected were used to strengthen arguments toward quantitative results from the calculation of descriptive percentages. The data of interview results were also analyzed to become the basic in improvements at the final product revision stage.
The Alkin or CSE-UCLA model consists of five evaluation components, included: system assessment, program planning, program implementation, program improvement, and program certification [31][32][33]. Weighted Product (WP) method is one of the methods of artificial intelligence in the field of computer science that is used explicitly in making a decision. The formula of the WP method is divided into two formulas, included: 1) formula to determine normalization, and 2) Vector-V formula for determining to rank. The formula for determining normalization can be seen in Equation 3 [34][35][36].
Where, i = 1, 2,..., m, and ∑wj must have value = 1. wj can be positive or negative. Positive values indicate the profit attribute, and negative values indicate the cost attribute.
Notes: S: the value of the alternative preference; x: the value of criteria; w: the weight of criteria; i: alternative; j: criteria; n: number of criteria.
The ranking process in the Weighted Product method uses the Vector-V formula. The vector-V formula for determining to rank can be seen in Equation 4 [37][38][39].
Notes: V: the value of relative preference for ranking; x: the value of criteria; w: the weight of criteria; i: alternative; j: criteria; n: number of criteria; m: number of alternatives.

3-Results and Discussion
This research produces an overview of the usage trials stage, stage of final product revision, stage of dissemination and implementation of the Alkin-WP-based digital library evaluation software. At the usage trials stage was carried out test toward that evaluation software involved two education experts, two evaluators, two users (one head of the library, and one librarian). The instruments of software usage trials in the form of questionnaires consist of 10 question items. The results of the software usage trials can be seen in Table 3, while the software display that had tested can be seen in Figures 2 to 5. Ten items used in the usage trials were obtained from the results of content validity of 15 items questions. Those 15 items questions included: item-1 (the software is easily installed), item-2 (software display is attractive), item-3 (the layout of each form is consistent), item-4 (there is a feature that makes it easy for users to make an assessment on the evaluation aspects of the system assessment component refers to the Alkin model), item-5 (there is a feature that makes it easy for users to edit the evaluation aspects of the system assessment component refers to the Alkin model), item-6 (there is a feature that makes it easy for users to make an assessment on the evaluation aspects of the program planning component refers to the Alkin model), item-7 (there is a feature that makes it easy for users to edit the evaluation aspects of the program planning component refers to the Alkin model), item-8 (there is a feature that makes it easy for users to make an assessment on the evaluation aspects of the program implementation component refers to the Alkin model), item-9 (there is a feature that makes it easy for users to edit the evaluation aspects of the program implementation component refers to the Alkin model), item-10 (there is a feature that makes it easy for users to make an assessment on the evaluation aspects of the program improvement component refers to the Alkin model, item-11 (there is a feature that makes it easy for users to edit the evaluation aspects of the program improvement component refers to the Alkin model), item-12 (there is a feature that makes it easy for users to make an assessment on the evaluation aspects of the program certification component refers to the Alkin model), item-13 (there is a feature that makes it easy for users to edit the evaluation aspects of the program certification component refers to the Alkin model), item-14 (some features in the software make it easy to save data, edit, update and delete), item-15 (the software can accurately carry out the evaluation process and show the right recommendations). The results of that questionnaire's content validity can be shown as follows. Judges test results from two experts on usage trial questionnaires of the Alkin-WP-based digital library evaluation software, then included in the cross-tabulation (22). Details of that cross-tabulation can be seen in Table 5.
Based on the results in Table 5, the next step was the calculation process of questionnaires' content validity. Content validity calculation of the usage trial questionnaires refers to Equation 1. The complete calculation can be explained as follows: Content Validity = 10 5 + 0 + 0 + 10 = 10 15 = 0.667 If the results of the content validity above were matched with the instruments validity categorization that refers to Guilford's classification of validity, then the content validity result (rxy = 0.667) was included in the high category of validity. If viewed from the result of content validity that had shown in Table 5, there were 10 items used in the usage trials of the evaluation software, included: item-1, item-2, item-3, item-4, item-6, item-8, item-10, item-12, item-14, dan item-15.  Figure 2 shows the initial display of the Alkin-WP-based digital library evaluation software. The main menu display is used as a starting pointer to be able to access all forms in this evaluation software. Menus that are provided included: file, edit, process, profile, result, and help. The 'file' menu contains several sub-menus pointing to several forms, included: input data of the interest rating scores, input data of the aspects and evaluation components, input data of indicators and evaluation aspects. The 'edit' menu contains several sub-menus pointing to several forms that its data can be edited. The 'process' menu contains sub-menus pointing to several forms, included: the average value calculations of the evaluation aspects, and the normalization process. The 'profile' menu contains a sub-menu to the software maker's identity. The 'result' menu contains a sub-menu that pointing to the evaluation results form. The 'help' menu contains several sub-menus pointing to several forms, included: the form of database settings, and the form of user access settings. The average value of each evaluation aspect shown in Figure 3 above was obtained from the average calculation of each interest rating score given to each item of the evaluation instruments. The total numbers of respondents who assess each aspect of the evaluation instruments were 35 people. The assessment scores given by respondents to each aspect were in the scores range of 3, 4, and 5. The average value is needed later in the normalization process. The normalization results shown in Figure 4 above were obtained based on the multiplication results of each average in evaluation aspects that had powered with the decision-maker weight. The formula is used for the normalization calculation can be seen in Equation 3. The use of that formula can be shown as follows.  Based on the data in Tables 6, Table 7, and Equation 3, then the normalization calculation process can be completed. That calculation process can be shown as follows.  Table 6. The results of the vector-V affected the determination of the evaluation results shown in Figure 5. The vector-V was obtained from the calculation using Equation 4. The simulation of the vector-V calculation process using data from the normalization calculations that had obtained previously can be shown as follows. Based on the scores of the vector-V results were able to be determined the rank of evaluation components starting from the lowest value to the highest value. The priority component can be determined from the lowest vector-V value, so it makes it easier to provide accurate recommendations in a short time. The final product revision was conducted based on suggestions which were given by respondents, such as education experts, evaluators, and users (the head of library and librarian) when conducting the software usage trials through interviews. The several suggestions had given by respondents can be seen in Table 8. Based on some of the suggestions shown in Table 8 above, then it was able to be conducted adding the feature of "update data" on the "interest rating score" form, the "evaluation aspect" form, and the "evaluation instrument" form. The visualization from some adding of those features can be seen in Figures 6 to 8. The update feature shown in Figure 6 functions to revise the interest rating score. This feature can actively function if the user carries out right-clicks on the interest rating sections that want to be updated. The feature of 'update the interest rating' is found in the form of 'input data of the interest rating scores'. The contents of each field on this form can be edited if opened through the 'edit' menu located on the main menu display of Alkin-WP-based digital library evaluation software. In the 'edit' menu is found a sub-menu for editing data of the interest rating scores. The aspect update feature shown in Figure 7 functions to make revisions to evaluation aspects. This feature can actively function if the user carries out right-clicks on the evaluation aspects that want to be updated. The feature of 'update the data' is found in the form of 'input data of aspects and evaluation components'. The contents of each field on this form can be edited if opened through the 'edit' menu located on the main menu display of Alkin-WP-based digital library evaluation software. In the 'edit' menu is found a sub-menu for editing data of aspects and evaluation components.

Evaluation Components Code of Evaluation Aspects
Adding features to update the data of aspects or evaluation components.
Adding features to update the interest rating data.

Figure 8. The display of adding of update features on the evaluation instrument form.
The update feature shown in Figure 8 functions to revise the evaluation instrument items. This feature can actively function if the user carries out right-clicks on the instrument items that want to be updated. The feature of 'update the data' is found in the form of 'input data of indicators and evaluation aspects'. The contents of each field on this form can be edited if opened through the 'edit' menu located on the main menu display of Alkin-WP-based digital library evaluation software. In the 'edit' menu is found a sub-menu for editing data of indicators and evaluation aspects.
At the dissemination stage was carried out the socialization related to the introduction and procedures for using the Alkin-WP-based digital library evaluation software. The socialization was carried out through a workshop by inviting several prospective software users, included: six evaluators and nine users (three the head of the library and six librarians). The things which were discussed in the dissemination can be seen in Table 9.
Material topics which were discussed when the dissemination.

No
Material Topics 1 Introduction to the purpose and benefits of evaluation software presence 2 The management procedure of user data 3 The management procedure of the interest rating data 4 The management procedure of the evaluation aspect 5 The management procedure of evaluation instrument data 6 The management procedure of preference weights 7 The procedure for the average determination of each aspect 8 The procedure for carrying out the normalization process and evaluation results 9 The determination procedure of the evaluation results

10
The determination procedures of recommendations At the implementation stage was carried out the implementation of the evaluation software that involved several parties to conduct an assessment, included: two education experts, 15 evaluators, and 18 users (five head of library and 13 librarians). The instruments were used to assess the implementation process of evaluation software in the form of questionnaires consist of 15 question items. The assessment results toward the implementation process of Alkin-WPbased digital library evaluation software can be seen in Table 10.
Adding features to update the data of indicators and evaluation aspects. Fifteen items used in the assessment of the Alkin-WP-based digital library evaluation software implementation were obtained based on the results of the content validity of 18 question items. Those 18 question items included: item-1 (the software is easily installed), item-2 (software display is attractive), item-3 (the layout of each form is consistent), item-4 (there is a feature that makes it easy for users to make an assessment on the evaluation aspects of the system assessment component refers to the Alkin model), item-5 (there is a feature that makes it easy for users to make an assessment on the evaluation aspects of the program planning component refers to the Alkin model), item-6 (there is a feature that makes it easy for users to make an assessment on the evaluation aspects of the program implementation component refers to the Alkin model), item-7 (there is a feature that makes it easy for users to make an assessment on the evaluation aspects of the program improvement component refers to the Alkin model), item-8 (there is a feature that makes it easy for users to make an assessment on the evaluation aspects of the program certification component refers to the Alkin model), item-9 (ease of scores data input for each evaluation aspect), item-10 (accuracy of the determining results of the average score for each evaluation aspect), item-11 (ease of score data input the weights of decision-makers), item-12 (accuracy of the weights revision results), item-13 (accuracy of the determining process of normalization results), item-14 (accuracy of the ranking process), item-15 (some features make it easy to save, edit, update and delete the data), item-16 (the software accurately carries out the evaluation process and shows appropriate recommendations), item-17 (the accuracy of the decision-making process), item-18 (accuracy in providing recommendations). The results of that questionnaire's content validity can be explained as follows. Judges test results from two experts on the implementation assessment questionnaires of the Alkin-WP-based digital library evaluation software, then included in the cross-tabulation (22). Details of that cross-tabulation can be seen in Table 12.
Based on the results in Table 12, the next step was the calculation process of questionnaires' content validity. Content validity calculation of the implementation assessment questionnaires refers to Equation 1. The complete calculation can be explained as follows. If the results of the content validity above were matched with the instruments validity categorization that refers to Guilford's classification of validity, then the content validity result (rxy = 0.833) was included in the excellent category of validity. If viewed from the result of content validity that had shown in Table 12, there were 15 items used in the implementation assessment of the evaluation software, included: item-1, item-2, item-3, item-4, item-5, item-6, item-7, item-8, item-10, item-11, item-12, item-13, item-14, item-15, and items-16.
Based on the software usage trial results had shown in Table 3 previously, it appears that average the effectiveness level in the evaluation software was 88.67%. It indicates that the Alkin-WP-based digital library evaluation software was classified as a good category if seen from the scores' range of 80% to 89% showed in Table 2. Therefore there was no need to conduct the major revisions toward that the evaluation software. If viewed from the results shown in Figures 6 to 8, it can be seen that facilities for conducting updates toward interest rating scores, evaluation aspects data, and evaluation instrument items were made using the pop-up menu. Therefore it is easier to conduct updating data by right-clicking on the sections that will be updated. Based on the topics of the material presented in the dissemination stage that had shown in Table 9, it explains that the operating stages of the Alkin-WP-based digital library evaluation software had been socialized fully.
Based on the assessment results toward the evaluation software implementation had shown in Table 10, it appears that the effectiveness level of evaluation software was 88.34% so that it was also classified as a good category. Therefore there was no need to carry out the major revisions toward the Alkin-WP-based digital library evaluation software. If viewed from the average level of software effectiveness results had shown in Tables 3 and 10, it appears that the effectiveness level of software was classified as a good category. Therefore, the software has ready to be used on a wider scale.
Some of the key parameters needed to run this evaluation software are the Alkin evaluation component and the WP method. The evaluation components of the Alkin model consist of: system assessment, program planning, program implementation, program improvement, and program certification. The Alkin evaluation components are important in determining the digital library effectiveness because are used as standards for determining the success of digital library implementation. In addition, the Alkin evaluation components are also used as criteria that provide initial data in the normalization process for the WP method. The formulas in the WP method are used in the calculation process for determining the right recommendations in making decisions.
The sensitivity of Alkin evaluation components in supporting the accuracy of this software to provide evaluation results is very high. This is because all evaluation components of the Alkin model can measure the effectiveness of digital libraries completely and thoroughly in five domains. Those domains, included: 1) the domain of measuring legal regulations that foundation the existence of digital library, 2) the domain of measuring the readiness of resources that support the digital library implementation, 3) the domain of measuring the socialization form of the existence of digital library, 4) the domain of measuring how to operate the digital library, 5) domain of measuring satisfaction level of digital library services.
There are several previous research results that strengthen the position of this research and show the sensitivity of this research in contributing to the evaluation of educational services. Research conducted by Alokluk and Al-Amri [40] showed the use of a formative-summative evaluation model in evaluating digital library services. The limitation of Alokluk and Al-Amri's research was it had not shown an evaluation component that was able to evaluate the socialization procedure of digital libraries. Rahimi [43] showed the evaluation of digital library services. The limitation of Okeji and Mayowa-Adebara's research was it had not shown an evaluation related to the socialization of the digital library implementation, but tends to evaluate the readiness of human resources and supporting infrastructure for the digital library formation. Based on the limitations of those previous studies, it clearly shows the position and contribution of this research as a solution. This research had been able to present an evaluation component that can measure the socialization form of the existence of a digital library.
The limitations of some of those previous studies can also affect the results of evaluating the digital library effectiveness because the evaluation components used in previous studies were incomplete. There is one component that is not owned and not raised in the evaluation process, namely a component that measures the socialization form of the existence of a digital library. However, all evaluation components in this research had been used so that the evaluation results become more accurate. The novelty of this research is the discovery and implementation of software innovation that utilizing the combination of the WP method and Alkin evaluation model to be used as an evaluation tool of digital library effectiveness. This evaluation software can provide appropriate decisions and recommendations based on the effectiveness level of each evaluation component starting from the highest score to the lowest score.
The novelty from this research had become a solution for the constraint of Niqresh's research [9] by showing the components and aspects based on the Alkin evaluation model that can be used in evaluating digital library effectiveness. The research obstacles of Xie et al. in 2021 and Asemi et al. in 2021 [10,11] had been answered through this research by utilization of the WP method. WP calculation is used to determine the effectiveness level from each evaluation component starting from the highest to the lowest in effectiveness level. The constraint of Stiller and Petras study [12] had been answered through this research by showing calculation process quantitatively used the WP method to determine digital library effectiveness. The constraint of Muglia et al.'s research [13] had also been answered by showing the evaluation indicators of the Alkin model as a measuring tool for the effectiveness of digital library services at computer colleges in Bali. Besides, it has novelty and excellence, this research also still has an obstacle. Its display is still based on desktop, so it is difficult to access online by decision-makers or evaluators in conducting the evaluation. The evaluators or decision-makers must prepare the right time and space to conduct the digital library services evaluation process using a stand-alone computer and has completed by evaluation software installed.

4-Conclusion
Generally, this research had been able to show the effectiveness level of implementation/utilization of the Alkin-WP-based digital library evaluation software. It is evidenced by the implementation of evaluation software to several computer colleges in Bali Province that had been able to run well. This is reinforced by the result of effectiveness percentage for the software usage trial was 88.67% which included the good category. Besides, the result of percentage effectiveness for assessment toward the evaluation software implementation was 88.34% which included in the good category. Therefore, this evaluation software can be used effectively as a tool for evaluating the digital library effectiveness. The evaluation results from the software produce accurate decisions and recommendations. The accuracy is viewed from the lowest to highest effectiveness level that is obtained from the combination of the WP method calculation and the Alkin evaluation model. This evaluation software is not limited to evaluating the digital library effectiveness but can also be used to evaluate the quality of conventional library services. This evaluation software has evaluation components that are suitable for evaluating educational service programs (example: digital library). Those components including system assessment, program planning, program implementation, program improvement, and program certification. System assessment is used to evaluate the rules that foundation of a digital library formation. Program planning is used to evaluate the readiness of human resources and equipment resources that support the digital library implementation. Program implementation is used to evaluate procedures for socializing the existence of a digital library. Program improvement is used to evaluate the digital library performance procedures. Program certification is used to evaluate the satisfaction level of digital library users. Future work that can be done to solve the constraints of this research is developing this evaluation software in the form of web or mobile technology so that it is more easily accessible by evaluators and stakeholders.

5-2-Data Availability Statement
The data presented in this study are available in article.

5-3-Funding
The authors received financial support from the Directorate General of Strengthening Research and Development of the Ministry of Research, Technology and Higher Education of the Republic of Indonesia for the research, authorship, and/or publication of this article.

5-4-Acknowledgements
The authors also express their gratitude to the Rector and Chair of the Research and Community Service Institute, Universitas Pendidikan Ganesha, who give the opportunity and permission to the authors for carrying out this research.

5-5-Conflicts of Interest
The authors declare that there is no conflict of interests regarding the publication of this manuscript. In addition, the ethical issues, including plagiarism, informed consent, misconduct, data fabrication and/or falsification, double publication and/or submission, and redundancies have been completely observed by the authors.