Psychometric Properties of the Teleprimary Care Oral Health Clinical Information System (TPC-OHCIS) Questionnaire Using the Rasch Model

Background The Tele Primary Care Oral Health Clinical Information System (TPC-OHCIS) was implemented in Malaysia to digitalize health care and reduce numerous ground-level manual tasks. This study measures the psychometric properties of the TPC-OHCIS questionnaire among healthcare workers (HCWs) at primary healthcare clinics (PHC). Method A pilot study was conducted at PHC, which implemented the TPC-OHCIS application for service delivery. The questionnaire contained 65 items with four response categories, grouped into four scales: technology, organization, external support, and human resource. The questionnaire items were analyzed using the Rasch model in Winsteps 3.72.3. Results There were 319 respondents who participated (98.8%). The Cronbach alpha was 0.93. The construct validity was determined by a positive point measure correlation (PMC) value, with an infit and outfit mean square (MNSQ) range of 0.4-1.5 and a Z-standardized (ZSTD) range of -2.0 to 2.0. The person and item reliability were 0.93 and 0.97, respectively, indicating excellent reliability. The questionnaire was unidimensional, where the raw variance explained by measures was >40%. Conclusion The questionnaire was deemed fit for an actual survey after 18 items had been deleted. It has good psychometric properties and is practically applicable for evaluating HCWs on the TPC-OHCIS application implementation process monitoring using the local Malay language. High reliability and unidimensionality were achieved, supporting its use in digital healthcare. With this validated questionnaire, it will enhance digital healthcare implementation and streamline manual tasks.


Introduction
Telehealth is defined as revolutionizing the delivery of healthcare services by using information and communication technologies to exchange valid information for the diagnosis, treatment, and prevention of diseases [1,2].The telehealth application allows healthcare workers (HCWs) to share data between facilities regardless of distance.The telehealth data-sharing platform offers various features, including electronic medical records (EMR), online appointments, electronic prescriptions, public health surveillance, vaccination, environmental health, institutional health management, and online medical education among HCWs [1,2].Globally, telehealth systems have improved resource coordination from different localities, facilitating access to essential healthcare services in the community [3,4].In 1991, Canada became the first country to use telehealth, subsequently adopted by the United Kingdom, Hong Kong, Taiwan, Korea, Jordan, Saudi Arabia, Singapore, and 70 other nations utilizing a District Health Information System (DHIS) [5,6].Telehealth has been adapted and renamed in various countries while retaining the same function of integrating information technology into healthcare.For instance, it is known as Opth Web in Singapore, HAKEEM in Jordan, and Tele Primary Care Oral Health Clinical Information System (TPC-OHCIS) in Malaysia, among others [5].In developed countries, telehealth applications are more commonly used in conventional healthcare, whereas in developing countries, they serve as an alternative to conventional healthcare.However, the requirements for these applications are more demanding in developing countries, as they need to provide broad populations with basic healthcare services and bridge the distance between rural areas and specialized hospitals typically located in large cities [7,8].In Malaysia, the TPC-OHCIS application is a form of telehealth initiated to enhance the continuity of care through a specific digital system designed for primary care service delivery, covering a 'womb to tomb' approach [9,10].The TPC-OHCIS application was pilot-tested at the Seremban Health Clinic in Negeri Sembilan and was designed to support service delivery in the outpatient department, maternal and child health unit (MCH), dental healthcare outpatient, pharmacy, medical laboratory, radiography, and disease surveillance monitoring.Later, the application was expanded to 122 other clinics in Malaysia [10].TPC-OHCIS is a web-based application that allows HCWs to record data during service provision at the clinic or during home visits, as it can function offline [10].The system automatically updates record data upon internet availability.The TPC-OHCIS application is free of physical boundaries, allowing patients to walk into any governmental healthcare facility equipped with this system and receive treatment as their data are accessible at any of these locations [9].
Since the COVID-19 outbreak in February 2020, Malaysia has experienced various health service delivery disruptions at primary care levels [11].Consequently, there have been significant interruptions in the MCH services and school health programs due to school closures following the imposed movement restriction order to curb COVID-19 infection [5].Fortunately, the availability of digital technologies such as TPC-OHCIS has enhanced healthcare delivery despite the movement restrictions [9].To date, there has been no evaluation of the TPC-OHCIS application's usage on improving the efficiency of health indicator monitoring.Therefore, this study aims to assess the validity and reliability of the TPC-OHCIS questionnaire for the implementation of the application in primary healthcare settings during the COVID-19 pandemic.
The findings of this study will be used to provide insights for better TPC-OHCIS implementation and utilization in healthcare delivery in Malaysia.Assessing the ability of the healthcare delivery system to adapt and shift towards providing services through telehealth applications is important to avoid catastrophic consequences.

Study design and sample
A cross-sectional study was conducted at primary healthcare clinics (PHCs) equipped with the TPC-OHCIS application in three selected states: Federal Territory of Kuala Lumpur and Putrajaya, Selangor, and Pahang.The TPC-OHCIS questionnaire was developed based on various combinations of the technologyorganization-environment theory [12], human organization technology-fit theory (HOT-fit) [13], diffusion of innovation (DOI) theory [14], and technology acceptance model theory [15].A total of 319 respondents were recruited through convenience sampling, meeting the inclusion criteria of having used the TPC-OHCIS application in handling MCH services at the PHC for at least one month.The targeted respondents were MCH staff directly involved in data input, management, and interpretation using the TPC-OHCIS application.For validation of the TPC-OHCIS questionnaire, a convenience sampling technique was used in three of the fourteen states of Malaysia.Later, the validated questionnaire will be used to measure national TPC-OHCIS implementation across all states.The validation process was conducted during the COVID-19 pandemic, and only available HCWs who met the inclusion criteria were invited to participate in the study.

Instrument
The questionnaire was developed using a standard tool for managing health information for decisionmaking, employing the Technology-Organization-Environment (TOE) framework [8], and adapted to local service needs in compliance with the four aforementioned theories [12][13][14][15].Ten experts, including the primary health program manager at the TPC-OHCIS Unit of the Ministry of Health Malaysia (MOH), researchers, and the Chief of Information Technology, were requested to review the 65-item TPC-OHCIS questionnaire.The relevance of each questionnaire item was rated on a scale of one to three (1 = not clear, 2 = revision needed, 3 = very clear) for clarity, and the essentiality of the questionnaire was determined by a similar scale (1 = not essential, 2 = useful but not essential, 3 = essential).The questionnaire was made bilingual in English and Bahasa Malaysia after a forward-backward translation conducted by two linguistics experts.Three experts in related fields conducted a content validity evaluation.The expert panel scores were analyzed with an acceptable cut-off content validity index score of 1 [16].Once reviewed, the forwardbackward translation was repeated.Subsequently, a pilot study was conducted with respondents including medical doctors and nurses at the PHC equipped with the TPC-OHCIS application and responsible for the MCH units.Reliability analysis was performed using the Rasch model in Winsteps 3.72.3for Cronbach's alpha and Cohen's kappa testing [17,18].

Data analysis
Each questionnaire item was analyzed using the Rasch model in Winsteps 3.72.3.The instrument's reliability and validity were evaluated in a 5-step process using the program.Initially, the person and item polarities were examined.Problematic objects and individuals were identified using fit statistics and were either eliminated or altered.Next, the person and item separation (s) indexes and other reliability metrics were assessed.Subsequently, the unidimensionality and local independence of the inventory were calculated.The item difficulty level was examined using the item-person map (Wright map).

Person and Item Polarity
All items and dimensions measured were required to have positive point measure correlation (PMC) values [17].The PMC should be >0.30,which demonstrates that the items can measure the proposed dimensions effectively [17,18].An item that demonstrates high polarity can differentiate among respondents [19].Items and people with negative PMC values were deleted as they were considered outliers (person) and indicated that a dimension was not being measured (item) [20].

Person and Item Fit
The ability of the items to measure the latent trait or dimension was determined by analyzing the item fit [17][18][19][20][21].The person fit was examined to identify individuals who may have contributed an abnormal reaction compared to the norm [17].The range 0.4-1.5 for the mean square (MNSQ) outfit and infit was deemed appropriate for the person and item fit of polytomous data (Likert scale).The Z-standardized (ZSTD) value was between -2 and +2.

Person and Item Reliability
As the inventory had a high Cronbach alpha (Kuder-Richardson 20 formula KR-20) score, all items within the same dimension demonstrated outstanding internal consistency [17,19].According to the Rasch model, a value ≥ 0.80 indicates high reliability for both the item and the person [17].The Rasch model has high personal reliability if the items can distinguish among respondents for the variables or constructs being measured [17,18,22].

Person and Item Separation Indexes
The person separation index is defined as the ability of the questions to distinguish and categorize respondents' qualities or characteristics from the measuring variables [20].The item separation index is used to confirm item hierarchy, distributing items based on item difficulty [23].Person and item separation indexes with higher values indicate excellent separation [24].Values in the 1.5-2.0range are considered appropriate, whereas good person and item separation requires a value > 3.0.

Unidimensional and Local Independence
Unidimensionality and local independence ensure items contribute to a single construct [17,24].Item correlation is defined when all items relate to the same variable [17,18].A minimum of 40% of the raw variance is required for an accurate measurement of unidimensionality [17,19].Unexplained variance in the first contrast should not exceed 15% [17,19].The eigenvalue of the first contrast should fall within the 1.4-3.0range [17].Local dependence is identified by correlations between items [19].Standard residual correlation is conducted between items to determine dependency, where a value > 0.70 indicates that the items should be deleted as they share the same traits and are highly dependent on each other [17,18,20].

Item-Person Map (Wright Map)
Wright maps demonstrate interval-level measurement by logit scale to identify the hierarchy of a respondent's ability based on item difficulty [17].The difficulty-based item distribution enables the evaluation of how well the questions (items) are formulated [17].The best Wright maps fit when both item and person scores are zero (0) values at the same level, which demonstrates that the items are well answered by the respondents with an almost identical understanding of the items [17].Unfortunately, a Wright map is seldom perfect.The gap between the items, respondents' measure, respondents' fit, item fit, and item measure encourages further examination of whether the model fits the formulated theory [25].

Research ethic
The Research Ethical Committee of the National University of Malaysia and the National Medical Research Registry granted ethical approval for the conduct of this study.We adhered to the ethical principles to protect the dignity, rights, and welfare of the respondents.Research information sheets on the research description, purpose, benefit, and personal data confidentiality were provided before the respondent consented to sign the consent form to participate in the study.

Results
A total of 319 respondents were analyzed in this pilot study, which was conducted at PHC with the TPC-OHCIS application in three states of Malaysia: Federal Territory of Kuala Lumpur-Putrajaya, Selangor, and Pahang.Demographic data (age, gender, race) and job-related information (designation, length of service, and role in the MCH unit as a TPC-OHCIS user, vendor, or information officer) were collected for the respondents.Four respondents were identified as upper outliers and six as lower outliers; these were removed from the data set.Items DPTC40, CVEN39, ASEC14, and CVEN38, which appeared at the bottom of the Wright map and were considered outliers (more than 2 standard deviations), were also deleted from the questionnaire.Ultimately, only 309 respondents and 61 items were used for further analysis.

Item fits
Table 1 describes the overall item analysis by presenting the summary statistics for the initial dimension.There were four scales (dimensions): Technology (ARA 1-5, ACOMPATIBLE6-9, ACOMPLEX10- The PMC should yield a positive value, as a negative value indicates that the questionnaire did not measure the intended parameter [13,16].Items ACOMPATIBLE8, ACOMPATIBLE9, ACOMPLEX10, ASEC14, ASEC15, BTP31, DEISK46, DEISK47, DEISK48, DCITEX51, DCIO56, and DCIO57, which yielded negative PMC values, were therefore deleted.The same items also had MNSQ values greater than 1.5, where the normal range for the MNSQ outfit is between 0.4 and 1.5 [17].

Reliability and separation indexes
The item and person reliability of the questionnaire were 0.97 and 0.90, respectively, which exceeded 0.90 and indicated high reliability and extreme acceptability [17,18].The high person reliability demonstrated that the respondents in this pilot study exhibited varying abilities to respond to the questionnaire, which also relates to the findings from the Wright map [26].The Cronbach's alpha (KR20) of 0.93 indicated good internal consistency and qualified it for subsequent actual data collection [17].The high item reliability confirmed that the items could be accurately located according to the latent variable.In this study, the person separation index of 3.05 demonstrated that the 61 items could separate the respondents into three categories (agree, neither agree nor disagree, or disagree) regarding whether the TPC-OHCIS application made their jobs easier during the COVID-19 pandemic.A person separation index value greater than 3.0 is considered excellent separation [17].

Unidimensional and local independence
Table 3 depicts the unidimensional analysis results based on the standardized residual variance.The table demonstrated that the raw variance explained by the measures was 40.5% (minimum requirement: 40%) and the model raw variance explained by the measures was 41%.This indicated that the construct validation was almost equivalent to the Rasch-predicted model.The Rasch model requires a minimum value of 40% for raw variance explained by measures and 60% for excellent unidimensionality [26].Unexplained variance in the first contrast value should not exceed 15% for ideal measurement [22].In this study, both the raw variances explained by measures (40.5%) and unexplained variance in the first contrast (8.7%) were achieved.In Rasch analysis, local dependence is defined to identify any correlations between the items [17].Table 4 shows the standardized residual correlations used to identify dependent items.Items DEISK47, ARA1, ACOMPATIBLE9, and DPEU61 were deleted due to factor loadings greater than 0.7 (Figure 1).A higher standard residual correlation value greater than 0.70 indicated that these items were dependent and shared the same traits [14].Items ARA2 and ARA3 were rephrased.Aside from the deleted items, all other items were considered valid for use in an actual study.

Format analysis
Format analysis defines the scoring adequacy.In this pilot study, a 1-4 Likert scale was used.Table 5 demonstrates that the separation complied with the range of 1.4 < s < 5, which was the study's aim [17,26].Table 6 demonstrates that the respondents were able to differentiate and apply the scale as intended.The observed count in each category was significant (i.e., >10), showing a monotonic increase in category measures without any disorder.An increase in the distance between the threshold across categories aligns with Linacre's liberal criteria [18].However, the distances between the Andrich thresholds of adjacent categories were small (d < 1.4), presenting a problematic issue.The criteria are explained as follows: "a category represents too narrow a segment of the latent variable or corresponds to a concept that is poorly defined in the minds of the respondents."The category probability curves illustrated smooth rolling hills (Figure 1).

Person-item map
The Wright map illustrates the complexity of the items based on the logit scale [17].In this study, the means of the item and person were similar (Figure 2), indicating that 50% of the respondents agreed with the questions, and 50% did not.Up to 68% of the respondents answering the questions were within 1 SD, enabling them to discern the true meaning of the items.

Pooled confirmatory factor analysis (CFA)
The second-level constructs were tested separately to match the model desired before testing the entire model (pooled CFA).A combined CFA is also necessary to assess the correlation between all constructs measured in the discriminant validity procedure, where high correlation values above 0.85 are considered to have a shadow over each other.After all requirements for validity and reliability tests were met, structural equation modeling (SEM) was performed, including all items for each construct for the analysis of the standard regression coefficient (standardized regression weight) and the normal regression coefficient (regression weight).Figure 4 displays the SEM analysis of the study.The standardized regression weight values were as follows: technological construct at 0.20, organizational construct at 0.87, environmental construct at 0.46, and human construct at 0.23.An increase of one standard deviation in the technological, organizational, environmental, and human constructs will increase compliance by 0.20, 0.87, 0.46, and 0.23 standard deviations, respectively.
A 1-4-point Likert scale was sufficient to evaluate the theoretical framework's intention and analyze the questionnaire hypothesis.This was demonstrated by the scale falling within the 1.4 to 5 range [17,26].The average (in ascending order from negative to positive) also demonstrated that respondents were able to assign value based on the respective items [26].Figure 2 depicts the clear definition of the scale by the respondents and the absence of continuous numbers in the bell shape [26].
Unidimensionality was demonstrated as all questionnaire items were aligned in the same direction and queried the same value.Furthermore, it also indicated that the questionnaire did not contain more than one dimension, which was evidenced by the 40.5% of the raw variance explained by measures.
The questionnaire demonstrated very good reliability regardless of the presence of internal consistency (Cronbach's alpha KR20 = 0.93), where the item and person reliability were 0.97 and 0.90, respectively.These values exceeded 0.90, indicating high reliability and substantial acceptability [13,14].The high person reliability showed that the respondents in this pilot study exhibited varying abilities to respond to the questionnaire, which also related to the Wright map findings [22].The Cronbach's alpha (KR20) of 0.93 indicated good internal consistency and qualification for subsequent actual data collection, and the high item reliability proved that the items could be located accurately according to the latent variable [17].
In this study, up to 50% of the respondents agreed that the TPC-OHCIS could ease their daily jobs during the COVID-19 pandemic, while the other half disagreed.Rasch analysis using differential item functioning revealed that nurses were among the disagreeing respondents, as shown in Figure 3. Nurses are the HCWs who use the TPC-OHCIS system most frequently in Malaysian primary healthcare services.Yet, their perception of the system has never been evaluated.Using a validated TPC-OHCIS Questionnaire aids in evaluating the implementation of the TPC-OHCIS application for improvement and national rollout.The appropriate measuring tool used among the implementers of the system will facilitate planning for quality improvement efforts in service delivery.

Limitations of the study
The present study has its limitations, including the use of a convenience sampling method amid the COVID-19 pandemic.This study was conducted in 2021 when movement control orders were implemented in Malaysia.The study was conducted entirely online, and all surveys were distributed to the respondents' superiors without a proper briefing on the overall idea of the research, except for the study information stated at the beginning of the survey Google Form before the respondent consented to participate.There is a possibility of respondent bias, as no supervision or explanation was provided to guide answering the questionnaire, which was self-administered.Therefore, each respondent answered as they perceived fit without understanding the true intent of the survey fully.In addition, the prolonged period of the COVID-19 pandemic may have caused fatigue among health personnel, likely leading them to answer the questions carelessly and without focus.Respondents may have responded to the statements by either agreeing or disagreeing just to complete the form (acquiescence bias).

Recommendation
Future researchers can use more effective methods to avoid sampling bias, such as simple random sampling, which provides equal odds for every member of the population to be chosen as a respondent in the study.We aim to use this validated questionnaire for a national study to cover all states that have installed and implemented the TPC-OHCIS in Malaysia; therefore, verification and improvement of the questionnaire will be undertaken once the actual study is completed.

Conclusions
Achievement in incorporating four theories in the development of a 65-item questionnaire covering four important scales demonstrated the robustness of the TPC-OHCIS in measuring the primary objective.The validation and reliability assessment, customized to the application and services used, contribute to accuracy in measurement.This tool can assist the Ministry of Health, which owns the TPC-OHCIS application, in evaluating the implementation of its application and in supporting their healthcare providers' services during its implementation.Focusing on the MCH unit, which is a pillar of primary healthcare services, has allowed for a broad scope of implementation stage coverage.Collecting feedback from targeted respondents who used the system enabled identification of the desired characteristics that needed evaluation.Finally, the newly designed TPC-OHCIS questionnaire can be applicable for evaluating other system applications at the primary health clinic level or hospital level, especially when the implementation process requires a collective effort from multiple disciplines or job categories.3

8.
Data received from other devices (tablet /laptop/smartphone) outside health facilities in the TPC-OHCIS application can be easily merged into the database for analysis.Data yang dimasukkan melalui telefon/laptop ke dalam applikasi TPC-OHCIS boleh disatukan dalam database untuk analisa 9.
Information data is shared seamlessly across our organization regardless of the location.Data informasi dikongsi tanpa batasan akses dalam organisasi dimana jua.

11.
I need a long time to understand and familiar with the TPC-OHCIS application.Saya ambil masa yang lama untuk pandai menggunakan aplikasi TPC-OHCIS.
12. I do not find enough time to study and upgrade my technology skills before using the TPC-OHCIS application.
I feel secure in using the TPC-OHCIS application, keying patients' data, and sharing it across my organization.Saya yakin keselamatan data pesakit yang dimasukkan di aplikasi TPC-OHCIS dan dikongsi dalam organisasi saya.

15.
I would feel safe using the TPC-OHCIS application to retrieve patient data.Saya rasa data pesakit selamat untuk di muat turun dari aplikasi TPC-OHCIS.

17.
I am concerned about how much I can trust the vendor.Saya berhati hati dalam memberi sepenuh kepercayaan kepada vendor.
Organization scale BCHAMP Presence of specified liaison officer Pegawai penyelaras.

18.
A specified liaison officer will inform top managers and vendors about the faulty TPC-OHCIS application.

20.
A specified liaison officer has a good relationship with both vendors and top managers.Pegawai penyelaras mempunyai hubungan yang baik dengan pembekal dan pengurusan tertinggi.

22.
A specified liaison officer provides training/courses for the users a few times a year.Pegawai penyelaras menyediakan latihan /kursus berkaitan sistem beberapa kali setahun.

24.
We have a reliable computer network in our use.Kami mempunyai jaringan rangkaian komputer yang stabil.

27.
Top Management always supports and encourages using the TPC-OHCIS application for job-related tasks.

28.
Top management provides the necessary help and resources to enable staff to use the TPC-OHCIS application.Pegawai atasan menyediakan keperluan dan sumber bagi pekerjanya menggunakan aplikasi TPC-OHCIS.

30.
Top management gives feedback to vendors on every dismay or unsatisfactory comment from staff Pegawai atasan memberikan maklumbalas dari stafnya berkaitan masalah sistem kepada pembekal.

33.
I find it difficult to use the TPC-OHCIS application because it cannot be upgraded due to insufficient budget.
The vendor entertains each one of our complaints dutifully.Pembekal menangani setiap aduan dengan bertanggungjawab.

39.
I have a platform to voice problems regarding the TPC-OHCIS application directly to the vendors.Saya mempunyai platform untuk mengutarakan keluhan berkaitan aplikasi TPC-OHCIS terus kepada pembekal.I like to work using the online system Saya suka kerja mengunakan sistem atas talian.

FIGURE 2 :
FIGURE 2: Wright item-person map for the TPC-OHCIS subscale.TPC-OHCIS: Tele Primary Care Oral Health Clinical Information System.

Figure 3
Figure3shows the person differential item functioning (DIF), which indicates different probabilities of endorsing a given item on a multi-item scale after controlling for overall scale scores.

FIGURE 4 :
FIGURE 4: Pooled confirmatory factor analysis of the measurement model.

TABLE 1 : Descriptive analysis of all items of the TPC-OHCIS questionnaire.
I do not find enough time to study and upgrade my technology skills before using the TPC-OHCIS application.(ACOMPLEX12) I often find it too complex to understand and use this TPC-OHCIS application (ACOMPLEX13) I feel secure in using the TPC-OHCIS application, keying patients' data, and sharing it across my organization (ASEC14) TPC-OHCIS: Tele primary care oral health clinical information system; MEASURE: Item Difficulty Estimate; MNSQ: Mean Square; ZSTD: Standardized Fit.

Table 2
shows that the final round of analysis reflected that the results were satisfactory at both the person and item levels.Several indicators illustrate this improvement, as follows: (1) good person and item separation (3.53 and 6.14, respectively), and (2) acceptable person and item reliability (0.93 and 0.97, respectively).

TABLE 6 : Category frequency and threshold values for the TPC-OHCIS dimensions.
TPC-OHCIS: Tele Primary Care Oral Health Clinical Information System.