Health professionals’ expectations of e-consultations – a qualitative study
It will be more difficult to observe patients and perform clinical assessments. Nor do all patients have sufficiently good digital skills or adequate health literacy.
Background: Digital services have the potential to replace and streamline areas of the health service, reduce the health challenges society faces, and give patients a greater opportunity to be proactive in their choice of treatment. A growing number of patients are offered access to digital patient portals and e-consultations in addition to, or instead of, traditional hospital consultations. The features of digital portals vary, but for the most part, patients can access their own health information and communicate asynchronously with care providers.
Objective: The objective of the study is to explore health professionals’ expectations of implementing and using asynchronous, text-based e-consultations in a digital patient portal for outpatient follow-up of patients with a chronic illness.
Method: The study has an exploratory qualitative design, where we carried out nine semi-structured, individual interviews with health professionals who planned to use e-consultations in a pilot project. The sample consisted of registered nurses (n = 4) and doctors (n = 5). A thematic content analysis was performed on the material.
Results: The main themes that emerged in the analysis were as follows: Selection of patients, Professional accountability and a satisfactory standard of health care and Reorganisation of practice. The informants felt that it was necessary to carefully select the patients who were to be offered follow-up via e-consultations. These patients must meet certain criteria, particularly in terms of adequate command of Norwegian, digital skills and a high degree of health literacy and ability to understand information. The informants were concerned about whether asynchronous text-based e-consultations met expectations for professional accountability and a satisfactory standard of health care, since the opportunity to observe patients and perform assessments was reduced. Even though e-consultations may improve work efficiency, the informants believed that their introduction would change procedures and require a reorganisation of practice.
Conclusion: Health professionals are concerned about whether asynchronous e-consultations ensure professional accountability and a satisfactory standard of health care and believe they may change practice and the organisation of the health service. Consequently, listening to health professionals when digital services are implemented, used and developed, should be a key priority in order to maintain professional accountability and a satisfactory standard of care
The health and care services are gradually becoming more digitalised (1). Digital services can potentially take over and improve efficiency in areas of the health service, reduce health challenges in society, streamline patient pathways and give patients a greater opportunity to be proactive in their choice of treatment. Meanwhile it is vital to ensure that patients have access to simple and secure digital services in order to strengthen user participation (2).
This study examines health professionals’ expectations of using asynchronous, text-based e-consultations in a digital patient portal. Patients with a serious chronic illness wanted to have the option of more flexible outpatient follow-up, with some check-ups moved to a digital patient portal instead of having a traditional consultation.
Digital patient portal
Digital patient portals allow patients to check their own health information, ask for renewal of prescriptions, and have text-based electronic consultations (e-consultations) with health professionals. E-consultations entail secure asynchronous communication with care providers (3).
Despite comprehensive research on digital services in health care, few studies describe the importance of the services because the methods used to evaluate and examine their impact are often unconnected and fragmented (4). However, studies show that using services in a digital patient portal can improve the patient’s sense of mastery and control of their own health, if the portal is adapted to the individual patient group (5, 6).
The use of e-consultations can also improve patient safety and clinical care as well as strengthen the decision-making ability of patients with a chronic illness, where self-monitoring and self-discipline are of prime importance (7).
E-consultations have been adopted in several areas of the primary and specialist health services in Norway, and there are clear indications that this will increase going forward (8). As a result of the COVID-19 pandemic alone, the number of e-consultations in Norway increased by 300 per cent during the first week of lockdown from 21 to 27 March 2020, compared to the week before lockdown (9).
Patient preconditions
Boosting the patient’s self-efficacy is regarded as one of the key goals of digital health services (10), and results in patients being better equipped to take control of their own health (11). Patients who use patient portals actively report greater user participation and control, as well as more involvement in their own health (12).
However, this requires that the portal’s user-friendliness and functionality correspond to the patient’s comprehension of their personal health information (12). Health literacy refers to one’s ability to understand health information and can be defined as ‘a person’s ability to understand, assess and apply health information so as to be able to make evidence-based decisions on their own health’ (1, p. 5).
Chronically ill younger patients with digital proficiency and interest or capacity to comprehend health information, use digital patient portals more often than other patient groups (13). Patients who use a digital patient portal also have more confidence in their care providers (3).
Studies describe factors such as self-efficacy, ability to understand health information, ethnicity, education and knowledge of their own illness as affecting patients’ use and utilisation of a digital patient portal (7, 14).
Patients who seldom use digital portals and their services are more unsure about whether sensitive personal data are treated confidentially, may have had negative experiences previously or be unaware that the option is available to them (15, 16).
Health professionals’ expectations and experiences
Several factors impact on whether e-consultations via a digital patient portal are introduced and used. Allocating enough time for thorough training and information – in addition to health professionals’ skills – is a vital prerequisite (7, 10).
Health professionals’ digital expertise and motivation, their attitudes towards innovation and change, and their engagement are believed to influence the use of digital portals. Such factors also play a major role in whether patients will make use of them (16). Resource shortages, poor accessibility and a lack of understanding of the benefits are perceived as barriers (17).
According to health professionals, patients’ ability to understand health information is crucial for their use of digital services (16). At the same time, they assume that some patients will not use e-consultations, because they are not capable of doing so or are not interested (10, 18).
Research also shows that health professionals fear that their professional autonomy may be threatened when the patient is assigned a more active role via e-consultations (10). At the same time, a Spanish study showed a dramatic increase in the number of e-consultations during the COVID-19 pandemic due to social distancing measures, when previously perceived barriers to implementation apparently disappeared or were reduced (19).
The objective of the study
The objective of the study was to explore health professionals’ expectations to the introduction and use of asynchronous e-consultations in a digital patient portal, as part of their outpatient follow-up of patients with a serious chronic illness.
Method
The study has an explorative qualitative design using semi-structured, individual interviews as a method of data collection.
Sample and procedure
The informants were recruited at an outpatient clinic in a Norwegian hospital. The head of unit received information about the study and circulated the request to participate. Nine out of eleven possible informants agreed to take part. Both registered nurses (RNs) (n = 4) and doctors (MDs) (n = 5) participated – seven women and two men.
The outpatient clinic was selected as a pilot site for the use of e-consultations via a digital patient portal. They were to monitor members of a patient cohort who had actively requested an e-consultation alternative. This was to consist of asynchronous written communication between patients and care providers.
Prior to an e-consultation, the patient was to fill in a digital form based on the health professionals’ current routines for face-to-face consultations with patients with the illness in question. Via the portal, the health professional would be notified when their patient had submitted the form, and then they could respond to the shared information, ask questions, renew prescriptions and suggest future treatment.
Data collection
Data were collected using a semi-structured interview guide (Appendix 1 – only Norwegian). The first author carried out the interviews at the informants’ workplace in autumn 2019. The authors had no previous relationship with the informants. We audio recorded the interviews, and the interviews varied in length from 13 to 45 minutes (27.5 minutes on average). The interviews were transcribed verbatim.
Data analysis
We coded and analysed the transcribed data material thematically, in line with Braun and Clarke’s approach (20). First, we identified and coded the meaning units in the transcript and then synthesised them to themes and codes, establishing sub-categories under three main themes (Table 1).
Ethical considerations
Participation in the study was voluntary. The participants were informed about the study and gave written consent. All information was anonymised and treated confidentially. We made audio recordings using a mobile device and submitted the recording directly for storage in Services for sensitive data (TSD) at the University of Oslo.
The study was approved by the Norwegian Centre for Research Data (NSD) (reference number 428901) and the data protection officer at the hospital in question (reference number 19/21438).
Results
The informants described several factors of significance when deciding whether to use e-consultations via a digital patient portal. The main themes that emerged in the analysis were as follows: Selection of patients, Professional accountability and a satisfactory standard of health careand Reorganisation of practice (Figure 1).
Selection of patients
The informants emphasised that e-consultations are most suited to patients who want to interact in this way, and who meet certain criteria or preconditions: As expressed by one of the informants: ‘… is for people who visit [the hospital] once a year, but don’t feel it’s necessary and feel healthy. They have to be a bit interested because there’s quite a lot to learn, they have to have a computer and be able to write in Norwegian, so it’s certainly not for everyone’ (informant 1).
Several informants, both RNs and doctors, expected that communicating with patients via e-consultations might be challenging, especially if patients are unstructured: ‘When you are writing and chatting with a patient, there can be an awful lot of back and forth if they are not highly structured’ (informant 2).
Another informant said: ‘I expect patients to be aware of what this entails so that they don’t expect to get more help from us or use the service more often than intended’ (informant 5).
These statements show that the informants were concerned that patients who are offered an e-consultation possess the necessary capabilities and resources, are interested, have an adequate command of Norwegian, and understand and accept the terms of use. In addition, they should be in a stable phase of their illness. The informants believed, therefore, that patients should be selected based on whether or not they met these criteria.
Professional accountability and a satisfactory standard of health care
The informants stressed that the absence of clinical assessment and general observations might imply that patients became sicker over time, because the established routines of a standard consultation were dropped. ‘Not seeing the patients clearly entails risk. Something might have happened to their medication, or there may be adverse effects that they don’t think to ask about or haven’t noticed themselves’ (informant 2).
Another informant said: ‘It’s good to see the patients because then you can observe body language, see how the patient is feeling, note changes in weight, appearance, skin, all the signs you can’t see when you’re only writing to each other’ (informant 5).
The informants also mentioned the increased risk of misinterpretations as an area of concern: ‘What I find worrying is that misunderstandings can arise. If you send an answer in writing, you don’t know if the recipient has understood what you have written, unless they reply and ask about it, …. maybe they think they have understood, but I don’t really know whether that is the case’ (informant 3).
The informants also mentioned the increased risk of misinterpretations as an area of concern.
All the informants were worried about how the use of e-consultations could influence the relationship they have with patients, and possibly give rise to adverse effects on treatment and follow-up: ‘I think we overlook an awful lot of things [already], and we will also lose the kind of contact we have when we’re sitting in the same room’ (informant 9).
They also pointed out that patients may need the social and interpersonal contact that face-to-face consultations with their health professional involve: ‘Our job is to make their everyday existence as normal as possible. For some individuals, we’re the only people who know that they have this illness’ (informant 2).
Meanwhile, several informants believed that the use of e-consultations might lead to more openness on the part of their patients: ‘For some, it can be a better way of opening up because they find it easier to raise and discuss difficult issues when we’re not physically sitting together. It’s maybe easier to write than say something face to face’ (informant 5).
The informants were thus concerned about whether e-consultations provided the necessary conditions for safeguarding professional accountability and a satisfactory standard of health care when the opportunity to observe the patient directly is reduced.
In addition, they were afraid that digital asynchronous communication could affect interpersonal relationships and thus have an adverse impact on the patient. Conversely, informants also pointed out that patients may find it easier to open up when using e-consultations.
Reorganisation of practice
Several informants believed it would be time-consuming to implement and offer health care via e-consultations: ‘It will undoubtedly take more time because now the patient has to write down their question and then I have to wait until it appears on my monitor, and then I have to write an answer. The fact that the patient doesn’t need to come here is more efficient use of their time, but not necessarily mine. I think it will mean more work’ (informant 1).
Many anticipated that e-consultations would not necessarily improve work efficiency, but if it did, it would free up time: ‘I don’t think there will be less work, but if the work is kept within set limits, it may prove to be a useful supplement. If we save time, we can spend it on patients who are in greater need’ (informant 9).
Many anticipated that e-consultations would not necessarily improve work efficiency.
Another informant said as follows: ‘I don’t have very high expectations unless this replaces something else, and I don’t think it will. I think it will be just one more channel to relate to’ (informant 6).
The informants were also interested in setting up routines for using e-consultations: ‘We must restructure our work practices. If we are to cope with this, we must be able to interact with the patient without interruption. We must create some routines based on how we envisage this’ (informant 5).
All the informants said it was important that e-consultations were equated with ordinary consultations in terms of time and reimbursement: ‘Time should be allocated for e-consultations in the appointments book, and this must be paid in the same way as a face-to-face consultation so that there is no invisible work’ (informant 8).
To summarise, the informants did not expect e-consultations to replace other communication channels or improve work efficiency, and in their view, e-consultations should be equated with ordinary consultations.
Discussion
Factors expected to impact on e-consultations
The informants pointed out several important preconditions for offering patients e-consultations: a well-established and stable treatment pathway, medication administered under prescription, and that the person concerned had the ability to carry out necessary self-observation. These factors show that patients’ insight into their own illness and a clear understanding of health information increase the probability of using and deriving benefit from e-consultations (27).
When patients interact with their health professional digitally, they must observe, assess and explain symptoms to a greater extent than before. They must communicate changes to the health professionals and comprehend the information they receive in return, which requires both good language skills, and a reasonable capacity to understand health information (14, 21, 22).
When patients interact with their health professional digitally, they must observe, assess and explain symptoms to a greater extent than before.
Previous research has also reported that patients should have a basic level of digital skills and technical competence, must be interested in digital services and have access to a computer or smartphone in order to log on to the patient portal (16).
Furthermore, health professionals must be confident that the patients have an adequate understanding of their own health situation, and that they share relevant information. A study in which patients with diabetes received instruction from health professionals via e-consultations found that it was crucial that the first meeting was face-to-face (23). In this way, an empathetic/productive relationship could be established, and together they could make an appropriate plan for successful digitally mediated interactions. Asynchronous e-consultations should therefore be used in a treatment context where the patient and the health professionals already know each other.
Another key finding in this study is the informants’ concern that asynchronous e-consultations may not offer a satisfactory standard of health care. This concern is particularly linked to the reduced opportunity to carry out clinical assessment and examinations, with the result that changes in the patient’s condition and new symptoms are discovered later than in an ordinary consultation. This may imply that health professionals need additional skills when it comes to identifying the patient’s symptoms and needs via asynchronous e-consultations (24).
The informants also highlighted the potential danger of misunderstandings since it would be more difficult to discover whether the patient had understood the information or not. Asynchronous, text-based e-consultations require patients to express themselves in writing. Research has shown that health professionals are careful about how they word things, and often adapt their language to avoid misunderstandings in e-consultations.
In addition, it is easier to discover if the patient has understood what has been said in face-to-face communication as opposed to written communication – and this factor is stressed as an important element of face-to-face consultation (17, 21, 24). Therefore, offering e-consultations based on patients’ language skills, capacity to understand health information and motivation for engagement, may be crucial in ensuring a high professional standard in consultations and reducing the potential for misunderstandings.
The informants assumed that asynchronous e-consultations were a secure form of communication, where personal integrity and privacy were safeguarded, pointing out that the portal must be fast and easy to use in practice if it is to be at all useful. Earlier research stresses that digital services must be efficient and user-friendly, and that to achieve this, patients and health professionals should be involved in the development of such services (6, 15).
The informants assumed that asynchronous e-consultations were a secure form of communication, where personal integrity and privacy were safeguarded.
The informants assumed that few patients would want to sign up for the service, because they believed that patients prefer direct communication with health professionals. This view is supported by research that shows that patients who do not use e-consultations say that they prefer face-to-face consultation and believe that e-consultations would change their relationship with health professionals (3, 25, 26). However, patients who have used digital services report greater satisfaction and are described as being more engaged in decision-making processes (22, 27).
The informants regarded the social and interpersonal interaction in face-to-face consultations as being of key importance in health care for the patient group in question. While some believed that patients would be more honest and address issues in an asynchronous e-consultation that they would not otherwise ask about, others were worried that the patients might feel more isolated because of losing the opportunity to share their experiences of their illness.
The informants in the study expected that relationships with patients would change with the new service. This concurs with the expectations expressed by health professionals in a Finnish study on the implementation of a digital patient portal (10).
The expected benefits of e-consultations
The informants expected a restructuring of practice when asynchronous e-consultations became available, and they were keen to maintain the important elements of a traditional consultation. They assumed that the text-based e-consultations would increase their workload because patients will acquire yet another channel for contacting health professionals. A study in which health professionals expressed concern about the unintended consequences of using e-consultations supports this view (28).
The informants were afraid that patients who used the new form of consultation would nevertheless be obliged to call the hospital unit to clear up misunderstandings in the exchange of messages, since not everything can be clarified digitally. McInnes et al. also express this concern (29).
In order to avoid additional work, the informants felt that users of a digital patient portal must have clear guidelines that are accepted by patients and health professionals alike. They considered it important that the work is kept within set limits in order to avoid a heavier workload (28).
Moreover, the informants were of the opinion that digitally mediated consultations must be equated with ordinary consultations, in terms of both time and reimbursement, and this is supported by another Norwegian study (26).
The informants were unable to identify any direct benefit of asynchronous e-consultations for their own work, but believed, on the other hand, that it might benefit patients. They also believed that if it improved work efficiency, the time saved could be spent on patients who needed more follow-up. However, this view was expressed prior to the COVID-19 pandemic.
The need for digital solutions that do not require physical attendance is now strongly in focus since the potentialities and benefits of such solutions are seen very differently today. Many of the COVID-19 initiatives are based on synchronous video consultations and not asynchronous, text-based, procedure-steered communication. Experiences over the past year may have impacted on health professionals’ expectations of whether patients will wish to use asynchronous e-consultations or not.
Limitations of the study
The findings of this exploratory, qualitative study, such as reorganisation of practice and selection of patients, concur with earlier research. This strengthens the credibility of the study. The findings may therefore have some transferability to other outpatient clinics that intend to offer asynchronous e-consultations to patients with a chronic illness.
A weakness of the study is that the data were collected from only nine informants in one outpatient clinic. However, these informants were knowledgeable about the actual practice and represented RNs and doctors, which may strengthen the credibility of the study.
The first author carried out the interviews and transcriptions. Parts of the coding and analysis were performed in cooperation with the co-authors for quality assurance of the work. The empirical data was collected in autumn 2019. The experiences gained during the COVID-19 pandemic from spring 2020 may have affected the findings of the study, even though there is reason to believe that concerns about maintaining professional accountability and a satisfactory standard of health care would remain the same.
Conclusion
The most important finding in this exploratory study is the informants’ concern about whether asynchronous, text-based e-consultations safeguard professional accountability in health care. The informants pointed out that health literacy, careful selection of patients and reorganisation of practice were important components for successful use of text-based e-consultations where professional accountability is upheld.
More research on e-consultations and digital patient portals is needed to understand the complexity of their use, particularly in relation to different functionalities and methods that measure effects.
Moreover, the experiences of health professionals and patients in using e-consultations are important for the further development and use of digital services, so that they satisfy requirements for patient safety and professional accountability in health care. At the same time, patients and health professionals must perceive them as useful and easy to use.
References
1. Helse og omsorgsdepartementet. Strategi for å øke helsekompetansen i befolkningen 2019–2023. Oslo: Helse- og omsorgsdepartementet; 2019. Available at: https://www.regjeringen.no/contentassets/97bb7d5c2dbf46be91c9df38a4c94183/strategi-helsekompetanse-uu.pdf (downloaded 10.11.2020).
2. Meld. St. 9 (2012–2013). Én innbygger – én journal. Digitale tjenester i helse- og omsorgssektoren. Oslo: Helse- og omsorgsdepartementet; 2012.
3. Anthony D, Campos-Castillo C, Lim P. Who isn't using patient portals and why? Evidence and implications from a national sample of US adults. Health Affairs. 2018;37(12):1948–54. DOI: 10.1377/hlthaff.2018.05117
4. O’Connor S, Hanlon P, O’Donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC Med Inform Decis Mak. 2016;16(1):120. DOI: 10.1186/s12911-016-0359-3
5. Lancaster K, Abuzour A, Khaira M, Mathers A, Chan A, Bui V, et al. The use and effects of electronic health tools for patient self-monitoring and reporting of outcomes following medication use: systematic review. J Med Internet Res. 2018;20(12):e294. DOI: 10.2196/jmir.9284
6. Huygens M, Vermeulen J, Swinkels I, Friele R, de Witte L. Expectations and needs of patients with a chronic disease toward self-management and eHealth for self-management purposes. BMC Health Serv Res. 2016;16(1):232. DOI: 10.1186/s12913-016-1484-5
7. Zanaboni P, Ngangue P, Mbemba GIC, Schopf TR, Bergmo TS, Gagnon MP. Methods to evaluate the effects of internet-based digital health interventions for citizens: systematic review of reviews. J Med Internet Res. 2018;20(6):e10202. DOI: 10.2196/10202
8. Helsedirektoratet, Direktoratet for e-helse. Rapport. E-konsultasjon – Helsedirektoratet og Direktoratet for e-helse. E-konsultasjon og triage hos fastlege 2019 [Internet]. Oslo: Helsedirektoratet, Direktoratet for e-helse; 01.11.2019 [updated 05.12.2019, cited 10.11.2020]. Available at: https://www.helsedirektoratet.no/rapporter/e-konsultasjon-helsedirektoratet-og-direktoratet-for-e-helse
9. Direktoratet for e-helse. Kraftig økning i bruk av digitale helsetjenester under koronakrisen [Internet]. Oslo: Direktoratet for e-helse; 30.11.2018 [updated 05.12.2019, cited 10.11.2020]. Available at: https://ehelse.no/aktuelt/korona-okt-bruk-av-digitale-helsetjenester-i-befolkningen
10. Kujala S, Hörhammer I, Kaipio J, Heponiemi T. Health professionals' expectations of a national patient portal for self-management. Int J Med Inform 2018;117:82–7. DOI: 10.1016/j.ijmedinf.2018.06.005
11. Verdens helseorganisasjon (WHO). Ottawa: WHO; 1986. Available at: https://www.who.int/teams/health-promotion/enhanced-wellbeing/first-global-conference (downloaded 10.11.2020).
12. Tulu B, Trudel J, Strong DM, Johnson SA, Sundaresan D, Garber L. Patient portals: an underused resource for improving patient engagement: an underused resource for improving patient engagement. Chest. 2016;149(1):272–7. DOI: 10.1378/chest.14-2559
13. Hoogenbosch B, Postma J, De Man-Van Ginkel JM, Tiemessen NAM, Van Delden JJM, Van Os-Medendorp H. Use and the users of a patient portal: cross-sectional study. J Med Internet Res. 2018;20(1):e262. DOI: 10.2196/jmir.9418
14. Javier SJ, Troszak LK, Shimada SL, McInnes DK, Ohl ME, Avoundjian T, et al. Racial and ethnic disparities in use of a personal health record by veterans living with HIV. J Am Med Inform Assoc. 2019;26(8–9):696–702. DOI: 10.1093/jamia/ocz024
15. Otte-Trojel T, de Bont A, Rundall TG, van de Klundert J. What do we know about developing patient portals? A systematic literature review. J Am Med Inform Assoc. 2015;23(e1):e162–8. DOI: 10.1093/jamia/ocv114
16. Kooij L, Groen WG, van Harten WH. Barriers and facilitators affecting patient portal implementation from an organizational perspective: qualitative study. J Med Internet Res. 2018;20(5):e183. DOI: 10.2196/jmir.8989
17. Zhao JY, Song B, Anand E, Schwartz D, Panesar M, Jackson GP, et al. Barriers, facilitators, and solutions to optimal patient portal and personal health record use: a systematic review of the literature. AMIA Annu Symp Proc. 2018;2017:1913–22. Available at: https://pubmed.ncbi.nlm.nih.gov/29854263/ (downloaded 10.11.2020).
18. Grünloh C, Myreteg G, Cajander Å, Rexhepi H. «Why do they need to check me?» Patient participation through eHealth and the doctor-patient relationship: qualitative study. J Med Internet Res. 2018;20(1):e11. DOI: 10.2196/jmir.8444
19. Sust PP, Solans O, Fajardo JC, Peralta MM, Rodenas P, Gabaldà J, et al. Turning the crisis into an opportunity: digital health strategies deployed during the COVID-19 outbreak. JMIR Public Health Surveil. 2020;6(2):e19106. DOI: 10.2196/19106
20. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101. DOI: 10.1191/1478088706qp063oa
21. Irizarry T, Dabbs AD, Curran CR. Patient portals and patient engagement: a state of the science review. J Med Internet Res. 2015;17(6):e148. DOI: 10.2196/jmir.4255
22. Kruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: a systematic review. J Med Internet Res. 2015;17(2):e44. DOI: 10.2196/jmir.3171
23. Brandt CJ, Søgaard GI, Clemensen J, Søndergaard J, Nielsen JB. Determinants of successful eHealth coaching for consumer lifestyle changes: qualitative interview study among health care professionals. J Med Internet Res. 2018;20(7):e237. DOI: 10.2196/jmir.9791
24. Das A, Faxvaag A, Svanæs D. The impact of an eHealth portal on health care professionals’ interaction with patients: qualitative study. J Med Internet Res. 2015;17(11). DOI: 10.2196/jmir.4950
25. Zickmund S, Hess R, Bryce C, McTigue K, Olshansky E, Fitzgerald K, et al. Interest in the use of computerized patient portals: role of the provider–patient relationship. J Gen Intern Med. 2008;23(supplement 1):20–6. DOI: 10.1007/s11606-007-0273-6
26. Lyles CR, Allen JY, Poole D, Tieu L, Kanter MH, Garrido T. «I want to keep the personal relationship with my doctor»: understanding barriers to portal use among African Americans and Latinos. J Med Internet Res. 2016;18(10). DOI: 10.2196/jmir.5910
27. Wong JIS, Steitz BD, Rosenbloom ST. Characterizing the impact of health literacy, computer ability, patient demographics, and portal usage on patient satisfaction with a patient portal. JAMIA Open. 2019;2(4):456–64. DOI: 10.1093/jamiaopen/ooz058
28. Miller DP jr., Latulipe C, Melius KA, Quandt SA, Arcury TA. Primary care providers' views of patient portals: interview study of perceived benefits and consequences. J Med Internet Res. 2016;18(1):e8. DOI: 10.2196/jmir.4953
29. McInnes DK, Solomon J, Bokhour B, Asch S, Ross D, Nazi K, et al. Use of electronic personal health record systems to encourage HIV screening: an exploratory study of patient and provider perspectives. BMC Res Notes. 2011;4:295. DOI: 10.1186/1756-0500-4-295
Comments