Healthcare personnel’s assessment of their competence after a course in trauma nursing

Trauma is the leading global cause of death and disability (1). In western countries, injuries are the main cause of death among people aged between 1 and 44 (2). An organised and formalised trauma system with dedicated trauma teams has been shown to optimise patients’ clinical outcomes after injury and lead to higher survival rates (3, 4).

Based on recommendations in a report on the trauma system in Norway ( Traumesystem i Norge), Norwegian trauma hospitals must meet certain criteria for resources, and trauma team specialists must meet competence requirements (5).

Competence can be defined as ‘the aggregate knowledge, skills, abilities and attitudes that make it possible to perform relevant functions and tasks in accordance with defined requirements and goals’ (6, p. 48). Self-perceived competence relates to how individuals view their own competence in different areas, and is not an objective measurement (7).

Background

Oslo University Hospital, Ullevål is Norway’s largest trauma hospital, with around 2000 trauma patients every year. Its trauma team consists of 15 specialists, each with their own specific duties (8).

Trauma treatment at Oslo University Hospital, Ullevål began in 1984 with the activation of teams of trauma specialists and the implementation of guidelines for receiving and treating trauma patients in line with the hospital’s trauma manual (3, 8). The trauma manual is based on Advanced Trauma Life Support (ATLS), a course in life-saving treatment for doctors. The course aims to provide theoretical knowledge and practical skills in relation to trauma patients (9-11).

Course in Trauma Nursing

Advanced Trauma Care for Nurses (ATCN) is an international course for nurses in trauma teams. This course is based on ATLS. In Norway, the Course in Trauma Nursing is organised as a two-day basic course in trauma treatment. The course model is based on ATLS and ATCN, and provides an introduction to standardised principles for receiving and treating injured trauma patients.

The Course in Trauma Nursing focuses on initial treatment upon arrival at a hospital qualified to deal with trauma patients. The course began in 2011 and has since expanded its reach to 40 Norwegian hospitals with a trauma function. The target group for the course is nurses who are involved in the reception of trauma patients, but it is also offered to other professions with an active role in trauma treatment. The course has been quality assured by the Department of Traumatology at Oslo University Hospital, Ullevål and the National Advisory Unit on Trauma.

Several studies claim there is a need for more knowledge about the impact of trauma courses and trauma training on nurses (12-15). Evidence shows that the level of professional knowledge increases after participation in ATLS and ATCN (16-19). However, the literature identifies shortcomings in studies that have evaluated the impact of self-perceived competence in nurses after participating in trauma courses.

Objective of the study

The objective of our study was to investigate healthcare personnel’s (henceforth referred to as course participants) self-perceived change in competence in the reception and treatment of trauma patients before and after completing the Course in Trauma Nursing.

We formed the following hypothesis: course participants’ competence in the reception and treatment of trauma patients will be improved two months after completing the Course in Trauma Nursing. The aim of the study was to answer two research questions:

• To what extent does the Course in Trauma Nursing contribute to self-perceived change of competence in the reception and treatment of trauma patients?
• What correlations are there between gender, age, number of years of further education, trauma team participation, participation in previous trauma courses and changed competence after the Course in Trauma Nursing?

Method

Design

The study has a prospective survey design, where we have applied the pretest-posttest method to a single group. We collected data in an electronic questionnaire two weeks before the start of the course and two months after the end of the course.

Sample

Seven hospitals that arranged the Course in Trauma Nursing in November and December 2014 were invited to participate in the study. One of the seven hospitals did not provide consent to the study before the start of the course and was therefore excluded. The hospitals varied in geographical location and size.

A coordinator for the course at each institution asked registered course participants if they wanted to participate in the study, and 94 participants were invited. Inclusion criteria were nurses, specialist nurses, radiographers and ambulance workers who signed up for the Course in Trauma Nursing. Exclusion criteria were course participants who did not complete the course or answered less than 50 per cent of the questions in the questionnaire.

Questionnaire

For this study, we further developed an existing, validated questionnaire (20). Figure 1 provides a summary of this.

The original questionnaire consisted of four parts. Part 1 dealt with concordance between specialist nurses’ self-perceived competence and job requirements. Part 2 related to concordance between colleagues’ expectations for competence and self-perceived competence after completion of their education. In Part 3, we asked for suggestions for improvements to the education programme, and in Part 4 we recorded background data.

We excluded Parts 2 and 3 because they were not relevant to our study. Part 1 contained 36 closed questions about self-perceived competence, divided into seven competence areas. Four of the seven areas were excluded because they were not relevant to the Course in Trauma Nursing.

The three competence areas that we used in the questionnaire for this study were medical competence, teamwork competence and choice and improvisation competence.

We revised and adapted the questions in Part 1 to the course syllabus in order to survey the competence of the course participants. The resulting questionnaire was given the title ‘Evaluation of self-perceived competence in the reception and treatment of trauma patients before and after the Course in Trauma Nursing’ (EVAKITS1 for the pretest and EVAKITS2 for the posttest).

The questionnaire consisted of 23 closed questions about practical and theoretical competence in the reception and treatment of trauma patients. Of these, 17 questions related to medical competence, three related to teamwork competence and three concerned choice and improvisation competence.

The response alternatives were in the form of a seven-point Likert scale, where 1 = I do not feel very competent at all and 7 = I feel very competent. We asked the course participants to evaluate their self-perceived competence using the scale for each question within the three competence areas. The questionnaires were produced electronically using the Questback computing program.

Panel of experts

We created a panel of experts in order to reinforce the validity and reliability of the questionnaire. The remit of the panel was to use their experience to evaluate the questionnaire as a data collection method (21). Twelve nurses from different hospital departments were invited to join the panel.

The panel was made up of nurses and specialist nurses, about half of whom had completed the Course in Trauma Nursing. They gave us feedback on the questionnaire content and question formulation (22). This feedback led us to reformulating some questions that seemed unclear, without changing the areas of competence and content.

Data collection

We collected the data in the period from November 2014 to February 2015. The Course in Trauma Nursing coordinator instigated the first contact with the course participants. We sent letters with information about the purpose of the study and informed consent to the respondents before EVAKITS1 was distributed. Return of the questionnaire was regarded as consent to taking part in the study.

In order to investigate changes in competence, it was necessary to compare the course participants’ responses from EVAKITS1 and EVAKITS2 and then consider the group’s overall change. Respondents who did not answer EVAKITS1 were not therefore sent EVAKITS2.

Analysis

The data were transferred electronically from Questback to the Statistical Package for the Social Sciences (SPSS), version 22.0 for Mac. We used descriptive statistics with frequency and percentages for categorical variables to describe the demographics of the course participants.

We also used mean and standard deviations for continuous variables. The response alternatives on the Likert scale were treated as continuous because the Likert scale was greater than 5 and summarised as a total score (23).

Each question had a minimum score of 1 and a maximum score of 7. Medical competence gave a maximum score of 119. Teamwork competence and choice and improvisation competence each had a maximum score of 21. The majority of the answers to the questions had a normal distribution.

In order to answer the first research question, we examined the change in competence with a paired sample t-test. For the second research question, we examined how the independent variables (number of years of further education, trauma team participation, age, trauma course participation and gender) impacted on the dependent variable (mean change examined for 23 questions in the questionnaire). The mean change was the differential between mean posttest and mean baseline.

First, we tested the independent variables in univariate analyses. We then performed a multiple regression analysis where we examined the variables that were significantly related to change in competence (the dependent variable). Only variables that achieved significance in the univariate analyses for the same question were controlled in the latter analysis. P-value <0.05 was considered statistically significant.

Ethics

The study is approved by the Norwegian Centre for Research Data (NSD). The data we collected were stored separately from personal identification information and password-protected linked identifiers between course participants and the data material.

Results

Sample

Of the 94 participants, 52 responded to the pretest (55.3 per cent). Forty-five of those who responded to the pretest returned the posttest questionnaire (86.5 per cent). A total of 16 men (31 per cent) and 35 women (69 per cent) took part.

Seventy-seven per cent of the course participants were involved in a trauma team. The mean clinical experience for receiving and treating trauma patients was 8.5 years (SD 8.3). Table 1 gives an overview of the participants’ socio-demographic variables.

Competence after completed course

Course participants reported a significantly greater increase in medical competence and choice and improvisation competence two months after the Course in Trauma Nursing compared to the pretest (see Table 2). Course participants also reported an increase in teamwork competence, but for the question relating to knowledge of own area of responsibility in the trauma team, the increase was not significant.

How individual variables impact on changed competence

The individual variables that were controlled for change are shown in Table 3.

Gender

Regarding the questions ‘Receiving and treating pregnant trauma patients’ and ‘Assessing trauma patients according to the Glasgow Coma Scale (GCS)’, men found that their competence improved to a greater extent than women.

Trauma team

For the question ‘Having knowledge of your own area of responsibility in the trauma team’, course participants who were not part of a trauma team reported a significantly greater improvement in their competence compared to those who did work in a trauma team.

Trauma course

Trauma course was the variable that impacted on changes for most questions. Course participants who had not participated in trauma courses other than the Course reported an improvement in competence for four of the questions, compared to those who had attended other trauma courses.

All of the questions related to medical competence: ‘Receiving trauma patients according to the ABCDE approach’, ‘Fitting and adjusting neck collars’, ‘Receiving and treating respiration and/or circulation in unstable trauma patients’ and ‘Receiving and treating trauma patients with unstable head injuries’.

Age and number of years of further education

Three questions showed a significantly negative change for age. When age increases by one year, the change is reduced by 0.04 for the question ‘Initiating measures to clear respiratory passages’. ‘Identifying different types of shock’ was reduced by 0.03 and ‘Rapid prioritising/re-prioritising of tasks when the situation so demands’ was reduced by 0.03.

As Table 3 shows, the changes for the answers to the aforementioned three questions were also adversely affected by the number of years of further education. Because the two variables showed significant change for the same three questions, we performed a multiple linear regression analysis to investigate whether the combination of variables impacted on each other.

The analyses showed that when the number of years of further education is constant and the age increased by one year, the course participants reported a reduction in competence for one of the questions: ‘Rapid prioritising/re-prioritising of tasks when the situation so demands’ (p = 0.044) (data not shown).

The combination of age and further education was not significant for the other two questions. None of the remaining independent variables showed change for the same question, so a multiple regression analysis was only performed for the aforementioned two variables.

Discussion

The results show that all course participants largely reported a positive change in competence in all competence areas, from baseline to two months after the course.

Questions showing the greatest change in competence

As Table 2 shows, the course participants experienced the greatest change in competence in relation to the questions ‘Fitting and adjusting neck collars’, ‘Log rolling patients’ (see the fact box), ‘Receiving and treating pregnant trauma patients’ and ‘Initiating measures to stop blood loss in accordance with the massive haemorrhage protocol’.

The practical exercise stations in the Course in Trauma Nursing focus on teaching course participants procedures that are highly relevant to the reception and treatment of trauma patients. Fitting a neck collar and log rolling are themes of two of the exercise stations. Course participants are taught the theory before practising the procedures under the supervision of an instructor.

The large differential in mean scores may indicate that the course participants find that the exercises help to improve competence. The findings are supported by the literature, which shows that training in technical simulation improves skills and increases the rate of retention (24-25).

The reception and treatment of trauma patients in the period between the baseline and the posttest may be a factor that impacts on the change of competence as a result of course participants being able to put their knowledge from the course into practice. The Course in Trauma Nursing has a standardised, theoretical education programme. Lectures are given on, for example, injury mechanisms and special patient groups such as the elderly, children, burns patients and pregnant patients.

Pregnant trauma patients 

Pregnant trauma patients are a rare patient group (8, 26). It is therefore considered unlikely that a marked change in competence is correlated with two months of experience from practice. It is more likely that the lecture on pregnant trauma patients has contributed to their knowledge. The improvement in course participants’ competence is due to the fact that they have learned how to receive and treat this patient group.

Massive haemorrhage is greatest challenge

Haemorrhages account for 40 per cent of trauma-related deaths, and the most challenging surgical situations in trauma treatment are caused by massive haemorrhages (27, 28). Patients in haemorrhagic shock are critically ill and require immediate blood transfusion and surgical intervention (29).

On the Course in Trauma Nursing, the guidelines on massive transfusion protocol are discussed in the lecture on shock treatment. The participants’ increase in self-perceived change in relation to the question ‘Initiating measures to stop blood loss in accordance with the massive haemorrhage protocol’ may be the result of newly acquired knowledge from the theoretical lecture.

Positive change in competence

The questions about medical competence focused specifically on the theoretical lectures in the Course in Trauma Nursing. Teamwork competence and choice and improvisation competence are not covered so specifically in theoretical lectures. Nevertheless, the results show a positive change of competence for both areas of competence.

Teamwork competence entails the team members’ ability to perform their duties in a flexible manner, among other things. The reception and treatment of trauma patients can be challenging and resource-intensive because many parallel actions take place simultaneously.

This requires every person in the trauma team to carry out their specific duties satisfactorily. At the same time, knowledge about their own duties and the duties of others is a contributing factor in functional teamwork (30).

The positive self-perceived change in relation to the question ‘Having knowledge of other trauma team members’ areas of responsibility’ may be due to the fact that the Course in Trauma Nursing is offered to specialist nurses and other professions involved in trauma teams.

Joint training for the different professions involved in a trauma team can generate interest across the groups. The Course in Trauma Nursing instructors are nurses with different specialities from trauma teams. This gives the course a broad scope in terms of areas of responsibility since the respective occupational groups can provide more detailed descriptions of their duties.

Questions without significant change in competence

Knowledge of own area of responsibility in the trauma team was the only question that showed no statistical significance after participants had completed the course. Mean baseline scores for this question had the highest output value. This value indicates that the course participants felt that their competence was already very good prior to taking the course. There was therefore less potential for improvement.

Choice and improvisation competence also shows a positive change from baseline to posttest, and had the lowest mean change differential. In hindsight, the formulation of questions in this area of competence could be called into question since the questions are rather vague and are open to interpretation.

The questions are not necessarily a good indicator of the effectiveness of the actual course. However, the overall positive change may be due to the fact that the Course in Trauma Nursing generally contributes to knowledge, which makes participants feel more confident and competent in making choices and improvising.

Involvement in a trauma team as a contributing factor

The results for two of the questions showed significantly greater competence for course participants who were not involved in a trauma team than for those who were part of a trauma team. These two questions were ‘Fitting and adjusting neck collars’ and ‘Having knowledge of your own area of responsibility in the trauma team’.

All trauma patients received and treated by a trauma team must have a neck collar fitted prior to their arrival in hospital. Patients who arrive without a neck collar are fitted with one as a matter of urgency.

The differential between the two groups can be explained by the fact that course participants who are not involved in a trauma team benefit more from the practical exercise station because they are less knowledgeable about the procedure in the first place. Even if a procedure has not been carried out in practice after the Course in Trauma Nursing, new knowledge can contribute to a self-perception of greater competence.

Gender as a contributing factor

The mean change in competence between the sexes showed a significant differential in just one question. For the question ‘Receiving and treating pregnant trauma patients’, men reported more of an improvement in competence than women.

Trauma course as a contributing factor

Compared with course participants who had been on a trauma course, participants who had not participated in a trauma course showed a significant positive change for four of the questions about medical competence.

The changes shown for the relevant questions may indicate that the Course in Trauma Nursing learning curve is steeper for participants who have not taken a trauma course than for those who have participated in such a course. The reason may be that course participants who have been on a trauma course, such as ATCN, already have knowledge within the areas covered in the Course in Trauma Nursing. Thus, it is not quite so apparent whether this group’s competence has changed.

Age and number of years of further education as a contributing factor

The course participants’ age and number of years of further education showed a significantly negative change for the same three questions: ‘Initiating measures to clear respiratory passages’, ‘Identifying different types of shock’ and ‘Rapid prioritising/re-prioritising of tasks when the situation so demands’.

It is conceivable that newly qualified course participants with less work experience have greater self-confidence, and therefore give themselves higher scores. At the same time, younger participants may have a steeper learning curve because they tend to learn faster than the older participants (31).

Nevertheless, it is difficult to explain why age and number of years of further education only affected three of the 23 questions when combined in the regression analysis. However, for the question ‘Rapid prioritising/re-prioritising of tasks when the situation so demands’, age had a negative effect on the change when the number of years of further education was constant.

This indicates that age is the variable in this question that had the greatest negative impact on the changes. Because both of the aforementioned variables relate to a small minority of the questions, we cannot conclude that age and further education were contributing factors for changes in competence after the course.

Strengths and limitations

The biggest challenge in this study is the small sample. The sample size makes it particularly difficult to examine sub-samples. We may not, therefore, be able to identify statistical changes that could have been shown in a larger sample. This particularly applies to the results shown in Table 3. It was also difficult to make a calculation of strength before the study started as we lacked input data.

Course participants may be influenced by their expectations of the course, leading them to give their competence a higher score in the posttest (28). At the same time, renewed knowledge may make participants feel that they are more competent. There may also be a correlation between change in competence and motivation, prior knowledge and the field of practice’s interest in competence development (32).

These factors are important and must be taken into consideration when the results are to be generalised vis-à-vis other samples or studies. In the linear regression analysis, we performed multiple analyses. This may be a weakness of the study since statistical significance can be achieved by chance (33).

Good response rate

The sample is taken from hospitals that vary in geographical location and size, something that strengthens the representativeness of the study. The original response rate of 55.4 per cent should ideally have been higher in order to be able to generalise (22, 34).

The response rate for EVAKITS2 (85.6 per cent) strengthens the internal validity of the study. The purpose of the study is to measure the subjective competence of course participants. The internal validity of the study is impaired because of the study’s inability to measure the changes without a control group, and this is the greatest weakness of the study.

Control group recommended

The pretest-posttest design of the study, where the same group is used without a control group, makes it difficult to check for sources of error, bias and contributing factors (35). We call on researchers who wish to carry out future studies to use a control group.

We further developed the validated questionnaire aimed at evaluating competence after further education in anaesthesia nursing, paediatric nursing, critical care nursing and theatre nursing ( Evaluering av kompetanse etter videreutdanning i anestesi-, barne-, intensiv- og operasjonssykepleie). As not all of the questions were relevant to our study, a panel of experts evaluated the relevant part of the questionnaire to ensure that it was clinically credible and valid (36).

Some working groups in the sample population were too small to investigate the mean change in competence for different professions. Future studies with larger samples are, however, encouraged to investigate change across the Course in Trauma Nursing working groups.

Conclusion

The study shows that the Course in Trauma Nursing contributes to increased self-perceived competence for course participants who receive and treat trauma patients, from baseline to two months after the course. The changes after completion of the course are statistically significant for 22 of the 23 questions concerning medical competence, teamwork competence and choice and improvisation competence. Thus, we can conclude that the hypothesis that we put forward before the study is supported.

There is essentially no differential in self-perceived change in competence in relation to gender, age, participation in a trauma team and number of years of further education.

Course participants who have not previously participated in trauma courses report a greater improvement in competence for about a quarter of the medical competence questions than those who have attended a trauma course.

The Course in Trauma Nursing should continue to be a priority area for acute care hospitals with a trauma function, both for trauma team specialists and for other professions involved in the treatment of trauma patients.

Given the study’s positive finding two months after the Course in Trauma Nursing, it would be interesting to investigate the long-term effect of the course. It would also be useful to investigate whether taking part in the course increases participants’ interest in traumatology. 

References

1.       Mock C, Lormand JD, Goosen J, Joshipura M, Peden M. Guidelines for essential trauma care. Geneve: World Health Organization; 2004.

2.       Groven S. Performance assessment of a major Scandinavian trauma center during implementation of a dedicated trauma service. (Doktoravhandling). Oslo: Det medisinske fakultet, Universitetet i Oslo; 2014.

3.       Groven S, Eken T, Skaga NO, Roise O, Naess PA, Gaarder T. Long-lasting performance improvement after formalization of a dedicated trauma service. J trauma 2011;70(3):569–74.

4.       Lo CJ, Lee HY. Evaluation of an advanced trauma life support course in Taiwan. Formosan Journal of Surgery 2014;47:221–6.

5.       Røise O. Traumesystem i Norge: forslag til organisering av behandling av alvorlig skadde pasienter. Hamar: Helse Øst; 2006.

6.       Lai L. Strategisk kompetansestyring. Bergen: Fagbokforlaget; 2004.

7.       Kleve R-AT. Motoriske vansker av selvopplevd kompetanse. (Master's thesis). Oslo: Psykologisk Institutt, Universitetet i Oslo; 2012.

8.       Ullevål universitetssykehus. Traumemanualen – initialbehandling av den multitraumatiserte pasient ved Ullevål universitetssykehus. 6 utgave. Oslo: Fresenius Kabi; 2011.

9.       Bell RM, Krantz BE, Weigelt JA. ATLS: A foundation for Trauma Training. Ann Emerg Med 1999;34(2):233–7.

10.     Ali J, Adam R, Stedman M, Howard M, Williams JI. Advanced trauma life support (ATLS) program increases emergency room application of trauma resuscitative procedures in a developing country. J Trauma 1994;36:391–4.

11.     Kennedy DWG, Gentleman D. The ATLS course, a survey of 228 ATLS providers. Emer Med J 2001;18:55–8.

12.     Hadfield-Law L. Advanced trauma nursing course (ATNC). Care of Critically Ill 1994;10(1):18–21.

13.     Tippet J. Nurses acquisition and retention of knowledge after trauma training. Accid Emerg Nurs 2004;12(1):39–46.

14.     Hammond F, Saba M, Simes T, Cross R. Advanced life support: retention of registered nurses’ knowledge 18 months after initial training. Aust Crit Care 2000;13(3):99–104.

15.     Armstrong B, Crouch R, Read C, Palfrey R. Training nurses in trauma management. Emerg nurse 2013;21(4):14–8.

16.     Ali J, Adam R, Pierre I, Bedaysie H, Josa D, Winn J. Comparison of performance 2 years after the old and new (interactive) ATLS courses. J Surg Res 2001;97(1):71–5.

17.  Mohammad A, Branicki F, Abu-Zidan FM. Educational and clinical impact of Advanced Trauma Life Support (ATLS) courses: A systematic review. World J Surg 2014;38:322–9.

18.     Patient L. Trauma training: a literature review. Emer nurse 2007;15(7):28–37.

19.     Canzian S, Nanni J, McFarlan A, Chalkin K, Sorvari A, Barratt L et al. Application and evaluation of knowledge retention related to Advanced Trauma Care for Nurses (ATCN) Course content: a preliminary study. J Trauma Nurs 2016;23(4):202–9.

20.     Valeberg BT, Grønseth R, Fagermoen MS. Spesialsykepleieres opplevde kompetanse etter endt utdanning. Sykepleien Forskning 2009;4(3):206-213. Available at: https://sykepleien.no/forskning/2009/10/spesialsykepleieres-opplevde-kompetanse-etter-endt-utdanning(downloaded 16.11.2017).

21.     Kirkevold Ø. Praktiske tips ved spørreundersøkelser, del 2. Sykepleien Forskning 2014;9(1):92–4. Available at: https://sykepleien.no/forskning/2014/04/praktiske-tips-ved-sporreskjemaundersokelser-del-2(downloaded 27.11.2017).

22.     Gustavsen G. Spørreskjemametodikk etter kokebokmetoden. Oslo: Ad Notam Gyldendal. 1999.

23.     Johannessen A. Introduksjon til SPSS. Versjon 17. Oslo: Abstrakt Forlag. 2008.

24.     Park C. Simulation and quality improvement in anesthesiology. Anesthesiology Clin 2011;(29):13–28.

25.     Green M, Tariq R, Green P. Improving patient safety through simulation training in anesthesiology: Where are we? Anesthesiol Res Pract 2016;2016:4237523.

26.     Horstmann P, Larsen CF, Grønborg H. Adherence to protocol in pregnant trauma patients? A 12-year retrospective study. European J Trauma Emerg Surg 2014;40(5):561–6.

27.     Nascimento B, Callum J, Rubenfeld G, Neto JBR, Lin Y, Rizoli S. Clinical review: Fresh frozen plasma in massive bleedings – more questions than answers. Critical Care 2010;14(1):202.

28.     Gaarder C, Naess PA, Buanes T, Pillgram-Larsen J. Advanced surgical trauma care training with a live porcine model. Injury 2005;36:718–24.

29.     Colwell C, Moreira M E, Grayzel J. Initial evaluation and management of shock n adult trauma. 2015. Available at: https://www.uptodate.com/contents/initial-evaluation-and-management-of-shock-in-adult-trauma?source=see_link(downloaded 02.02.2017).

30.     Flovik AM, Normann L, Mølstad K. Sykepleie – et selvstendig og allsidig fag (Sykepleiehefte). 2008. Available at: https://www.nsf.no/Content/135904/Sykepleie%202008.pdf(downloaded 03.02.2015).

31.     OECD. Statistics Canada. Learning a living: First results of the adult literacy and life skills survey. Ottawa/Paris: Statistics Canada: OECD; 2005. Available at: http://www.oecd.org/edu/innovation-education/34867438.pdf(downloaded 16.11.2017).

32.     Ellis L, Nolan M. Illuminating continuing professional education: unpacking the black box. Int J Nurs Stud 2005;42(1):97–106.

33.     Goldman M. Why is multiple testing a problem? 2008. Available at: http://www.stat.berkeley.edu/~mgoldman/Section0402.pdf(downloaded 10.04.2017).

34.     Cozby P C. Methods in behavioral research. 10. ed. Boston, MA: McGraw-Hill Medical Publishing; 2007.

35.     Campbell D T, Stanley A. Experimental and quasi-experimental designs for research. New Jersey, USA: Ravenio Books, 2015.

36.     Malt U. Face validity. I: Store norske leksikon. Available at: https://snl.no/face_validity(downloaded 11.04.2017).

37.     American College of Surgeons, Committee on Trauma. Advanced trauma life support (ATLS®). 9. utgave. Chicago, IL: American College of Surgeons; 2012.