Document Detail

A question-answer pair (QAP) database integrated with websites to answer complex questions submitted to the Regional Medicines Information and Pharmacovigilance Centres in Norway (RELIS): a descriptive study.
Jump to Full Text
MedLine Citation:
PMID:  22422916     Owner:  NLM     Status:  PubMed-not-MEDLINE    
OBJECTIVE: To assess a question-answer pair (QAP) database integrated with websites developed for drug information centres to answer complex questions effectively.
DESIGN: Descriptive study with comparison of two subsequent 6-year periods (1995-2000 and 2001-2006).
SETTING: The Regional Medicines Information and Pharmacovigilance Centres in Norway (RELIS).
PARTICIPANTS: A randomised sample of QAPs from the RELIS database.
PRIMARY OUTCOME MEASURE: Answer time in days compared with Mann-Whitney U test.
SECONDARY OUTCOME MEASURE: Number of drugs involved (one, two, three or more), complexity (judgemental and/or patient-related or not) and literature search (none, simple or advanced) compared with χ(2) tests.
RESULTS: 842 QAPs (312 from 1995 to 2000 and 530 from 2001 to 2006) were compared. The fraction of judgemental and patient-related questions increased (66%-75% and 54%-72%, respectively, p<0.01). Number of drugs and literature search (>50% advanced) was similar in the two periods, but the fraction of answers referring to the RELIS database increased (13%-31%, p<0.01). Median answer time was reduced from 2 days to 1 (p<0.01), although the fraction of complex questions increased from the first to the second period. Furthermore, the mean number of questions per employee per year increased from 66 to 89 from the first to the second period.
CONCLUSIONS: The authors conclude that RELIS has a potential to efficiently answer complex questions. The model is of relevance for organisation of drug information centres.
Jan Schjøtt; Linda A Reppe; Pål-Didrik H Roland; Tone Westergren
Related Documents :
22531946 - Understanding the glass-forming ability of active pharmaceutical ingredients for design...
6758716 - Studies on clonidine and guanfacine withdrawal after short term treatment in the rat.
22494466 - Designing prodrugs for the treatment of parkinson's disease.
22797836 - Practices associated with serum antiepileptic drug level monitoring at a pediatric neur...
10212906 - What have been the strategies for the registration, positioning and control of medical ...
19365276 - New anticoagulants for atrial fibrillation.
Publication Detail:
Type:  Journal Article     Date:  2012-03-15
Journal Detail:
Title:  BMJ open     Volume:  2     ISSN:  2044-6055     ISO Abbreviation:  BMJ Open     Publication Date:  2012  
Date Detail:
Created Date:  2012-03-16     Completed Date:  2012-10-02     Revised Date:  2013-11-06    
Medline Journal Info:
Nlm Unique ID:  101552874     Medline TA:  BMJ Open     Country:  England    
Other Details:
Languages:  eng     Pagination:  e000642     Citation Subset:  -    
Section of Clinical Pharmacology, Laboratory of Clinical Biochemistry, Haukeland University Hospital, Bergen, Norway.
Export Citation:
APA/MLA Format     Download EndNote     Download BibTex
MeSH Terms

From MEDLINE®/PubMed®, a database of the U.S. National Library of Medicine

Full Text
Journal Information
Journal ID (nlm-ta): BMJ Open
Journal ID (publisher-id): bmjopen
Journal ID (hwp): bmjopen
ISSN: 2044-6055
Publisher: BMJ Group, BMA House, Tavistock Square, London, WC1H 9JR
Article Information
Download PDF
© 2012, Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to
Received Day: 17 Month: 11 Year: 2011
Accepted Day: 16 Month: 2 Year: 2012
collection publication date: Year: 2012
Electronic publication date: Day: 15 Month: 3 Year: 2012
pmc-release publication date: Day: 15 Month: 3 Year: 2012
Volume: 2 Issue: 2
E-location ID: e000642
ID: 3307033
PubMed Id: 22422916
Publisher Id: bmjopen-2011-000642
DOI: 10.1136/bmjopen-2011-000642

A question–answer pair (QAP) database integrated with websites to answer complex questions submitted to the Regional Medicines Information and Pharmacovigilance Centres in Norway (RELIS): a descriptive study Alternate Title:RELIS: a model for drug information centres
Jan Schjøtt123
Linda A Reppe45
Pål-Didrik H Roland5
Tone Westergren6
1Section of Clinical Pharmacology, Laboratory of Clinical Biochemistry, Haukeland University Hospital, Bergen, Norway
2Section of Pharmacology, Institute of Medicine, University of Bergen, Bergen, Norway
3Regional Medicines Information and Pharmacovigilance Centre (RELIS Vest), Haukeland University Hospital, Bergen, Norway
4Faculty of Health Sciences, Nord-Trøndelag University College, Steinkjer, Norway
5Regional Medicines Information and Pharmacovigilance Centre (RELIS Midt-Norge), St. Olav's Hospital, Trondheim, Norway
6Regional Medicines Information and Pharmacovigilance Centre (RELIS Sør-Øst), Oslo University Hospital, Oslo, Norway
Correspondence: Correspondence to Dr Jan Schjøtt;


The regional medicines information and pharmacovigilance centres (RELIS) were established in 1995 with two regional centres. Two additional centres were started in 1997 and 1998 with a fifth added in 2001. Today, RELIS is a national network made up of four (two centres in Oslo became one in 2010) drug information centres (DICs). They are localised at regional university hospitals in Norway, where pharmacists and clinical pharmacologists answer questions from healthcare professionals. In 1995, RELIS started developing a full-text question–answer pair (QAP) database through cooperation with a software company (Arnett AS, Bergen, Norway). The database was originally an internal (not public) Microsoft Access application but was converted to a web-based SQL database with separate public (through the RELIS homepage) and internal (through the RELIS intranet) interfaces in 2000–2001. QAPs are indexed and published by the staff to be searchable in Norwegian for healthcare professionals free of charge through the public interface. QAPs unsuitable for the public domain (eg, questions answered solely by a copy of a previous answer, questions which lack public interest and not associated with drugs or questions that contain information that could identify a patient through description of a rare disease) are allocated to the internal interface (RELIS internal database) only.

QAPs have been retrieved from the database through a simple or advanced search function since 1995. Today, the RELIS public database by default contains a simple search function where a drug or text word (eg, hypertension) is entered, but a more advanced search function with Boolean operators (AND/OR/NOT), categories (eg, interactions, pregnancy), text word, individual centre with question number is an option. In the internal interface, the advanced search function is default. The result of a search (eg, loratadine AND pregnancy) is presented as a report of QAPs organised chronologically (the newest on top). An individual QAP in the report could contain a previous answer (in the RELIS public database) as a linked reference available in full text.

The websites include the RELIS homepage ( targeted at healthcare professionals and the RELIS intranet available for RELIS only. The websites were developed in 2000, updated in 2010 and represent the public and internal domain of the RELIS network, respectively. On the homepage, RELIS publish relevant drug information including relevant QAPs from the database, announce meetings, present recent publications and describe news from the pharmacovigilance work. Importantly, healthcare professionals can register for a monthly newsletter (email) with selected drug information from the homepage. The intranet functions as a medium for all activity and cooperation between the centres in Norway. Thus, it contains important documents, training and quality guidelines, discussions and information to all employees. In addition it includes dedicated sections for individual centres, common working groups and projects.

In 1995 and up to 2006, the majority of questions to RELIS were communicated through telephone followed by email. However, in the last years, questions have been mainly submitted through a web question form available on the homepage. Healthcare professionals register demographic data (occupation and working address), the question (the patient being made anonymous) and a deadline for the answer (answer time). Demographic data are used for distribution of questions to the respective centres, but all information in the web question form are shared by all RELIS staff, regardless of location, through exclusive emails. Other questions (through telephone, fax, letters or ordinary email) not submitted through the web-based question form have still been shared by all staff since 1995 through an ‘answer in progress’ functionality in the RELIS internal database. The name of the employee who received the question and the one handling it is also registered. In the case of particularly complex questions involving several authors, co-signatures are used. In this way, the handling of a question is fully traceable and answers to subsequent similar questions can be based on re-use of QAPs and communication with the authors.

Physicians consider the information provided by RELIS in general and on drug use in pregnancy in particular to be of high quality and of significant clinical impact.1–3 Questions to the centres are mainly about psychoactive drugs and adverse effects similar to other DICs.4, 5 Less is known about complexity of questions and how this influences answer time. In the present study, we retrospectively analysed a randomised sample of QAPs in two time periods after RELIS started. The objective was to examine if this model for DICs has a potential to answer complex questions effectively.


Contents of the RELIS database were analysed to illustrate how questions were answered. Two periods of 6 years were compared (1995–2000 and 2001–2006). QAPs published in the public database in the respective periods were used. They reflect the process of answering better than QAPs unsuited for publication as described above. Overall publication frequency in the period (1995–2006) was 80% (range 66%–95% between individual centres). Relevant data from QAPs, including question number, were transferred to SPSS V.17.0 (SPSS Inc.) for analysis and subjected to a randomisation procedure in the application. From a total of 9697 QAPs, a randomised sample of 963 QAPs was used for further analysis. Each QAP was compared with the information in the associated paper version (all documents and information on answering the question) saved in the respective RELIS centre. The paper version contained the day the question was received and the corresponding day an answer was provided. One hundred and twenty-one QAPs were excluded because the questions concerned nutritional or herbal medicine (n=65), non-medical products from pharmacies, homeopathy, medical equipment, disinfection or chemicals (n=26), or questions that were answered solely by a previous answer or by sending copies of relevant scientific articles (n=30). Exclusion of these QAPs was based on the notion that DICs primary function and resources (personnel and available drug information sources) are directed to answer questions concerning drugs. Thus, based on a sequence of selection criteria (publication or not, randomisation to 10% and the final exclusion), 842 QAPs were included in the study. They were analysed for the number of drugs involved in the question, complexity, the type of literature search performed, use of references (the number of references used and/or use of the RELIS database as a reference) and the time needed for providing an answer (in days). The number of drugs involved in the queries were categorised as one, two or three or more. Questions concerning groups of drugs, for example, antipsychotics, were categorised as three or more drugs. Judgemental and/or patient-related questions were defined as complex. Factual questions, such as the therapeutic dose of a drug or its half-life, can easily be located in textbooks, monographs or databases. Judgemental questions require by definition the integration of data or knowledge and experience in the process of making a decision regarding a specific therapeutic problem.6 The original definition assumed that answers to judgemental questions could not be given in any single reference source, but this assumption has been excluded in the present and other studies.7 Judgemental questions are frequently patient-related because they involve drug information applied to a clinical situation associated with patient-specific characteristics. They can, however, also represent a more general drug-related problem. If the question was answered without searching the literature and without consulting colleagues, the literature search was categorised as none. If it was necessary to search the RELIS database, databases containing monographs like the Micromedex, the Summary of Product Characteristics for the drug, reference books and/or colleagues/other health professionals only, the search was categorised as simple. If searches in databases like Medline, Embase or Cochrane to obtain original articles were necessary, the search was categorised as advanced. The number of questions received each year were analysed by use of the statistical function in the database. This function allows RELIS to quantify the activity in the centres (eg, the number of questions from physicians or pharmacists from a particular county from a particular time period to a particular RELIS can be retrieved). The number of employees in the centres in the respective periods was estimated but did not include all cases of maternity leave, leave of absence and vacant positions. The mean number of questions per employee per year was calculated from data corresponding to the start (year) and end (year) of each period. Figure 1 summarises the present RELIS model together with the study sample and design.


The data were analysed using SPSS V.17.0 (SPSS Inc). Proportions between groups were analysed using χ2 comparisons. Discrete data between groups were analysed using Mann–Whitney U test. p Values <0.05 were considered significant.


Table 1 shows characteristics of a randomised sample of 842 QAPs from 1995 to 2000 (n=312) and 2001–2006 (n=530), respectively, from the RELIS public database. The mean number of questions per employee per year increased from 66 to 89 from the first to the second period.


This paper describes a QAP database integrated with websites for RELIS in Norway. The model is characterised by a traceable answering process of questions submitted to the centres, which are stored in full text in a searchable database. Continuous growth of the database makes it a valuable source of drug information for healthcare professionals. Our analysis of two periods after RELIS started demonstrates the potential of the database to be an effective reference source through re-use of previous answers. Furthermore, RELIS represents a model for DICs where the staff efficiently answers complex questions through cooperation in a network.

The RELIS model includes quality standards and in-house training which ensures that the process of answering questions is similar across the centres. The intranet contains several guidelines on how to handle QAPs (eg, how to write references, how to index in the database). Furthermore, each centre uses the same commercial drug information databases and textbooks as sources. How to answer a question is discussed within each RELIS by informal discussions or use of co-signature. It is also discussed between different centres due to the transparency and traceability of the model and regularly in the annual national meetings where all centres participate. Thus, a person trained and located in one centre can do work for a different centre through this model in case of shortage of staff or increased workload in one of the regions. In this way, the individual centres are effective backups for each other. The ‘answer in progress’ function facilitates communication about new questions. Similar questions submitted simultaneously or subsequently can be answered based on previous work and communication between individual centres. In this way, unnecessary double work can be reduced and complex questions asked frequently can be answered within a short time frame.

Our data show that the re-use of the RELIS database as a reference source in the answering of new questions was significantly higher in 2001–2006 than in 1995–2000. This is not surprising, as the number of available QAPs in the second time period was higher and the chance of finding relevant information increases with the number of questions received at RELIS. Re-use of the database as a source of information is expected to decrease the workload of repeated questions, but patient-specific characteristics of relevance to the answer process still has to be taken into account. Furthermore, complex questions not answered previously are associated with increased workload. This is illustrated by the finding that the average number of references in the QAPs did not decrease from the first to the second period and advanced search strategies were still used in most cases. We recently performed a prospective study where the extent of literature search was the individual factor best predicting the time spent answering questions in RELIS.8 However, in RELIS, repeated questions are subjected to a new literature search if 3 months have passed since a previous answer was provided, and this was more likely to happen in the period 1995–2006. With repeated questions today, a previous answer in the database is more likely to be updated and could be re-used. An illustrating example is the common question of antidepressants and pregnancy, which is usually patient-related and judgemental in its form. This type of question is frequently submitted to RELIS because information in product monographs is categorical and not useful for solving a clinical situation with a depressed pregnant patient in need of drug therapy. A recent answer from the database with updated literature search can be provided within much shorter time limits than would be expected.

Interestingly, the answer time has been reported to be the parameter associated with dissatisfaction with DICs among physicians. A DIC in Denmark reported a median time to answer (final answer in contrast to preliminary answer) of 25 days among satisfied physicians and 56 days among unsatisfied physicians.9 In our material, the median time before a written answer was submitted to the questioner was 2 days and 1 day in the periods 1995–2000 and 2001–2006, respectively. In the subgroup of judgemental and patient-related QAPs, the corresponding reduction was 4 to 2 days, and 2 to 1 day, respectively (data not shown). The average number of days before a written answer was given was reduced from 8 to 5 days in the two periods. These figures are influenced by a few extreme outliers (>100 days before an answer were documented as given). Furthermore, ‘not important’ is an option when healthcare professionals specify their deadline for the answer. Some questions are dependent on documentation from external sources (eg, pharmaceutical industry) not controlled by RELIS. Thus, the median number of days before an answer is given is probably more representative for the service than the average. However, the answer time depends greatly on how fast the questioner needs his answer and how many other questions that are received in the same time period. Increased staffing could also influence answer time. However, our calculation of the mean number of questions per employee per year does not account for all the increase in activity during the period after RELIS started. Several questions answered by telephone were not documented in the database, and in 2003, the centres increased the number of employees mainly to handle adverse drug reaction reports due to new tasks in pharmacovigilance. In the prospective study, we measured the time spent to answer each of 96 questions.8 The mean time needed to provide an answer was 4.0 h, whereas the median time was 3.0 h (range 0.5–24 work hours). This represents effective time, not the number of workdays (including other work and lunch) recorded in the present material. Interestingly, in the same study we also observed a minor impact of the number of drugs on answer time. This could be due to the experience that questions about drug interactions are managed effectively by use of drug interaction software.8

The number of patient-related questions was significantly higher in the second time period than in the first one, as were the number of judgemental questions. These two variables are related to each other in such a way that patient-related questions often are associated with judgemental considerations and vice versa. In the present study, they were used to indicate complex questions. In an earlier study, we found that more than 60% of the queries were judgemental and most of the queries were patient-related.8 The sample in that study was much smaller compared with the present results, but the numbers corresponds well. However, the assessment of judgemental/non-judgemental is quite subjective, and others have used the term consultative for questions that requires clinical advice on a special case. Consultative questions usually entail discussion with the inquirer on possible benefits and hazards of one or more courses of action in a clinical case.10 The justification for using judgemental classification versus consultative and other classifications is not clear-cut, and this makes comparisons between different studies difficult. However, DICs in Norway are similar to the Swedish, which have reported a particularly high proportion of consultative queries.11 During the history of RELIS, the number of questions from physicians has steadily increased in proportion to other groups of healthcare professionals. In 2011, 65% of 2565 questions came from physicians (40% of physicians from general practice), while 84% were patient-related.

Adverse effects and pregnancy represent the two most common categories of questions to RELIS and are a substantial part of the QAP database. Feedback to healthcare professionals who submit adverse drug reaction reports commonly involve use of this database. Furthermore, several reports were originally generated by a question by a physician concerning possible adverse effects. Thus, the database and the websites are strongly integrated with the pharmacovigilance work. Our answers have been evaluated to be of high quality and of significant clinical impact in several studies.1–3 These studies cover the period of 1995–1998 and 2006. Interestingly, the study from 2006 concerned drug information in pregnancy, which represents a category of questions where re-use of information in the database is particularly frequent.3 However, such evaluations are performed by a selected group of physicians and cannot be generalised to other healthcare professionals. Although consequences on a patient level are usually not known in these results, a particularly relevant finding was that almost 10% of physicians who evaluated the service reported that the information prevented unnecessary termination of a pregnancy.3

The present model could be of international interest for existing or planned network of DICs. A recent survey of 75 DICs in the USA found an increase in the number of complex questions in 70%, while 53% reported an increase in the time required to answer each question.12 A shared database represents a useful tool for uniform handling and documentation of complex questions. Furthermore, it becomes an increasingly important drug information source for healthcare professionals as well as employees in the network. Through a solution with web-based communication, the model could be of particular importance in poor and undeveloped countries and continents with frequent epidemics, outbreaks and health problems where rapid dissemination of drug information is essential.


We conclude that RELIS represents a model with a potential to efficiently answer complex questions. The model is of relevance for organisation of DICs.


To cite: Schjøtt J, Reppe LA, Roland P-DH, et al. A question–answer pair (QAP) database integrated with websites to answer complex questions submitted to the Regional Medicines Information and Pharmacovigilance Centres in Norway (RELIS): a descriptive study. BMJ Open 2012;2:e000642. doi:10.1136/bmjopen-2011-000642

Contributors: JS planned the manuscript and wrote it. LAR collected and analysed the material from the RELIS database. PDHR provided historical and technical data on the RELIS model. TW and JS designed the figure. JS designed the table. All authors revised the manuscript for important intellectual content and discussed drafts towards approval of the final version.

Funding:This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests: None.

Provenance and peer review: Not commissioned; externally peer reviewed.

Data sharing statement: All QAPs used in this study were obtained from the public RELIS database, which is searchable in Norwegian for healthcare professionals free of charge. QAPs from the database were compared with the information in the associated paper version (all documents and information on handling of the question) saved in archives in the respective RELIS centre.

We would like to thank all RELIS staff (past and present) who contributed to development of this model from 1995 to the present by providing drug information to healthcare professionals.

1. Schjøtt J,Pomp E,Gedde-Dahl A,et al. What do health professionals ask RELIS Vest service and how satisfied are they with the answers? (In Norwegian). J Norw Med AssocYear: 2000;120:204–7
2. Schjøtt J,Pomp E,Gedde-Dahl A. Quality and impact of problem-oriented drug information: a method to change clinical practice among physicians?Eur J Clin PharmacolYear: 2002;57:897–90211936710
3. Frost Widnes SK,Schjøtt J. Drug use in pregnancy-physicians' evaluation of quality and clinical impact of drug information centres. Eur J Clin PharmacolYear: 2009;65:303–819052735
4. Öhman B,Lyrvall H,Törnqvist E,et al. Clinical pharmacology and the provision of drug information. Eur J Clin PharmacolYear: 1992;42:563–71623895
5. Troger U,Meyer FP. The regional drug-therapy consultation service centre-a conception that has been serving patients and physicians alike for 30 years in Magdeburg (Germany). Eur J Clin PharmacolYear: 2000;55:707–1110663447
6. Grace M,Wertheimer AI. Judgmental questions processed by a drug information center. Am J Hosp PharmYear: 1975;32:903–41190246
7. Merritt GJ,Garnett WR,Eckel FM. Analysis of a hospital-based information center. Am J Hosp PharmYear: 1977;34:42–6831474
8. Reppe LA,Spigset O,Schjøtt J. Which factors predict the time spent answering queries to a drug information centre?Pharm World SciYear: 2010;32:799–80420922480
9. Hedegaard U,Damkier P. Problem-oriented drug information: physicians' expectations and impact on clinical practice. Eur J Clin PharmacolYear: 2009;65:515–2219137281
10. Davies DM,Ashton CH,Rao JG,et al. Comprehensive clinical drug information service: first year's experience. Br Med JYear: 1977;1:89–90832025
11. Llerena A,Öhman B,Alván G. References used in a drug information centre. Eur J Clin PharmacolYear: 1995;49:87–98751027
12. Rosenberg JM,Schilit S,Nathan JP,et al. Update on the status of 89 drug information centers in the United States. Am J Health Syst PharmYear: 2009;66:1718–2219767377


[Figure ID: fig1]
Figure 1 

Healthcare professionals with questions can search the Regional Medicines Information and Pharmacovigilance Centres (RELIS) public database or submit a web question form through the RELIS homepage. The questions with deadline and demographics are distributed to all RELIS staff, who have access to the question–answer pairs (QAPs) throughout their processing in the RELIS internal domain. All QAPs are stored in the RELIS internal database and selected for publication in the RELIS public database. A randomised sample of QAPs from 1995–2000 to 2001–2006 were retrospectively analysed and compared with regard to number of drugs involved (one, two, three or more), complexity (judgemental and/or patient-related or not), literature search (none, simple or advanced) and answer time (in days).

[TableWrap ID: tbl1] Table 1 

Characteristics of a randomised sample of 842 question–answer pairs in the RELIS public database 1995–2006

Period 1995–2000 2001–2006 p Value
Questions, n 312 530
 Drugs involved, category: % One: 49 One: 45 0.434§
Two: 19 Two: 22
Three or more: 32 Three or more: 34*
 Judgemental questions, n (%) 207 (66) 399 (75) 0.005§
 Patient-related questions, n (%) 170 (54) 379 (72) <0.001§
Answers, n 312 530
 Literature search, type: % None: 3 None: 2 0.315§
Simple: 43 Simple: 40
Advanced: 54 Advanced: 58
 Reference to RELIS database, n (%) 39 (13) 164 (31) <0.001§
 References, median (average) 3 (4) 4 (5) 0.100
 Days, median (average) 2 (8.4) 1 (5.0) 0.001

*The total number exceeds 100% because of rounding off to the nearest whole number.

Simple literature search included monographs (eg, Martindale, Summary of Product Characteristics), advanced literature search included articles (eg, databases like Medline and Embase).

Answer time is influenced by some extreme outliers (>100 days) and choice of deadline for answer (‘not important’ is an option).

§χ2 Test for contingency tables.

Mann–Whitney U test for two independent samples.

Article Categories:
  • Pharmacology and Therapeutics
    • Research
Article Categories:
  • 1506
  • 1684
  • 1702
  • 1704
  • 1723

Previous Document:  A serial qualitative interview study of infant feeding experiences: idealism meets realism.
Next Document:  Non-response in a nationwide follow-up postal survey in Finland: a register-based mortality analysis...