Skip navigation

Tag Archives: Research Methodology

Article by Rumrill, Phillip  D. Jr., and Fitzgerald, Shawn M. (2001) in Work, 16:2.


This paper describes one of the literature review method that is most interpretive-qualitative in nature, i.e. the narrative review [i]. Although the discussion is very specific to the area of rehabilitation research (one research area in education), I believe that the main points are also applicable to other fields in social and design science research [ii], especially IS research [iii].


I collect all the important points of discussion, and put them here without reference to the specific field where this paper is originally based on.


In the literature, researchers used narrative literature review to: (i) describe the current states of both art (i.e., practice) and science (i.e., research) in focused areas of inquiry; (ii) add dimensions of insight or application that are not available in existing literature, and; (iii) provide critical analyses of standing works. The results of the review are normally in the form of: (i) postulate or proposition of advance new theories and models; (ii) arguments of important and/or controversial topics; (iii) present of the “how to” tips and strategies to improve best practices, and; (iv) description of the latest developments in the field (present new perspectives pertains to the important and emerging issues).


General guideline to conduct narrative literature review:

  1. Identify a research area;
  2. Identify inclusion criteria for studies;
  3. Select studies that meet the inclusion criteria;
  4. Identify themes that emerge from the set of studies, and;
  5. Draw conclusions.


Limitations of narrative literature review:

  1. The nature of the method that is too subjective (in the determination of which studies to include, the way the studies are analyzed, and the conclusions drawn);
  2. The possibility of misleading in drawing conclusions (that are normally due to selection bias, subjective weighing of the studies chosen for the review, unspecified inclusion criteria, and failure to consider the relationships between study characteristics and study results, and;
  3. The problem in determining and integrating complex interactions (that may exist) when large set of studies is involved.


[i] Click here for the full list of literature review methods.

[ii] Until now, many of my colleagues are not in the know of what is actually termed as ‘design science’. Many of them asked me what the differences between ‘social science’ and ‘design science’ are. Why should we have ‘design science’ after the establishment and the long history of ‘social science’? For them, here is the answer.

[iii] Social science and behavioral research describe a significant proportion of all IS research (Straub, Boudreau & Gefen (2004, p.383)).





Article by King, William R., and He, Jun (2005) in Communications of the AIS, 16.


There are four methods in conducting literature review, i.e. Narrative Review, Descriptive Review, Vote Counting and Meta-Analysis. Look at the diagram at the end of this entry to view these methods in the qualitative-quantitative continuum.


Meta-analysis is considered as the most rigor method; it is the one which closest to the positivist tradition. Here are the advantages of meta-analysis:


  1. The major difference between narrative reviews and quantitative meta-analyses may well be that narrative reviews primarily focus on the conclusions reached in various studies, whereas meta-analyses focus on data, as reflected by the operationalization of variables, the magnitude of effect sizes, and the sample sizes.
  2. Compared to narrative reviewing, meta-analysis procedure is more standardized, less judgmental and less subjective (narrative reviewing is more towards interpretive in nature, thus conclusions drawn are very subjective and dependent to the reviewer’s knowledge and expertise).
  3. Meta-analysis draw conclusion from the data characteristics (i.e. the operationalization of variables, the magnitudes of effect sizes and the sample size) of several studies; whereas traditional methods are more focusing on the end result (conclusions) only. Viewing from this point, meta-analysis result is generated from the richer data; therefore it offers more meaningful and robust findings.
  4. Meta-analysis enables researchers to sample studies that show insignificant effects (note that in traditional methods, researchers are bound to select empirical studies that report significant effects only). With the powerful technique of meta-analysis method, studies with insignificant effects will be analyzed along with others that may at the end show significant effects. The end result is, of course, not necessarily significant, but one thing for sure, it is undoubtedly more accurate and more credible due to the sampling (of studies) that is not bias (to studies with significant effects only).
  5. On the same basis as point #4 above, meta-analysis enables researchers to estimate the cumulative impact of insignificant results that may in turn at the end be significant.
  6. Meta-analysis enables and promotes researchers to search for moderator variables in the less subjective way.


Despite the advantages listed above, it has its own limitations:


  1. Sampling Bias toward Empirical Studies – Meta-analysis is applicable to quantitative studies (such as survey, laboratory experiment, field study, and field experiment) that report the magnitude of the effect size only. Other type of studies such as mathematical modeling, frameworks and conceptual models, case studies, interview, speculation/commentary, secondary data, and qualitative research, are unlikely to be sampled in a meta-analysis study. Even if mixed-method studies are being sampled, meta-analysis will be based their findings on the quantitative part only.
  2. Garbage In, Garbage OutMeta-analysis does not generally differentiate studies by their quality (quality of research designs and approaches, sampling units, methods of measuring variables, data analysis methods, and presentations of research findings are not in consideration when sampling is carried out). This may lead meta-analysis to unfounded conclusions. Although some techniques have been introduced to correct this error, it is evident that application of such techniques had introduced other biases (related to the selection and weighting of quality criteria).
  3. Publication Bias – It is normal in the current practice that studies with significant effects are more likely to be published while insignificant studies are tend to be ‘reserved’. This phenomenon may leads to the so-called ‘unrepresentative sampling’, thus conclusion drawn may bias toward certain direction of significance.
  4. Apples and Oranges – All reviewing methods, not restricted to meta-analysis only, are based on the collection of past studies that have common characteristics. However, to find studies with ‘exact replications’ and ‘precisely the same’ are almost impossible. So what researchers can do is to select studies that ‘almost’ the same, and from there, they draw conclusions – this is analogous to taking apples and oranges and averaging such measures as their sizes, weights, tastes, etc.
  5. Small Sample Sizea meta-analysis should include at least 15 studies, otherwise the type I error (accepting a false null hypothesis) could be severely inflated. As point #4 above, it is very difficult to find 15 empirical studies that are the same!!


The statistics that normally used and reported in IS studies are: (i) effect size; (ii) sample size; (iii) possible moderators; (iv) construct reliabilities, and; (v) other measurement error indexes.


In the literature, effects sizes are of various forms. Look at the diagram at the end of this entry to view them.


There are two models of meta-analysis method available. Look at the diagram at the end of this entry to view them.


There are three primary meta-analysis methods available, i.e. (i) Hedges-Olkin’s method; (ii) Rosenthal-Rubin’s method, and; (iii) Hunter-Schmidt’s method. There is another method introduced in the IS domain, named Glass method. This paper does not provide complete procedure on each method, but the authors suggested readers to refer to Lipsey and Wilson (2001) for the thorough description of the whole process.[i]

Literature Review Methods

Literature Review Methods

Types of Effect Sizes

Types of Effect Sizes

Meta-Analysis Models

Meta-Analysis Models

[i] Lipsey, M.W., and Wilson, D.B. (2001) Practical Meta-analysis, Thousand Oaks, CA: Sage Publications.