Article by King, William R., and He, Jun (2005) in Communications of the AIS, 16.
There are four methods in conducting literature review, i.e. Narrative Review, Descriptive Review, Vote Counting and Meta-Analysis. Look at the diagram at the end of this entry to view these methods in the qualitative-quantitative continuum.
Meta-analysis is considered as the most rigor method; it is the one which closest to the positivist tradition. Here are the advantages of meta-analysis:
-
The major difference between narrative reviews and quantitative meta-analyses may well be that narrative reviews primarily focus on the conclusions reached in various studies, whereas meta-analyses focus on data, as reflected by the operationalization of variables, the magnitude of effect sizes, and the sample sizes.
-
Compared to narrative reviewing, meta-analysis procedure is more standardized, less judgmental and less subjective (narrative reviewing is more towards interpretive in nature, thus conclusions drawn are very subjective and dependent to the reviewer’s knowledge and expertise).
-
Meta-analysis draw conclusion from the data characteristics (i.e. the operationalization of variables, the magnitudes of effect sizes and the sample size) of several studies; whereas traditional methods are more focusing on the end result (conclusions) only. Viewing from this point, meta-analysis result is generated from the richer data; therefore it offers more meaningful and robust findings.
-
Meta-analysis enables researchers to sample studies that show insignificant effects (note that in traditional methods, researchers are bound to select empirical studies that report significant effects only). With the powerful technique of meta-analysis method, studies with insignificant effects will be analyzed along with others that may at the end show significant effects. The end result is, of course, not necessarily significant, but one thing for sure, it is undoubtedly more accurate and more credible due to the sampling (of studies) that is not bias (to studies with significant effects only).
-
On the same basis as point #4 above, meta-analysis enables researchers to estimate the cumulative impact of insignificant results that may in turn at the end be significant.
-
Meta-analysis enables and promotes researchers to search for moderator variables in the less subjective way.
Despite the advantages listed above, it has its own limitations:
-
Sampling Bias toward Empirical Studies – Meta-analysis is applicable to quantitative studies (such as survey, laboratory experiment, field study, and field experiment) that report the magnitude of the effect size only. Other type of studies such as mathematical modeling, frameworks and conceptual models, case studies, interview, speculation/commentary, secondary data, and qualitative research, are unlikely to be sampled in a meta-analysis study. Even if mixed-method studies are being sampled, meta-analysis will be based their findings on the quantitative part only.
-
Garbage In, Garbage Out – Meta-analysis does not generally differentiate studies by their quality (quality of research designs and approaches, sampling units, methods of measuring variables, data analysis methods, and presentations of research findings are not in consideration when sampling is carried out). This may lead meta-analysis to unfounded conclusions. Although some techniques have been introduced to correct this error, it is evident that application of such techniques had introduced other biases (related to the selection and weighting of quality criteria).
-
Publication Bias – It is normal in the current practice that studies with significant effects are more likely to be published while insignificant studies are tend to be ‘reserved’. This phenomenon may leads to the so-called ‘unrepresentative sampling’, thus conclusion drawn may bias toward certain direction of significance.
-
Apples and Oranges – All reviewing methods, not restricted to meta-analysis only, are based on the collection of past studies that have common characteristics. However, to find studies with ‘exact replications’ and ‘precisely the same’ are almost impossible. So what researchers can do is to select studies that ‘almost’ the same, and from there, they draw conclusions – this is analogous to taking apples and oranges and averaging such measures as their sizes, weights, tastes, etc.
-
Small Sample Size – a meta-analysis should include at least 15 studies, otherwise the type I error (accepting a false null hypothesis) could be severely inflated. As point #4 above, it is very difficult to find 15 empirical studies that are the same!!
The statistics that normally used and reported in IS studies are: (i) effect size; (ii) sample size; (iii) possible moderators; (iv) construct reliabilities, and; (v) other measurement error indexes.
In the literature, effects sizes are of various forms. Look at the diagram at the end of this entry to view them.
There are two models of meta-analysis method available. Look at the diagram at the end of this entry to view them.
There are three primary meta-analysis methods available, i.e. (i) Hedges-Olkin’s method; (ii) Rosenthal-Rubin’s method, and; (iii) Hunter-Schmidt’s method. There is another method introduced in the IS domain, named Glass method. This paper does not provide complete procedure on each method, but the authors suggested readers to refer to Lipsey and Wilson (2001) for the thorough description of the whole process.[i]
Literature Review Methods
Types of Effect Sizes
Meta-Analysis Models
[i] Lipsey, M.W., and Wilson, D.B. (2001) Practical Meta-analysis, Thousand Oaks, CA: Sage Publications.
RESEARCH METHODOLOGY 