Skip navigation

Category Archives: Qualitative Research

Article by King, William R., and He, Jun (2005) in Communications of the AIS, 16.

 

There are four methods in conducting literature review, i.e. Narrative Review, Descriptive Review, Vote Counting and Meta-Analysis. Look at the diagram at the end of this entry to view these methods in the qualitative-quantitative continuum.

 

Meta-analysis is considered as the most rigor method; it is the one which closest to the positivist tradition. Here are the advantages of meta-analysis:

 

  1. The major difference between narrative reviews and quantitative meta-analyses may well be that narrative reviews primarily focus on the conclusions reached in various studies, whereas meta-analyses focus on data, as reflected by the operationalization of variables, the magnitude of effect sizes, and the sample sizes.
  2. Compared to narrative reviewing, meta-analysis procedure is more standardized, less judgmental and less subjective (narrative reviewing is more towards interpretive in nature, thus conclusions drawn are very subjective and dependent to the reviewer’s knowledge and expertise).
  3. Meta-analysis draw conclusion from the data characteristics (i.e. the operationalization of variables, the magnitudes of effect sizes and the sample size) of several studies; whereas traditional methods are more focusing on the end result (conclusions) only. Viewing from this point, meta-analysis result is generated from the richer data; therefore it offers more meaningful and robust findings.
  4. Meta-analysis enables researchers to sample studies that show insignificant effects (note that in traditional methods, researchers are bound to select empirical studies that report significant effects only). With the powerful technique of meta-analysis method, studies with insignificant effects will be analyzed along with others that may at the end show significant effects. The end result is, of course, not necessarily significant, but one thing for sure, it is undoubtedly more accurate and more credible due to the sampling (of studies) that is not bias (to studies with significant effects only).
  5. On the same basis as point #4 above, meta-analysis enables researchers to estimate the cumulative impact of insignificant results that may in turn at the end be significant.
  6. Meta-analysis enables and promotes researchers to search for moderator variables in the less subjective way.

 

Despite the advantages listed above, it has its own limitations:

 

  1. Sampling Bias toward Empirical Studies – Meta-analysis is applicable to quantitative studies (such as survey, laboratory experiment, field study, and field experiment) that report the magnitude of the effect size only. Other type of studies such as mathematical modeling, frameworks and conceptual models, case studies, interview, speculation/commentary, secondary data, and qualitative research, are unlikely to be sampled in a meta-analysis study. Even if mixed-method studies are being sampled, meta-analysis will be based their findings on the quantitative part only.
  2. Garbage In, Garbage OutMeta-analysis does not generally differentiate studies by their quality (quality of research designs and approaches, sampling units, methods of measuring variables, data analysis methods, and presentations of research findings are not in consideration when sampling is carried out). This may lead meta-analysis to unfounded conclusions. Although some techniques have been introduced to correct this error, it is evident that application of such techniques had introduced other biases (related to the selection and weighting of quality criteria).
  3. Publication Bias – It is normal in the current practice that studies with significant effects are more likely to be published while insignificant studies are tend to be ‘reserved’. This phenomenon may leads to the so-called ‘unrepresentative sampling’, thus conclusion drawn may bias toward certain direction of significance.
  4. Apples and Oranges – All reviewing methods, not restricted to meta-analysis only, are based on the collection of past studies that have common characteristics. However, to find studies with ‘exact replications’ and ‘precisely the same’ are almost impossible. So what researchers can do is to select studies that ‘almost’ the same, and from there, they draw conclusions – this is analogous to taking apples and oranges and averaging such measures as their sizes, weights, tastes, etc.
  5. Small Sample Sizea meta-analysis should include at least 15 studies, otherwise the type I error (accepting a false null hypothesis) could be severely inflated. As point #4 above, it is very difficult to find 15 empirical studies that are the same!!

 

The statistics that normally used and reported in IS studies are: (i) effect size; (ii) sample size; (iii) possible moderators; (iv) construct reliabilities, and; (v) other measurement error indexes.

 

In the literature, effects sizes are of various forms. Look at the diagram at the end of this entry to view them.

 

There are two models of meta-analysis method available. Look at the diagram at the end of this entry to view them.

 

There are three primary meta-analysis methods available, i.e. (i) Hedges-Olkin’s method; (ii) Rosenthal-Rubin’s method, and; (iii) Hunter-Schmidt’s method. There is another method introduced in the IS domain, named Glass method. This paper does not provide complete procedure on each method, but the authors suggested readers to refer to Lipsey and Wilson (2001) for the thorough description of the whole process.[i]

Literature Review Methods

Literature Review Methods

Types of Effect Sizes

Types of Effect Sizes

Meta-Analysis Models

Meta-Analysis Models


[i] Lipsey, M.W., and Wilson, D.B. (2001) Practical Meta-analysis, Thousand Oaks, CA: Sage Publications.

Article by Long, Andrew F., Godfrey, Mary (2004) in IJSRM, 7:2.

 

Two of my colleagues from the Faculty of ice ——ment recently called me and asked me what constitute a good research in social science. No doubt, as a supervisor for a final year research project, they have to know precisely all the facets and dimensions that they have to look when they assess their students’ research. On top of that, they have a very great responsibility to guide the students on how to do the right research right. One thing for sure, they have to always be in hundred steps ahead of their students and in the position where they capable of seeing the whole tree within which one of the small leafs are their students’ research scope. I wish my two colleagues (who are actually best friends of mine) a very good luck in performing their great and noble tasks. For them and thousands of other supervisors, this paper may help them in guiding and assessing their students’ research.

 

As reflected by the title, the core contribution of this paper is the development of an evaluation tool to assess the quality of qualitative research. ‘Qualitative research’ in this paper is defined as ‘one that uses qualitative methods in both the gathering and analysis of the data, that is, visual and verbal (conceptual or thematic) rather than numerical data manipulation… using methods such as interviewing, focus groups, ethnography, participant observation, documentary analysis and life histories’.

 

The evaluation tool developed in this paper consists of the following elements:

 

Review Area

Key Questions

 

 

(1)   Phenomenon Studied and Context

 

 

1.1   Phenomena under study

What is being studied? Is sufficient detail given of the nature of the phenomena under study?

 

 

1.2   Context 1

 

        1.2.1   Theoretical

What theoretical framework guides or informs the study? In what ways is the framework reflected in the way the study was done?

        1.2.2   Framework

How do the authors locate the study within the existing knowledge base?

 

 

1.3   Context 2 – Setting

Within what geographical and care setting is the study carried out? [Note: please refer to the original paper for the full list of the key questions]

 

 

1.4   Context 3 – Sample

How is the sample selected? Is the sample appropriate to the aims of the study? [Note: please refer to the original paper for the full list of the key questions]

 

 

1.5   Context 4 – Outcome

What outcome criteria are used in the study? [Note: please refer to the original paper for the full list of the key questions]

 

 

(2)   Ethics

[Note: please refer to the original paper for the full list of the key questions]

 

 

(3)   Data Collection, Analysis and Potential Researcher Bias

 

 

3.1   Data Collection

What data collection methods are used to obtain and record the data? Is the information collected…[Note: please refer to the original paper for the full list of the key questions]

 

 

3.2   Data Analysis

How are the data analysed? [Note: please refer to the original paper for the full list of the key questions]

 

 

3.3   Potential Bias

[Note: please refer to the original paper for the full list of the key questions]

 

 

(4)  Policy and Practice Implications

To what setting are the study findings generalizable? [Note: please refer to the original paper for the full list of the key questions]

 

 

 

Note: readers are advised to read the whole paper as few concepts underlying the assessment aspects (such as adequacy and appropriateness) are crucial to be understood thoroughly prior to implementation.

 

 

 

Follow

Get every new post delivered to your Inbox.

Join 29 other followers