The process of evaluating sources can take place when you first encounter a source, when you're reading it over, and as you incorporate it into your project. In general, some level of evaluation should take place at all of these stages, with different goals for each. The last evaluation will be discussed in "Organizing the Review," but we'll go over the first two here.
The overall purpose of evaluating sources is to make sure that your review has the most relevant, accurate, and unbiased literature in the field, so that you can determine what has already been learned about your topic and where further research may be needed.
Asking the Right Questions by When you first encounter a potential source, you'll want to know very quickly whether it is worth reading in detail and considering for your literature review. To avoid wasting time on unhelpful sources as much as possible, it's generally best to run each article, book, or other resource you find through a quick checklist, using information you can find by skimming through the summary and introduction.
The two most common forms of early source evaluation are the "Big 5 Criteria" and the "CRAAP Test." These cover the same most significant variables for evaluation, and which one to use comes down to preference.
Big 5 Criteria:
The most important criteria for evaluating a potential resource are:
A good rule of thumb for Currency is that medical, scientific, and technology resources should be published within the last 5 years to prevent the information from being out-of-date; for less time-sensitive topics like history or the humanities, resources published within the last 5-10 years are often acceptable.
CRAAP Test:
A helpful mnemonic to remember the evaluation criteria, CRAAP is an acronym for:
Helpful questions for initial evaluation:
Once a resource has passed the initial evaluation, you are ready to begin reading through it to more carefully determine if it belongs in your project. In addition to the questions posed above, which are always relevant to evaluating sources, you should look at your potential sources of literature with an eye to the following questions:
1. Is there any bias visible in the work?
You already began this process in the previous step and hopefully eliminated the most obviously unreliable sources, but as you read it is always important to keep an eye out for potential blind spots the author might have based on their own perspective. Bias is not inherently disqualifying -- a biased article may still have accurate information -- but it is essential to know if a bias exists and be aware of how it might impact how the information was gathered, evaluated, or delivered.
Peer-reviewed sources tend to be less likely to have this risk, because multiple editors had to go through the resource looking for mistakes or biases. They need to meet a much higher academic threshold.
2. How was the research conducted? Are there any strengths or weaknesses in its methodology?
It is important to understand how the study in your source was administered; a significant part of the literature review will be about potential gaps in the current research, so you need to understand how the existing research was done.
3. How does the author justify their conclusions?
Either through the results of their own research or by citing external evidence, an article, book, or other type of resource should provide proof of its claims. In the initial research process you checked to make sure that there was evidence supporting the author's assertions; now it is time to take a look at that evidence and see if you find it compelling, or if you think it doesn't justify the conclusions drawn by the article.
4. What similarities do these articles share?
Grouping your literature review by categories based on subtopic, findings, or chronology can be an extremely helpful way to organize your work. Take notes while you're reading of common themes and results to make planning the review easier.
5. Where does this research differ from the other sources?
While very close to the previous question, this one emphasizes what unique information, methodology, or insights a particular source brings to the overall understanding of the topic. What new knowledge is being brought to the table by this source that would justify it appearing in your literature review?
6. Does this source leave any unanswered questions or opportunities for further research?
Most scholarly journal articles will include a section near the end of the article addressing limits of their study and opportunities for further research. Examine these closely and see where other sources you have fill in the gaps, and where perhaps additional research needs to be done to gain a more complete understanding of your topic.
As you go through this process, you might find yourself eliminating certain sources that no longer seem like they fit with your project's goals, or getting inspired to search for additional sources based on new information you've found. This is a normal part of the research process, and additional searching may be necessary to fill in any gaps left after evaluating the sources you already have.
