For all types of sources:
- Author - are they an expert? An expert in what?
- Publication - who is the audience? How does audience affect what gets published?
- Relevancy
- Currency
- Citations / references: whom do they cite? Are those reliable sources?
- What else have they written and where was it published?
For scholarly conversations, specifically:
- Why do folks want to get published in these journals?
- job concerns - getting a job and keeping it
- participating in the scholarly conversation - journals are the only place it happens
- the research in journals is the most up to date, the most cutting edge, the newest knowledge on a topic - it's how the field is advanced
- Pay attention to funding - who is paying for this research and are there conflicts of interest? Can the author benefit financially from this research, from his or her claims being accepted?