One of the distinguishing characteristics of skeptics.SE is that we expect answers to be referenced preferably to peer reviews scientific publications. But do we also need to differentiate among good science and sloppy science? To express this point another way, can we trust all peer reviewed science equally well?
As an example that points to the nature of the problem, I recently posted a (now closed) question based on some quotes from James Lovelock where he alleged that as much as 80% of the published work on CFC science (and possibly also climate science) was sloppy or fraudulent: http://skeptics.stackexchange.com/questions/14602/does-the-quality-of-published-scientific-research-decline-when-there-is-a-consen. There are issues with the way I posed the question (not least because I attached a reason to Lovelock's claimed effect) but it is clear that Lovelock thinks the majority of the publications in some areas are rubbish.
Lovelock isn't the only person to raise the issue. Attempts to replicate much of the medical literature often result in low success rates and the whole literature reporting clinical trials of pharmaceuticals is systematically biased by selective publication (see Ben Goldacre's book Bad Pharma).
I'm going to attempt to redo the question on the main site, but the point of raising the issue here is that it seems to me to pose a very significant problem for skeptics.SE. If a large proportion of the literature is sloppy, fraudulent or wrong, how does the community differentiate among well-referenced answers quoting dodgy science and well-referenced answers quoting good science?
I don't have any good suggestions other than to somehow improve the weight of answers using meta-analyses. I'd be interested to see whether others think the problem is serious and whether there are other ways this community could deal with it.
I came across this article from The New York Times and though it worth including as it gives a remarkably clear insight into why some published types of scientific study produce less reliable results than others (and why poor reporting of the differences results in many misleading newspaper and media stories). In fact it would almost be worth including is on skeptics.SE as standard advice for how to report scientific results. The article argues (my emphasis):
R.C.T.s [randomised controlled trials] are often very difficult to set up properly and can take many years to carry out. As a result, most research we read about involves just correlational studies. John Ioannidis, in a series of highly regarded analyses, has shown that, in published medical research, 80 percent of non-randomized studies (by far the most common) are later found to be wrong. Even 25 percent of randomized studies and 15 percent of large randomized studies — the best of the best — turn out to be inadequate...
Media tend to present almost any scientific result they report as valuable for guiding our lives... Too many news reports present experimental results as providing good advice on which we can reliably act. In most cases those results would be better viewed as mistakes pointing to a next step that will be a bit less mistaken.
Science reporting would be much improved if we had a labeling system that made clear a given study’s place in the scientific process. Is it merely a preliminary result (a small-scale heuristic study meant to suggest a hypothesis that will itself require many stages of further testing before we have a reliable conclusion)? Is it a larger-scale observational study (showing a correlation but by no means establishing a causal connection)? Is it a large-sample randomized controlled test (establishing a causal connection, given specific conditions)? Or, finally, is it a well-established scientific law that we know how to apply in a wide range of conditions?
Paying attention to this (crude) classification would greatly improve many answers here.