Is our moderately low Answers-Per-Question ratio a sign of poor health?

Question

Observation: Skeptics.SE has historically had a moderately low ratio of answers-to-questions. [Ref - currently 'Okay', but has been 'Needs-Work']

Hypothesis: Skeptics.SE tend to have well-researched answers which require more effort than other sites, which tend to have multiple suggestions. A relatively low Answers-To-Questions ratio is to be expected, and not a good metric for the health of the Skeptics.SE.

Prediction: With less answers per question, but high-quality questions, we should get more Enlightened badges (10+ votes, first answer) than other StackExchanges web-site. With the alternative explanation - we are just poor at producing answers - we'd expect the answers given to not get 10+ votes.

Method: Get the number of Enlightened badges divided by the number of Questions for each of several StackExchange sites, until bored.

Results:

• Skeptics: 373 Enlightened Badges of 1877 Questions: 19.9%
• Gaming: 1038 of 10,963: 9.5%
• English: 794 of 11296: 7.0%
• TeX: 746 of 11506: 6.5%
• Programmers: 905 of 14118: 6.4%
• Game Developers: 226 of 5068: 4.4%
• Stack Overflow: 30953 of 2284146 : 1.4%
• Code Golf: 6 of 458: 1.3%

Conclusion: Skeptics.SE has a far higher rate of Enlightened Answers than other sites. This is consistent with the hypothesis that while Skeptics tends to get fewer answers than other Skeptics.SE sites, the average quality of them is higher.

Known shortcomings:

• Selection criteria not specified.
• Only statistical analysis is "inter-ocular impact".
• Alternate explanations?

UPDATE

In accordance with Sklivvz's request, a pretty new graph - click to enlarge - with a lot more data points - all the ones I could (conveniently) fetch through the API.

Skeptics is ranked #2 in the enlightenment stakes. A surprise result from SciFi there!

Here is the raw data. Each row represents the site's name, the number of answers per question and the number of enlightened answers per question.

 scifi 2.21 0.243
 skeptics 1.675 0.199
 german 2.777 0.176
 cstheory 2.015 0.134
 boardgames 2.805 0.096
 gaming 1.776 0.095
 photo 3.307 0.091
 rpg 3.672 0.09
 astronomy 2.004 0.083
 travel 2.199 0.082
 writers 3.701 0.078
 english 2.843 0.07
 parenting 4.049 0.07
 cooking 3.017 0.069
 gardening 1.868 0.067
 tex 1.813 0.065
 programmers 5.512 0.064
 japanese 2.118 0.057
 bicycles 3.528 0.055
 security 2.834 0.055
 gamedev 2.612 0.045
 productivity 3.76 0.045
 diy 2.183 0.043
 music 2.82 0.041
 math 1.733 0.035
 ux 3.813 0.035
 unix 1.951 0.035
 electronics 2.705 0.035
 pm 4.127 0.032
 fitness 2.199 0.032
 money 2.681 0.03
 dba 2.031 0.027
 quant 2.179 0.026
 philosophy 2.906 0.025
 physics 2.13 0.024
 graphicdesign 2.224 0.024
 homebrew 2.699 0.022
 stats 1.809 0.02
 android 1.629 0.019
 askubuntu 1.626 0.018
 webapps 1.819 0.017
 sqa 3.368 0.017
 gis 2.076 0.016
 codereview 1.92 0.016
 stackoverflow 2.152 0.014
 codegolf 6.902 0.013
 webmasters 1.888 0.012
 apple 1.867 0.012
 mechanics 1.796 0.011
 superuser 1.938 0.01
 judaism 2.235 0.009
 serverfault 2.102 0.008
 onstartups 3.503 0.008
 stackapps 1.304 0.006
 drupal 1.393 0.006
 wordpress 1.455 0.002
 sharepoint 1.615 0.002

Here is the python code that generated the data I used for the graph:

#!/usr/bin/env python

from stackauth import StackAuth
import stackexchange # Initially run against 1.1_4
import time


ALL_SITES = [
    # Grabbed sites from sites.py on py-stackexchange website.
    # Deleted meta sites.
    # Commented out sites that don't work for unknown (versioning?) reason.
    stackexchange.StackOverflow,
    stackexchange.ServerFault,
    stackexchange.SuperUser,
    stackexchange.WebApplications,
    stackexchange.Gaming,
    stackexchange.Webmasters,
    stackexchange.Cooking,
    stackexchange.GameDevelopment,
    stackexchange.Photography,
    stackexchange.StatisticalAnalysis,
    stackexchange.Mathematics,
    stackexchange.HomeImprovement,
    stackexchange.GIS,
    stackexchange.TeXLaTeX,
    stackexchange.AskUbuntu,
    stackexchange.PersonalFinanceandMoney,
    stackexchange.EnglishLanguageandUsage,
    stackexchange.StackApps,
    stackexchange.UserExperience,
    stackexchange.UnixandLinux,
    stackexchange.WordPress,
    stackexchange.TheoreticalComputerScience,
    stackexchange.Apple,
    stackexchange.RoleplayingGames,
    stackexchange.Bicycles,
    stackexchange.Programmers,
    stackexchange.ElectricalEngineering,
    stackexchange.AndroidEnthusiasts,
    stackexchange.OnStartups,
    stackexchange.BoardandCardGames,
    stackexchange.Physics,
    stackexchange.Homebrew,
    stackexchange.ITSecurity,
    stackexchange.Writers,
    #stackexchange.AudioVideoProduction,
    stackexchange.GraphicDesign,
    stackexchange.DatabaseAdministrators,
    stackexchange.ScienceFictionandFantasy,
    stackexchange.CodeReview,
    stackexchange.CodeGolf,
    stackexchange.QuantitativeFinance,
    stackexchange.ProjectManagement,
    stackexchange.Skeptics,
    stackexchange.FitnessandNutrition,
    stackexchange.DrupalAnswers,
    stackexchange.MotorVehicleMaintenanceandRepair,
    stackexchange.Parenting,
    stackexchange.SharePoint,
    stackexchange.MusicalPracticeandPerformance,
    stackexchange.SoftwareQualityAssuranceandTesting,
    stackexchange.JewishLifeandLearning,
    stackexchange.GermanLanguageandUsage,
    stackexchange.JapaneseLanguageandUsage,
    stackexchange.Astronomy,
    stackexchange.Philosophy,
    stackexchange.GardeningandLandscaping,
    stackexchange.Travel,
    stackexchange.PersonalProductivity,
    #stackexchange.Cryptography,
    #stackexchange.Literature,
    #stackexchange.SignalProcessing,
    #stackexchange.FrenchLanguageandUsage,
    #stackexchange.Christianity,
    #stackexchange.Bitcoin,
    ]

stack_auth = StackAuth()

print "(Site, Answer:Question Ratio, Enlightenment:Question Ratio)"

nicknames = []
answer_question_ratios = []
enlightened_question_ratios = []

for site_name in ALL_SITES:
    nickname = str(site_name).split(".")[1]
    site = stackexchange.Site(site_name)
    successful = False
    while not successful:
        try:
            stats = site.stats()

            all_badges = site.all_badges()

            enlightened_badge, = filter(lambda badge: badge.name=="Enlightened", all_badges)

            answer_question_ratio = float(stats.total_answers)/stats.total_questions
            enlightened_question_ratio = float(enlightened_badge.award_count)/stats.total_questions

            print "    (%r, %.3f, %.3f)," % (
                nickname,
                answer_question_ratio,
                enlightened_question_ratio,
                )
            successful = True
        except Exception as e:
            # Probably networking issue. Loop around again
            print e
            time.sleep(20) # Don't hammer the server

Answer 1

I see some various possible factors to consider:

Objectivity

From the chart, some of the sites with higher answers/question seem more "relative" compared to this site. I asked a question on parenting.SE and was practically reprimanded for asking for sources! Very different environment.

Opinions = answers on some SE variants; thus many are given.

---

Possible right answers

On others, it is more obvious that there are multiple ways to solve a given problem (programming, photography, [La]Tex formatting...). Here, rebuttals or confirmations are based on science, and there is often scientific consensus enough that it's not apparent that numerous answers are necessary.

---

Tendency to find the same sources

Because users sometimes hunt and find the same sources, multiple answers have seemed like tweaks in narrative commentary while repeating the exact same quotes/citations. An example of a time I thought this happened is the question, Do Russians drink as much alcohol as their stereotype suggests?

I add this only to say that sometimes I'm not sure that numerous answers implies health. They sometimes all say pretty much the same thing. (Another example).

---

User development

When I see deleted answers, they tend to share common characteristics:

• The user has 1 or 101 reputation (perhaps a bit more)
• The answer is a few sentences and anecdotal
• There is a comment from Sklivvz

As these users develop, if they stick around, hopefully the community will continue to be populated with interesting backgrounded people who will answer questions in different ways. For example:

• Larian's background had a huge impact in his ability to know how to answer: Is it possible to fly 4 helicopters anywhere in a country without official knowledge?.
• Some example of my own questions where my background knowledge helped me know where to look.

While everyone is held to citation standards, background knowledge/experience plays a huge part in how one approaches the questions, and hopefully a growing diversity of users will increase this ratio.

---

Moving forward

Some final thoughts:

• What about tweaking the "reprimand message" for bad answers? I don't follow how long it takes between Sklivvz's usual comment about someone needing sources and deletion. I wonder if the message could be changed to, "Can you try to find a source to support your answer and edit it with a citation?" I guess I wonder what these answerers think -- do they give up? Do they feel justified in their opinion and think they don't need a source? I don't know what happens, but most don't ever seem to get updated and just die. Changing this would help keep more answers alive and increase the ratio. I understand if efforts along these lines are unreasonable due to moderator energy taxation and the validity of the mandate to rtfm (where m = forum rules/FAQ).
• High-rep users waiting longer to answer questions. I admit that some questions just nag at me and I start digging in. I love researching and putting together comprehensive summaries. I already did this for personal decisions and having a site that encouraged it for other inquiries was like finding heaven. But, if a question exists with a high quality answer, why would a newer user answer it? And if new users don't answer questions because high-reppers have already done it, they aren't able to get the experience of writing them, getting critiqued, editing, and getting the point reward. I admit that this is speculation, but it could be frustrating/discouraging.
• Living with it. Maybe, as suggested in the question, the high enlightenment badge ratio means that the site is just different. Like I said, maybe it's just a different kind of "health." Questions get answered and answered well. Others don't feel any need to add new ones. It'd be interesting to have another discussion about what SE site "health" means? Users who stick around for longer than a question? Mostly [accepted] answered questions? Views per day? Google pagerank for a given search?

Answer 2

There is a tendency to criticize any answer that does not completely address the question. Because these are researched and referenced answers getting a complete answer takes a lot of work and research. That takes time. I have several times invested a few hours in researching an answer just to have someone else post the same thing(or very similar too) I was working on.

We could encourage partial answers with positive comments, up votes, and with a statement encouraging people to get involved and share their part of the answer. Though I am not certain this would lead to better answers I would expect it would lead to more.

Update: Does physical exercise control diabetes?- The post partially answers the question and shows relevent information with out fully answering the question but the first 2 comments attack it because it is not fully answers.

Is there any benefit to placing a silver spoon in a water carafe? - Contributing information about possible benefits which the Poster conceded he can not verify but which helps answer why it may be believed. Again not the full answer but could contribute.

Do razors with more blades work better? - Here again we have a response that has pertinent information that does not fully address the question but could definitely contribute to answering the question. It is challenged in comments as not fully answering the question.

There are lots more this was just the first 3 I found on my front page. I am not pointing fingers I think quite a few of us(yes myself included) tend to do it. I am not trying to say that what we did was wrong before but rather a concerted and visible effort to change that impression would go a long way toward encouraging more answers and keep more people engaged in the community. I think this could be a fun and relevant community with great questions and answers. But I get a general feeling of oppression about both questions(especially notability), and answers(mostly completeness and references), that make it intimidating to participate.