The problem with extrinsic motivationTL;DR: Stack Exchange favors standard, fast, and complete answers at the expense of novel, creative, and excellent answers. This bias results directly from reputation ratings, badges and other extrinsic motivations, which form the core of the system and can not be removed. Therefor, the quality of answers taken as a whole will likely never increase (though it may be good enough for nearly everyone at the current level).
As a preface, I've stopped using Stack Overflow after participating in the beta and using the system for a while. I'm not trying to stir up controversy or ruffle feathers, though that seems likely to happen. Rather this is a postmortem of one user's experience with Stack Overflow.
I've always enjoyed answering programming questions. It's a regular part of my job and one of the reasons I enjoying coming in each day. I remember when I discovered Usenet and the vast flow of questions from all over the world. It was like discovering a new continent. For a long time, I participated on comp.lang.perl.misc where I learned so much about Perl and programming in general. And I had fun.
Sadly, Usenet has problems that are intrinsic to the system. The same questions are asked over and over. Then the same answers are given. Then the same corrections and refutations of the answers. Then the flamewars. It got old and I eventually gave it up.
So when Joel Spolsky started talking up Stack Overflow it seemed like an ideal replacement for Usenet. And it does correct a number of problems with previous systems. Notably, very basic and common questions are not commonly asked on Stack Overflow anymore. People seem to be able to find their answers without having to ask and if by some chance they do ask, they are quickly pointed to the canonical question. It's also a plus that people can correct minor issues with other people's answers without implying they are losers or stupid.
Stack Overflow does so many things right and it was just the sort of system I thought I wanted, but I started losing interest. For a long time, I thought it was because there was no meta.stackoverflow.com for people to have a place where they could, well, talk about meta-issues. But after months away from the system, I think the problem runs much deeper.
When I first answered a Stack Overflow question, the little number next to my name started to go up. It was pretty neat and made me happy to have answered well. As I submitted more answers, the number started to go up a little faster. Pretty soon I started looking for questions to answer that would make my number improve even more. I love answering questions, but I also liked getting my number to go up. Pretty soon, I found that I made a little calculation before starting to answer a question: "Will this make enough of a difference in my number to be worth my time?" By the end, I was avoiding answering questions that I was interested in answering because there were already a few good-enough answers and I was unlikely to increase my number.
If you've taken a psychology class or two, you probably see the problem already: my intrinsic motivation was completely replaced by an extrinsic motivation. While there is debate among psychologists as to whether extrinsic rewards always drive out intrinsic motivation, few would agree that extrinsic rewards are better or more effective than intrinsic ones. We instinctively know that people perform better for love of the game than for a paycheck. When the extrinsic reward is a little number tacked to the end of your name, it seems even less likely to be effective.
The FAQ says: "Reputation is completely optional. Normal use of Stack Overflow — that is, asking and answering questions — does not require any reputation whatsoever." The first sentence is, of course, a lie. If you participate on Stack Overflow, you have a reputation number whether you like it or not. In addition, the "reputation" concept forms the core of the Stack Overflow experience by design. The second sentence clarifies the first to explain the nugget of truth: you don't need to understand the reputation concept in order to begin working with Stack Overflow. In other words, the reputation system is extrinsic by design.
While researching motivation theory, I came across this paper [1]. One paragraph in particular seems to sum up the Stack Overflow experience for me:
Extrinsic rewards have been found to reduce intrinsic motivation, but not in all circumstances. The majority of published research has dealt with the effect on motivation rather then performance, but consequent effects can be evident in performance, and there are many theoretical predictions supported at least in part by empirical findings. When people are intrinsically motivated they tend be more aware of a wide range of range of phenomena, while giving careful attention to complexities, inconsistencies, novel events and unexpected possibilities. They need time and freedom to make choices, to gather and process information, and have an appreciation of well finished and integrated products, all of which may lead to a greater depth of learning and more creative output. Extrinsic rewards tend to focus attention more narrowly and to shorten time perspectives, which may result in more efficient production of predefined or standardised products. Job satisfaction and long term commitment to a task may also be affected.
It's tempting to think that hiding or truly making reputation optional would fix the issue, but I'm not sure that a fix is possible even if it were desirable. If you look around at questions and answers on Stack Overflow, you'll find they are quite standardized, efficient and timely. In fact, from the point of view of a programmer looking for an answer, Stack Overflow works nicely, thank you very much. It seems like there are plenty of people who can thrive or ignore extrinsic motivation, so there doesn't seem to be a broad problem here. If you want to make an omelet, you're going to break a few eggs.
Just don't expect to get chickens.
So it's been over a year since I "asked" this question and I don't have any answers. I still don't use Stack Overflow (or any of the Stack Exchange sites), so I don't have any new positive data. But I do have some negative data (in the sense of not finding something I expected to find): I've not missed asking or answering programming questions on the internet.
I appreciate all your answers to this question: they are generally spot-on. I find as I re-read the question and the answers, that my problem may very well be just that: my problem and not the system's. To paraphrase Sir Winston Churchill (as inspired by Pekka):
Many forms of [Q&A; site] have been tried and will be tried in this world of sin and woe. No one pretends that [Stack Exchange] is perfect or all-wise. Indeed, it has been said that [Stack Exchange] is the worst form of [Q&A;] except all those other forms that have been tried from time to time.
Despite appearances, that is high praise.
Any system may be changed over time, but like all systems, the Stack Exchange system has developed social (and technological) inertia such that any change which might solve my fundamental problem would require years and a strong hand at the tiller to implement. Reputation and all the extrinsic motivation it entails are as central to Stack Exchange as campaign financing is to modern democracy. One can't simply wish it away. Nor does it serve any purpose to hide the number or pretend it doesn't exist.
Plato would never agree that Democracy is better than "all those other forms [of government] that have been tried from time to time." He believed that the people should be ruled by the most capable rulers, who would also be elite thinkers and philosophers. Churchill would have seemed to Plato a fool to suggest such a thing. But we live in an Aristotelean world and elitism has more or less been supplanted by practical egalitarianism. Not that there isn't room for experts and deep thinking (the opposite rather). Instead, the ideal isn't judged according to some abstract form that exists only in the imagination, but by observing the best and most complete examples that may be found. In keeping with the rest of Western society, Stack Exchange rewards the best and most complete answers to questions at the expense of some abstractly ideal answers.
Which brings me indirectly to what I think the answer to my question might be: ask and answer questions about topics that I'm actively learning rather than topics I've already formed Platonic ideals about. Looking back at the questions and answers I most enjoyed writing in my time with Stack Overflow, I see that they tended to be on topics I had not yet mastered. Recently, I've been learning Lua and LaTeX, so when I wrote on those topics I found myself completely engaged in the responses I might get. An upvote gave me a rush of excitement not unlike the feeling of victory when scoring points in a game. A downvote led me to scramble to find the source of criticism and correct it. Meanwhile, answers related to Perl, ksh or C, which I've master to my satisfaction, I submitted out of duty with a feeling of dread over any response. Negative responses tended to be themselves mistaken or at least unhelpful in teaching me anything new. Even positive responses were hollow since they added nothing to my understanding. Further, worse answers (in my opinion of course) might receive more positive response due to the vagaries of the system.
I actually came back to Stack Exchange because I'm so frustrated trying to find answers to my gardening questions. It reminded me of the first few times I tried to find answers to programming questions via search engines: lots of junk with occasional gems scattered in all directions. It occurred to me that what I was looking for was "Stack Overflow for gardeners". Lo and behold: "Gardening and Landscaping" has just finished the "Commitment" stage on Area 51. My plan is to participate there (and as you might have guessed, the "Philosophy" site when it starts) and see how it goes.
ACCEPTED]
I absolutely agree there is a dark side to the reputation system, influencing the way we post and word answers more deeply than is comfortable. No doubt. What the rep system fails to support is deep research on a subject, the exchanging of ideas, trying out different possibilities, developing new things, playing without the pressure of producing an answer that can be upvoted and accepted. I assume - forgive me if I'm wrong - that this goes into the direction you mean by "chickens."
But all these things are still possible to do, and they happen every day. The really interesting conversations sometimes take place in the comments sections. Rather few of them, that is true, but that is not necessarily only down to the system.
What makes SO extremely valuable in my opinion, despite the narrowness that the reputation system sometimes encourages, is the fact that it works, that means, brings together a huge number of extremely good programmers. It keeps you up to date about best practices, tools and standards simply through the constant buzz of the huge crowd. It provides a platform of exchange that - at this moment - has no comparison on the global Internet. It is functioning well in many ways I wouldn't want to miss - in ease of use, quality of questions and answers, visitor frequency and transparency in management.
Still, if a new platform arises that has the goal of producing "chickens" in the sense you mean it, being less focused on shallow reputation points, I'll be very interested to see it.
Apparently, I'm now officially involved [1] in Stack Exchange again. How did that happen?
You might remember that about a year ago [2] I planned to participate on Philosophy [3] and Gardening [4]. It turns out that I don't have many questions about gardening as I only have a couple of windowsills, a balcony and a deck to work with. G.SE is a good site, but not about a subject that tickles my interest very often. And it turns out that asking philosophy questions is extremely difficult. Rewarding, but hard to do well. I'm in the middle of reading Plato's [5] Complete Works [6], and if I could find the time [7], I'd ask more questions there.
I'd probably be bouncing around from one site to another (more than I already do, I mean) if it weren't that the Biblical Hermeneutics site hit beta last fall. It turned out to be a perfect storm of the right topic, the right group of co-contributors, and all at the right moment in my life. Everything clicked for me.
Time and again, I've notice the same pattern1 in myself when becoming involved in anything on the Internet:
Interest in the topic.
This is where we all start. At this point, if Stack Exchange doesn't have a topic you are interested in, it probably doesn't exist.2 It's also the moment at which intrinsic motivation is at it's maximum. You don't come to a Q&A site because of its awesome community or a karma system or to get swag; you come to find the answer to your question.
This is also where the vast majority of people end their involvement. Once you find out why your table insert is so slow [8], there's no particular reason to come back. In fact, many more people will read the answer to their question and never need to create an account. In general, that's just fine: the internet has become marginally better.
But some people cause problems even at this stage. The most obvious (to me at least) are students who ask homework questions, but have no real interest in learning the material. What you find is that they lack even the most basic intrinsic motivation that a Q&A site requires for operation. Individually, they aren't a big deal, but taken in aggregate these folks are a huge aggravation. For as long as I've been on the internet [9], people have been barging into existing communities without reading the FAQ, and I don't see why SE is any different.
Interest in the site.
It's probably obvious, but if a site is to become a locus for people to indulge in their interest in a topic, they are going to develop an interest in the site itself. Almost by definition, interest in the site is extrinsic motivation, since its not the topic. At best, via a harmless indirection [10], the user focuses on the site as a means to the end of learning about the topic.
Now there nothing wrong with a little system [11] to encourage users to continue interacting with the site. There's also nothing wrong with getting to know the other participants and even making friends. There's nothing wrong with thinking about rules of interaction. But there is a problem with losing your first love: the topic itself. That was the thesis of this question, so I won't rehash it here.
But there are plenty of other bad behaviors that stem from decreasing intrinsic motivators. For instance, there are all sorts of tricks that one can use to obtain high reputation without adding value to the site. Trolls of various species thrive on the attention they can garnish from undermining the community. Ordinary folks tackle boredom by creating polls, shopping questions, puzzles3, joke posts, and so on. All of these are cancers [12], some malignant and others benign, that will kill a community if allowed to spread.
Interest in the community.
Every now and again, for certain people at certain times, something changes subtly. Instead of the site being the locus for learning about the topic, the community becomes a home base. At that point, its entirely possible for intrinsic motivation to disappear, which is troubling. If you've ever been at a meeting with someone who just enjoys hanging out with everyone else and doesn't care about the subject of the meeting, you've witnessed the problem.
On the other hand, it's at this point that truly selfless behavior is possible. "I care about the community" is often the same thing as "I care about somebody else's well-being as much as I do my own". That's powerful. It touches something far deeper [13] than any of the individual topics represented on Stack Exchange. In some ways, an extrinsic motivation becomes an intrinsic motivation.
Rather than wax philosophical, however, let me point to one definite area where interest in the community is unambiguously positive and altruistic: ♦ moderators. On a small-town site like Biblical Hermeneutics, being a moderator is mostly fun. We get to be a little bit like Andy Taylor [14]: people mostly respect us and appreciate the work we do. But on Stack Overflow, ♦ moderators are more like parking enforcement. Nobody would take that job unless they were paid, but there are currently 15 volunteers [15] doing just that. For the life of me, I don't know why except that they really care [16] about the community.
Actually I don't have one at the moment. I suspect that I won't ever learn to love Stack Overflow [17], but then again I thought I would always hate Los Angeles. So you never know.
Footnotes:
There's a good chance my thinking on this comes from The Four Loves [18].
Or it needs more support on Area 51 [19].
My [20] particular [21] vice [22].
For the life of me, I don't know why except that they really care about the community. -- That's exactly why. Unfortunately, we're outnumbered. There will always be far more new people coming in that wish to use the site for their own purposes that are at odds with the community's standards than we can keep up with. - Robert Harvey
Some psychologists teach that human beings have 5 basic psychological motivators:
Everyone has a bias towards one or two of them.
Stack Overflow provides an experience that feeds into all of them to one degree or another. The primary motivator, though, is prestige - there is a number, a metric, that is determined solely by the actions your peers take on your posts.
The others are fed to a lesser degree, but it seems you are saying it would be nice if Stack Overflow focused less on prestige and balanced the system so that it equally attracted those, for instance, that are more motivated by Curiosity and Connection.
These are very good thoughts, the best of which (a question to ponder, actually) being what I gleaned to be the general idea of your original post:
Intrinsic motivation is known to produce better results than extrinsic motivation, but is the presence of extrinsic motivation actually a problem?
While I know through my own participation that the extrinsic reward of reputation has been at times exciting and even driving, I feel I can, with own experience as evidence, present two ideas:
Yesterday I answered a question about JavaScript's pushState [1] on StackOverflow. I came across the question after asking one of my own on the topic, and seeing that quite a bit of time had already passed without a satisfactory answer (one zero score), and even though the topic was new to me, I believed that I could put together a decent answer.
At the time I originally answered [2], I probably did the sort of calculation that you do — I don't want to put more time into this than it's "worth".
Shortly after, but right before I had to leave the computer for a few hours, I got a response from the poster, calling into question the assertions I had made. While I was away, I thought about the response and wondered if maybe I ought to look into it further and approach it from a different angle. By the time I was back, I had concluded that my original answer was lacking, and even though (as the question had obviously had no more traffic) it wouldn't "pay off", I was driven to provide a better answer, and (in my own, hopefully humble opinion) put together one of my better answers on SO.
This whole experience happened before I had read this post, and it hadn't even occurred to me to consider what the real drive for answering was. But, by the time I put substantial edits into the post, it certainly wasn't reputation, as I was really only hoping for the asker to be satisfied with my answer.
In this experience, I was intrinsically motivated by completeness, correctness, and the goal of helping someone.
Or, perhaps this question is actually "Why isn't reputation, as an artifical value, suppressing the intrinsic motivation?"
In this I want to focus on the intrinsic motivation of helping others.
In my experience, there is a fairly simple (but rough) formula for gaining reputation:
Good Answer × Traffic = Reputation
In other words, if I put together a well-organized, correct answer, I more or less know it will get upvotes if there is traffic (so Good Answer becomes a constant): Traffic = Reputation
So consider this: If my goal is to help the most people, how would I go about doing that? I would certainly look for the posts that will bring the most traffic. In other words, reputation. Reputation is actually a fairly good measurement of how much I am helping others.
Then, your value question (which was presented, originally, as a problem) "Will this make enough of a difference in my number to be worth my time?", is actually:
Will this
make enough of a difference in my numberhelp enough people to be worth my time?
Which, I believe, is a perfectly reasonable intrinsic motivation. This can even hold true for answering questions which already have several "OK" answers — we know that the traffic for our answer will be reduced, and even though it may help a couple people who scroll that far, it just might not be worth our time. (And if that's happening, it actually is a problem — but the point is that in that case, reputation is not the source of the problem; instead, it's a value measuring tool).
Your answer to your own question probably hits the nail right on the head:
In keeping with the rest of Western society, Stack Exchange rewards the best and most complete answers to questions at the expense of some abstractly ideal answers.
There may be some aspects of the SE system that actually do prevent some of the best answerers from answering questions at time. For my part, though, I don't believe that it's simply because there is not enough of an artificial reward to doing so. Every participant, while each holding different reasons for investing their time on Stack Exchange, must find some return on that investment. For myself, I relate a lot to your insight of loving to answer questions where you are learning. I also like to know that I've helped bring some truth or accuracy to a cloudy subject. Everyone might be different, but I think the general population is going to be mostly unaffected by that little number.
[1] http://stackoverflow.com/q/6193858/143295pushState question is exceptional. (And to your credit.) - Jon Ericson
There will always be people who try to look for something like rep or badges, even if the system itself doesn't offer them. Because even if that's not your primary motivation, it's fun to see how you stack up to other users.
Do some users take it too seriously? Yeah. Do good answers still get posted to obscure questions? Yeah.
Personally, I lost most of my interest in gaining reputation once I was able to edit. I still enjoy finding interesting questions though, whether that ends in reading the answers or posting my own. Ultimately, SO is just another Usenet replacement, with its own rules and traditions, various people who take them seriously, and hoards of casual users who just want to learn or share their knowledge with others.
IMHO, so long as SO doesn't place any additional value on reputation (allowing high-rep users privileged access to cash or valuable prizes say...), this'll be a non-issue for most people.
um... ok?
So, to summarize, "Stackoverflow, due to its reward structure, will never, ever, ever receive awesome answers."
I don't agree or disagree with that statement. As Bill Cosby says, "The proof is in the pudding."
What is present in terms of questions and answers will help some, and not others. This site isn't for everyone, though, and frankly if the attempt was made to please everyone, then it would be nearly worthless.
But since you've gone to such great lengths to analyze the motivation system, can you suggest ways to improve it, and share examples that demonstrate intrinsic motivation produces better results?
I am quite positive that reputation does have an extrinsic influence to some degree in all participants on SOFU. The influence may be so small that it is irrelevant compared to the degree of satisfaction in answering a question. I enjoy seeing my reputation go up - I won't lie. That said, I have found it fruitless to only work at reaping the rep.
What I do like about SO (specifically) is that there are a great number of programmers out there willing to give me advice and help me learn. Often times, on a forum, if a question requires a little bit of research, it is easily ignored because there is no reason to do the extra work to provide a well formulated answer. Reputation, however great or small, does provide that extrinsic influence that is needed.
Reputation has another affect. Those that fail to understand how the site works, or fail to abide by the rules of SOFU, see their reputation reduce. Eventually they lose interest and move on.
Regardless of the reputation gains, a good question is just that - a good question. A good answer is just that - a good answer.
I agree to a certain degree. What I found while using Stack Overflow is that people tend to answer mainly trivial questions: either googlable or basic for a given technology. This is the case, because many people view these questions (because they know the answer) and the answers get upticks. When I had some real problems though my questions were usually either left without answers or got answers that were not really answers to the question. why bother researching a difficult question when you can get 10 times more points for a 30 second answer to a trivial one?
One thing this extrinsic motivation system helped achieve is the large amount of recurrent users, including many very well-respected ones. I believe this is one of the major strengths of the site.
As I understand it, SO (and every other SE site) attempts to fill one role: to create a community in which you can ask a question and receive an answer. It does not, however, attempt to fill others. It is not a place to exchange lengthy discussions or to publish a new bold idea. SO is not the only place on the web. It serves a role and as such it is limited to that role. Journals, wikis, fora or blogs might be better suited for other types of exchange or discussion of novel ideas.
SO is a tool. It is one of the many tools available. Of course, you can choose which tools to use, but the more and the better tools you use, the faster you'll get your job done.
Although I agree that the motivation system may affect how you answer a question (or choose not to do so), in my opinion that aspect is not as significant as the effectiveness of the system to sort what people find to be helpful. My point is that I would still use SO if there wasn't a reputation system, but if I was still able to upvote good questions and answers.
By the way, I found your question because it was mentioned on the podcast [1], so I wonder if they found it because the system worked to bubble your question up or if your question went up because they mentioned it. I don't think Jeff and Joel answered your question, but at least you gave them something to think about.
[1] http://blog.stackoverflow.com/2011/06/se-podcast-06/The pertinent question I think is how the reputation system contributes overall and whether it is a net gain in terms of increasing participation within the community?
Many comments in this thread and others like it have indicated altruistic motives and personally motivated reasons for answering questions. However, I wonder if the overall level of engagement would be the same without the immediacy of feedback and recognition that is provided by the reputation system?
Does the reputation system help to add a little extra motivation to a task that people are already personally motivated enough to contribute to in the first place? In a community where expertise is highly regarded, I think the ability to compare yourself to others at a glance is probably a good thing in terms of motivating further participation.
