Currently, I'm working on revisions to the first article I've submitted to a journal for publication. (Incidentally, there's a post in the works about that revision process.) And as much as I had already showed the article to two readers prior to submitting it to the journal, and as much as I was (and still am) pretty pleased with the form in which I originally submitted it, one of the two reader reports came back with a mountain of suggestions for revisions and additions. For the most part, the anonymous reviewer was spot-on with his or her suggestions. Those comments are pushing me to do more research and interrogate my own basic assumptions; in the end, this process will make my article even better than it was. And with that being the case, I'm very glad that only five people (my two trusted readers, the two reviewers and the editor of the journal) will have seen the article in its original, acceptable-to-me-but-still-inferior version. No reason for the entire professional world to read my work not in its tip-top form.
As a result, I don't think I'd ever be inclined to submit an article to a journal that uses crowd-reviewing* rather than double-blind readership as its method of peer review. Part of the problem, for sure, is the fact that the academy hasn't quite caught up with emerging technologies. In simple terms, I have no way of knowing whether an article in a crowd-reviewed journal would count the same towards tenure as an article in a traditional double-blind peer-reviwed journal would. I'd like to think that if I were really partial to the journal or really believed in the idea of crowdreviewing that I'd submit there anyway, consequences be damned. But the bigger issues of expertise, overwhelm and discretion make that point completely moot.
The idea of crowd-reviewing is either genius or mad; for now I'm tending towards thinking the latter, though. Part of my hesitation is owed to my continued skepticism over the theory that crowds are wise. I still believe in expertise. I shudder every time I read about a project like Transcribe Bentham, and feel a certain schadenfreude every time a project like that is abandoned (specifically, see the top of page two of the linked story) because it turns out that scholarship does require methodological training.** I have no problem, for example, with doctors polling large numbers of their colleagues when they are confronted with a constellation of symptoms they can't explain because in that case everybody in the crowd has expert medical knowledge. And the reviewers for this journal are, ostensibly, proper medievalists.
But I still don't want the great masses reading my unwashed prose. (That's why I'm so ambivalent about this blogging experiment. I think it has value in the form that it takes and in being public rather than private, but I kind of keep hoping that nobody will notice. So far, so good.) A really good, careful edit by two experts is likely to be as good, if not better, than a whole pile of comments from a crowd that may include experts but may also include well-meaning but under-informed amateurs or worse. An additional problem with a crowd is that it is a crowd. Committees are less efficient the more people who sit on them, and likewise, even if every comment made about an article by a crowd were expert and excellent, there would invariably be an excess of noise. And finally, crowd-sourcing as peer-review leaves the web and the universe littered with early drafts easily accessible. Writing — even technical, professional, academic writing — is such an intimate thing to bare to the world that to show it off unfinished leaves the writer that much more and needlessly exposed.
Edited on 9/30/11 to add: I see that I've been getting some hits on this particular post from a password-protected section of the crowdreview site. I don't know if they've written something in response or just posted the link. I'm still very much learning about blogging etiquette, but that does strike me as a little unsporting.
*The term I have written here as crowd-reviewing is, correctly, a single compound word rather than a hyphenated combination of two words: Crowdreviewing, like its predecessor, crowdsourcing. I've hyphenated here for ease of reading on a screen.
** Turns out that there's a whole blog dedicated to the subject of crowd-sourced transcription of historical records.