Open The Echo Chamber

A great deal has been written about the tragic death of Aaron Swartz, so much that I considered remaining a reader and observer without offering comment.  But the Swartz case has me thinking again about access to academic research. Not one academic author of those articles was negatively impacted by Swartz’s act (downloading millions of scholarly articles from JSTOR with the intent of posting them online for free) – the more easily accessible the article, the more likely it is to be read and cited…and that is why we write articles.  It seems to me that most people don’t understand the fundamental absurdity of copyright in academic publishing.

I quote from one transfer-of-copyright document I recently had to sign:

In order to ensure both the widest dissemination and protection of material published in our journal, we ask Authors to transfer to [Journal Name] the rights of copyright in the articles they contribute. This enables our publisher, on behalf of [Journal Name] to ensure protection against infringement.

The whole point of publication is to get people to read and use my ideas – the very idea of infringement is pretty vague here.  I do not receive a cent for any academic article I publish, so infringement won’t affect my income. Anyone who plagiarizes me and gets caught will lose his or her career – I don’t need copyright for that. So there is no reason for me to sign this document. But what the document leaves vague is the fact this is not a voluntary transfer – the journal will not publish an article without such an agreement, and without publications the typical academic will have a pretty short career.  In short, the average academic is forced to sign away their rights to their work if they want to have a career (no publications means no tenure).  I don’t care about my rights, honestly, except when my work then ends up behind a paywall, downloadable at $30 a pop, nobody who needs to access it (i.e. colleagues in the Global South, or even colleagues at most development donors) can access it. Somebody is making a lot of money of my work and the work of my colleagues (see this article too), but it isn’t me.

However, there does seem to be an out here, at least for employees of state institutions, or those whose research is funded is funded under a federal contract.  From the same agreement I just quoted:

I hereby assign to [Journal Name] the copyright in the above specified manuscript (government authors not transferring copyright hereby assign a non-exclusive license to publish)… [my emphasis]

While I am sure this is not how it was intended when written (it is a clause to allow federal employees to publish publicly-funded research), I wonder if those of us either employed by a public entity, either directly or under a contract, can invoke that status to shift our copyright transfers into “non-exclusive licenses to publish.”  This would remove the copyright infringement argument used against Swartz, thus making it easier to pull articles from behind paywalls into the public sphere.  In short, we need to stop transferring copyright to for-profit entities any way we can…but this needs to happen in a manner that doesn’t blow up everyone’s careers.  Until the senior faculty in each discipline decide to intervene and shift emphasis to low cost, open-access journals, this could be a useful first step.  And low cost can be done – see Simon Batterbury’s comment about the Journal of Political Ecology on the post in the last hyperlink.

In short, academics need to step up and start resisting an academic publishing machine that makes serious money off of our job requirements, but provides little in return.  If we do so, perhaps we won’t need folks like Aaron Swartz to liberate our work – we can do it ourselves.

Vincent Calcagno has a fascinating piece up at the LSE impact blog, in which he looks at the review and publication histories of an absolute pile of articles.  There are whole set if interesting findings there that are well worth the read.  For example:

But, surprisingly, we found that about 75 per cent of all articles we declared to have been submitted to the publishing journal on first intention. Even assuming that, for some reason, authors were less likely to respond in the case of a resubmission, we still find that a majority of published articles are first-intent submissions. This suggests that authors are, overall, quite apt at targeting a proper journal and, conversely, that journals make sure they have a sufficient public: no journal was found to be entirely dependent on resubmissions from others.

However, the finding I found most interesting was this:

in a given journal and a given year, an article that had been resubmitted from another journal was on average more cited than a first-intent submission. Resubmissions were less likely to receive zero or one citation (about 15 per cent less, controlling for publication year and journal) and more likely to receive several (e.g. 10 and 50) citations, shifting the mean to higher values. This intriguing result suggests a “benefit of rejection”. The simplest explanation would be that the review process and the greater amount of time spent working on resubmitted manuscripts does improve them and makes them more cited, although other mechanisms could be invoked.

I wonder, though, if there is another factor that should be considered.  Peer review is inherently conservative – there is a lot of thought policing that goes on through this process (I’ve gone on about this before, here and here).  I wonder how many of the “resubmissions” were rejected not because of insufficient quality, but because they were doing interesting work that threatened one or more reviewers, leading to rejection.  This makes sense, as new and edgier work will eventually get cited more than middle-of-the-road replication of old results – at least, that has been my experience.  So perhaps Calgano has given us empirical evidence for the intellectual policing function of peer review.

So, a while back I decided to talk about how I negotiate peer review, semi-liveblogging my response to a revise and resubmit request from a pretty big development journal (see part 1, part 2 and part 3).  Well, I now have a response to my resubmission . . .

No.  To quote: “after much deliberation, the editors have reached a rather difficult decision. [The editors] feel that they cannot accept your revised paper.”

Yep, I have gone from revise and resubmit to outright reject.  This is . . . unusual, to be honest. More unusual, however, is the rationale for the rejection.  To quote from the decision:

What makes this difficult is that [the editors] recognize that you have in fact taken account of what the referees said, and have tried to accommodate their comments, but the editors feel that what has emerged from the revision process is not an appropriate paper for Development and Change.

Translation: you did what we asked, and addressed the referee comments, but in doing so you ended up with a paper that we think belongs at another journal.  Well, fair enough, this happens.  But why it is not appropriate is a little odd:

While they still believe that there is an interesting idea at the core of your paper, they don’t feel that the revisions have solved the initial problems, and they are not convinced that further rounds of revision would be any more successful. The intended contribution of the paper appears to be theoretical, but the paper hasn’t managed to work out that contribution in a way that will be accessible / comprehensible to our readers.

Soooo . . . I have an interesting paper, but the editors more or less think their readership can’t deal with the complexity of the argument.  [Note: I am disregarding the assessment that my revisons have not solved the initial problems, since they already have said that I took account of the referees’ issues – this is a contradiction I am just going to leave aside. That, and they did not show me any reviewer comments, so I have no idea what I did not resolve]  One of my colleagues has called this the oddest rejection he has ever seen.

Now, I want to be clear – the folks at the journal with whom I interacted throughout this process were very responsive and polite, and were kind even in their rejection (they were quite apologetic, actually).  I would submit to this journal again, though I admit to wondering exactly what aspect of my work might fit here, as I am confused by what they believe the capacity of their readers to be.

This, folks, is the nature of peer review – sometimes, you just have no idea what happened.  I am not privy to the internal conversations of the editorial board, and will not pretend to know exactly what happened here.  What makes this hard is that I did not receive any substantive comments on this second round of review, so I have no guidance at all on edits.  I am rereading the paper, adding a citation I had missed earlier, and making minor tweaks to the argument (the article I missed before actually strengthens the case for what I am doing in the manuscript).  I’ve sent it off to a trusted senior colleague to have a look, and to see where he thinks it might go next.  I will probably sound out the next editor in advance, just to make sure that s/he thinks the paper is appropriate before starting a long review process again . . .

Two years and counting, folks, since my initial submission.

Any editors out there interested?  Anybody?

Following on my previous post, another thought that springs from personal experience and its convergence with someone’s research.  If you look at my Google Scholar profile, you will note that in 2011 my citation counts exploded (by social science standards, mind you – in the qualitative social sciences an article with 50 citations or more is pretty huge).  Now, part of this is probably a product of my academic maturation – a number of articles now getting attention have been around for 3-4 years, which is about how long it takes for things to work their way into the literature.  However, I’ve also seen a surge in a few older pieces that had previously plateaued in terms of citations.  This can’t be attributed to a new surge in interest in a particular topic, as these articles cross a range of development issues.  However, they all seem to be surging since I got on Twitter and joined the blogosphere.  Bascially, it seems a new circle (circles?) of interested folks now has access to my work and ideas, and the result is that my work is finding its way into a new set of venues/disciplines that it might otherwise not have reached.  It is hard to be sure about this, as my 18 months on the blog and 1 year on twitter are just at the edges of how long it takes to get an article written, submitted, accepted and published, but clearly something is happening here . . .

This seems to be borne out by some work done by Gunther Eysenbach examining the relationship between tweets (references to a paper on twitter) and the level of citation that paper eventually enjoyed.  Eysenbach found that “highly tweeted” papers tended to become highly cited papers, though the study was quite preliminary (h/t to Martin Fenner at Gobbledygook.  You can find links to Eysenbach’s paper and Martin’s thoughts on it here).  This makes sense to me – but it requires a bit more study.  I like what Fenner and his colleagues are trying to do now, capturing the type of reference made in the tweet (supporting/agreeing, discussing, disagreeing, etc.).  Frankly, references in general should be subject to such scrutiny.  As one of my colleagues once said, if citation counts are all that matter we should write the worst paper ever on a subject, jam it into some journal that did not know better, publicize it and wait for the piles of angry negative citations to pile in . . . only we just have to count the citations, not admit that we are being cited because people hate us!

The altmetrics movement is starting to take off in academia (see, for example, this very cool discussion) I have not yet seen any discussion, though, of what social media might do to journal prestige.  While there will always be flagship journals to which disciplines full of tenure-track faculty will bow, once tenure is achieved this sort of homage becomes less important.  Given what I am seeing with regard to my citations right now, my desire to have my work have impact beyond my discipline and the academy, and my concerns for the policing effect of peer review (which emerges most acutely in flagship journals – see my posts here and here), why should I struggle to get my work into a flagship journal when I can get a quick turnaround and publication in a smaller journal, still have the stamp of peer review on the piece, and then promote it via social media to a crowd more than willing to have a look?  If I (or anyone else) can drive citations through mild self-promotion via social media, does the journal it is published in really matter that much?  I wonder what sort of effect this might have on the structure of publishing now – will flagship journals have to become more nimble and responsive, or will they soldier on without changes?  Will smaller journals sense this opportunity and move into this gap?  Will my colleagues embrace the rising influence of social media on academic practice?

Does any of this matter?  Not really.  If the emerging studies on social media and citation are correct, and my trends are sustainable, then one day I will be one of the “important” folks with a lot of citations . . . and I will be training my students to engage in conventional and non-conventional ways.  I will not be the only one.  Those of us who engage with social media, and train our students to do so, will eventually win this race.  Change is coming to academia, but the nature and importance of that change remain up in the air . . .

Been a while . . . been busy.  And yes, I stole that post title from Ralph Nader . . .

As those who follow this blog know, one of my big concerns is with the walls that academia is building around itself through practices like the current incarnation of peer review in specialist journals. It’s not that I have a problem with peer review at all – I think it is an important tool through which we improve and vet academic work. Anything that survives peer review is by and large more reliable than an unvetted website (like this one, for example).

But the practice of peer review in contemporary academia has turned really problematic. Most respected journals are more expensive than ever, making access to them the near-sole province of academics with access to libraries willing to purchase such journals. The pressure to publish increases all the time, both in rising demands on individual researchers (my requirements for tenure were much tougher than most requirements from a generation before) and in terms of an ever-expanding academic community. The proliferation of published work that has emerged from these two trends has not really improved the quality of information or the pace of advances – there is still a lot of good work out there, but it is harder and harder to find in an ever-growing pile of average and even not-so-good work. And I have found peer review to often function as a means of policing new ideas, slowing the flow of innovative ideas into academia not because the ideas are unsupported, but because these ideas and findings run contrary to previously-accepted ideas upon which many reviewers might have done their work. This byzantine politics of peer review is not well-understood by those outside the academic tent, and does little to improve our public image.

So I am wondering where the tipping point is that might bring about something new. Social media is nice, but it is not peer-reviewed. I tend to think about it as advertizing that points me to useful content, but not as content itself (I have a post on this coming next). I still want peer-review, or something like it. So, a modest proposal: senior colleagues of mine in Geography – yes, those of you who are full professors at the top of the profession, who have nothing to lose from a change in the status quo at this point – who will get together and identify a couple of open-access, very low-cost journals and more or less pronounce them valid (probably in part by blessing them with a few of your own papers to start). Don’t pick the ones that want to charge $1500 in publishing fees – those are absurd. But pick something different . . .

This, I think, is all it would take to start a real movement in my discipline – admittedly, a small discipline, so maybe easier to move. Just making our publications open to all is a tiny first step, but an important one – once a wider community has access to our ideas, they can respond and prompt us for new ones. Collaborations can emerge that should have emerged long ago. Colleagues (and research subjects) in the Global South will be able to read what is written about their environments, economies and homes, improving our responsiveness to those with whom, and hopefully for whom, we work. First steps can be catalytic . . .

I’ve made a few changes to my personal homepage (www.edwardrcarr.com).  This included cleaning up a few things, adding a few book reviews for Delivering Development, and updating my CVs.  However, today, for the first time since I set my homepage up, I have added a page . . . there is now a page for pre-prints.  I have become thoroughly fed up with the gatekeeping and slow pace of academic publishing – I was annoyed to start with, but after more than a year in an agency, and about 18 months engaged with a much wider environment/development community via the blog and twitter, I have come to realize that academic publishing, for all its rigor and legitimacy, is something of a liability.  There is no way anyone is going to wait around for my work, or anyone else’s work, to wend its way through peer review and the inevitable publication delays before it appears in print.

To address this, I am now posting work that I have submitted for review – it is polished, and sometimes it has seen a round of peer review already (those will be marked revised and resubmitted).  However, they are not fully finished, peer-approved work – which means they will likely change a little before they come out in final form.  My goal is to make this stuff available more or less as soon as I submit it.  I am open to comments and suggestions – I can still work them in before the final version goes out!

Some of you might wonder how this could affect the idea of double-blind peer review.  Well, in my experience, double-blind peer review in development studies – or indeed in any of the qualitative social sciences – is largely a joke.  In my field, we tend to invest a lot of time and effort working in a particular place, and so it is very, very easy to figure out who is writing about what.  I often know who the author of a piece is as soon as I read the abstract – and there are always enough details in any manuscript to facilitate a quick Google search that will identify the author.  Both pieces that I currently have on my website work from material for which I am well-known within my field.  For example, just mentioning the villages of Dominase and Ponkrum in Ghana in the livelihoods piece pretty much tells everyone who it is.  And the piece on academic engagement with development practice comes directly from a panel at last year’s Association of American Geographers Annual Meeting which was attended by more than 100 people, as well as an extended listserv exchange in the fall of 2010 that was sent out to several thousand subscribers of various lists.  Again, pretty much everyone will be able to figure out who wrote it.

So, the work is now up there for your perusal.  Have a look, and let me know what you think . . .

Colleague Ben Neimark at ODU recently asked me a tough question: “What makes for good (helpful to get published, strengthened,  intellectually creativity, etc.) peer review?”  I figured this might be of wider interest to academic colleagues, as well as those who see the entire academic publishing world as somewhat opaque.  So . . .

I think the challenge in producing a good peer review is to balance its dual imperative .  There is the part of peer review that ensures quality and offers constructive criticism (and I have received some in the case of my current livelihoods work – see here, here and here -, and have had some reviewers offer great stuff in the past).  Then there is the disciplinary policing that goes on through peer review, where reviewers don’t examine the quality of the data or argument, but simply argue against it because it challenges convention (which the reviewer likely belongs to or established) – see my comments about reviewer 1 at the bottom of this post.  This second function makes innovation very challenging unless you are very, very hardheaded (which I am).

In a nutshell, though, I think good peer review is that which looks at a paper for its stated aims and evaluates

1. are those stated aims actually new and interesting and
2. did the paper achieve the stated aims.

If standard 1) is not met, a good peer reviewer should be able to suggest where the real contribution of the paper lies – i.e. by suggesting literatures into which the author should place the manuscript.  If standard 2) is not met, the reviewer should explain exactly how and why this happened, and what sorts of remedial steps might solve the problem(s).  That is my minimum take . . .

I’m happy to hear the opinions of others . . .

In our last installment of this series (in which I try to lend some transparency to the publication process), I had requested (in great detail) further clarification before I revised and resubmitted an article to Development and Change. Yesterday, I received a reply.

DECH-11-094 – Livelihoods as Governmentality: Reframing the Logic of Livelihoods for Development

Dear Edward,

Thank you for your detailed response to the referee reports that were sent to you in connection with your paper.

The editors discussed your e-mail during their editorial board meeting last week, and have asked me to contact you. They found your response to the referee comments generally reasonable, and a good foundation for revising your paper. Assuming that you are still interested in doing so, they invite you to go ahead with the revisions as proposed.

So, this is good news.  But then the editor went an extra step – in other words, she made some editorial suggestions:

I would like to mention a couple of additional points. We ask all authors who submit revised manuscripts to include a list of revisions made. The response that you have sent to us indicates that some of the comments will not be addressed in the revised paper: it is important that your list of revisions also includes a rationalization for what you have NOT done, as well as for the changes that you have made. However, you might also consider whether any of those ‘non-responses’ need to be covered in some way in the revised manuscript itself. For instance, in your point 2 d) (ii) you suggest adding a footnote regarding (the lack of) remittances; it might be helpful to consider something similar for the point above it, on religious activity. And more broadly, rather than dismissing the report of Reader 1 because s/he seems to be asking for a totally different paper, you might consider whether you could reformulate any elements of your paper to ensure that similar criticisms could not be levelled against it by other readers, in the event of it being published. Finally, with regard to the point about precipitation: there is clearly a difference of opinion here, although the referee mentions an IPCC report (as well as other statistics), and you tell us that you are on the IPCC and have access to rain gauge data. Since the issue at stake seems to revolve around widely available quantitative data, presumably it can be resolved through the use of recognized sources and citations?

There is nothing wrong with this at all – it is a clear signal from the editor of the preferred tack to take while editing this piece.  Contrast this with the initial editorial statement I received. Which would you rather operate from?

The editor also acknowledged my concerns regarding one of the reviewers:

For the record I would like to stress that correspondence about the revision process does not imply any kind of commitment to a revised version of the paper. As stated in Friedl’s original e-mail, revised manuscripts are subject to further review and refereeing. In practice, the editors are often selective when it comes to sending revised papers back to referees, but they reserve the right to approach any of the original referees, and/or one or more new referees, as they see fit.

That is as close as an editor is going to come to saying “yeah, we’re not going back to reviewer 1.”

Had I not pushed back and laid out my concerns, I would be operating with the initial very vague guidance.  Now I have a very clear, achievable path to getting this done.  There is no guarantee it will be accepted, of course, but now my odds are greatly improved.

In the interest of transparency, this was my response:

Thank you for getting back to me, and doing so in such a detailed manner.  I greatly appreciate your editorial involvement in the revision process.  I will, of course, submit a cover letter that details all the revisions made in detail, and explain what issues the revisions were intended to address – and I will, of course, explain why I have not chosen to address certain comments.  I take your point with regard to reviewer 1 – in fact, if I look back at my notes on his/her review, I do see one or two things that I could address from that review, and I will try to think about how I might phrase things in the introduction to address the sort of reading I recieved from this reviewer.  I have no problem with footnoting the religion issue, and I will, of course, document my claims about the climate and the rain gauges.  I believe the reviewer was looking at the data for the Sahel, which is rather different than in coastal areas.

[Note: There is no harm in conceding a point to an editor, especially when that point is valid! It also signals to the editor that you are taking their points seriously.]

I fully expected my resubmission to be reviewed, which of course carries with it the risk of rejection or further revisions – though hopefully I will revise the paper in a manner that avoids this outcome.  I of course recognize your editorial right to select whatever reviewers you see fit to assess this revision – I merely wanted to voice my concerns with reviewer 1 for the record.  I appreciate that you and your colleagues do try to be selective about who reviews revisions, and having raised my concerns I am more than happy to proceed through this process with the reviewers you see fit.

[Note: read the last two sentences as me more or less saying "I'm glad you agree that reviewer 1 sucked."]

Again, my thanks for taking the time to send along detailed comments.  I will get to work on the revision shortly, and hopefully turn it back to you before the end of the month.

Upward and onward, y’all.

Yesterday I posted about a less-than-productive set of comments from at least one reviewer, and some contradictory comments from the other reviewers, related tof a manuscript I have in submission to a major development journal.  This is not at all uncommon – I’ve only had two or three articles ever accepted without revision, and have gone three or more rounds on a few.  However, to manage this sort of thing successfully requires good editorial guidance – i.e. what comments are relevant, whose comments are more appropriate (when there is a conflict in the comments), how much revision is really needed?

Well, my polite-but-firm 1500-word missive to the editors of the journal has resulted in immediate action – from my inbox this morning:

Dear Edward

Thank you for your email. We have an editorial board meeting next week during which I will circulate your comments. I will get back to you shortly thereafter.

Best wishes,

XXXX

Now, this guarantees nothing . . . however, now my very detailed response and suggested edits will end up in front of the editorial board, and hopefully I will get a sense of what is actually needed to make this publishable.  I also might get some sense of how they plan to review a resubmission . . . I really don’t want this to take another 5-6 months, and I really don’t want Reviewer 1 to ever see this piece again!

More as it happens . . .

Remember the article on livelihoods I referenced in my last post?  The one that has been through 13-odd reviewers over the past 6 months?  Well, it came back this morning with that least satisfying of responses: revise and resubmit.  The reviewers disagreed on the strengths and weaknesses of the paper, but all of them wanted at least some small changes.

Now, this is hardly the first time I’ve had to deal with this.  Happens all the time.  But this review, like a number of others I have encountered in my career, highlights for me the need for strong editors at journals.  The email from the journal read, in part:

By the time of [the editorial board] meeting, we had received the reports of the external referees. You will find copies of their comments in the attached document. The editors feel that the referees have raised some important points and highlighted some shortcomings in the paper, which would need to be addressed through a round of revisions before we could proceed further. Please note that the editors are forwarding these comments as you might find them helpful; individual comments do not necessarily represent the views of the editors, or a consensus regarding the direction of any possible revision. [my emphasis].

Well, that’s just super.  It amounts to “here are some comments, which might or might not be relevant. Good luck!”  This gives the author nothing to work from, especially when some of the comments are contradictory.

So, a quick lesson for all of you budding academics out there.  Letters like this require a response asking for clarification.  I hammered out an email requesting a conversation with the editor.  Yes, that’s right, you can talk to the editors!  My first paragraph:

Thank you for forwarding the reviewer comments on my paper (Manuscript ID DECH-11-094 entitled “Livelihoods as Governmentality: Reframing the Logic of Livelihoods for Development”).  I appreciate the time and effort the reviewers put into the paper, and the consideration the editorial board has given my submission.  I would very much like to revise and resubmit this paper, but I need a bit of guidance if I am to do so in a productive manner.  Ideally, I would like to talk with the editor in charge of this submission to resolve some of my ideas for revision, and some larger concerns before moving forward.  I have summarized them, in a general sense, below:

(Note that this is polite – always be polite! It’s amazing how many people fail to do this.)

I then wrote 1500 words on what I proposed to do – with great specificity.  I hope to speak directly to the editor myself soon.  That way, I will know how s/he plans to treat the manuscript when they receive revisions, and how s/he views my suggestions – this is huge, as it will tell me what I can and cannot skip in the comments.

Oh, and one of the reviewers was a disaster – did not get the paper, wanted it to be the paper they would have written, etc.  I headed this person off with the following paragraph:

The comments of reviewer 1 are external critique – that is, they demand that the paper be something other than what it is, do not acknowledge the stated goals of the paper, or whether or not the paper achieved those goals.  The reviewer clearly did not engage with the core theory of the paper (for example in arguing the lit review was too confusing and should be cut down), instead demanding it be reframed as a gender analysis (which it is not) with relevance to policy (not the goal of this piece, which is more about foundational theory upon which policy statements might be constructed – policy relevance is also not a criteria for Development and Change).  The reviewer appears to be upset because the paper is not the gender and development paper s/he would have written (incidentally, I already wrote a paper closer to what they wanted and published it in World Development about three years ago – this paper is in part an effort to move past the limitations of that analysis), and instead of looking at the literature I am drawing on, they simply demanded I reframe and cite all the things they think are important.  Generally speaking, I try to take responsibility for moments when the reader becomes confused, treating this as a symptom of unclear writing or thinking on my part – I hope this is clear from my responses to the other reviewers’ comments.  However, the complete disengagement of this review with the paper as it was written, coupled with reviewer 3’s assessment of the paper as well-written, suggests to me that this reviewer’s concerns cannot be addressed without writing a completely different paper – and that paper would not make much of a contribution anymore (something they acknowledge). Suggested edits: none

Yep, you can make the case for the removal of reviewers from a revise and resubmit – especially if they will never approve the paper for what it is.

I’ll keep you posted on progress . . .