Open Review Events (ORE) for Excellence in Digital Scholarship

We have seen decades of reports and institutes designed to ensure that digital scholarship receives credit in evaluation and promotion.  The 2013 “Guidelines for Evaluating Work in Digital Humanities and Digital Media” rightly emphasizes the transformation of “traditional notions of scholarship, teaching, and service”; candidates are advised to explain what they have accomplished in digital projects, indicating how such work may alter these categories of activity.  These points can be pushed further: How can new scholarship collaborate with, and benefit from the rigorous standards of, the old as it redraws disciplinary and evaluative categories?  How can evaluation and peer review be remodeled?
Digital humanities (DH) scholarship not only merges the three areas of evaluation (teaching, research, and service) and blends disciplines (e.g. computer science and anthropology), it also bridges the “cultures” of natural and social sciences and the traditional humanities. The sciences have developed widely accepted peer review for collaborative, vast, technical, long-term projects comparable to DH. Standards of excellence can adapt, in humanities as well as sciences, according to the criterion of reliability (can others attain the same results?) and the ability to produce new kinds of answers to persistent as well as reformulated questions.  All disciplines could benefit from the culture of poster sessions and preprints, in which peer review feeds into the work in progress to improve it.  I do not suggest the humanities need to catch up with sciences (the opposite is often the case), yet some scholarship outside the humanities provides valuable models.  Further, the approaches to evaluation in the arts can also apply in the humanities.

Effective communication and collaboration are needed to realize the goals of the MLA guidelines.  Digital scholars should avoid trumpeting new technology as if it supersedes longstanding methods, knowledge, and theories, which can discourage colleagues from participating.  Any relatively new field must educate not only its peers, evaluators, and current users but also its descendents.  We need to offer more undergraduate and graduate courses in digital humanities within academic departments, where traditional excellence has flourished.  Team teaching, for example by the Romanticist who simply uses email and JStor and the innovator in data mining, would demonstrate the mutual instruction that is essential in digital scholarship itself.  Further, graduate fellowships and tenure track or other appointments need to support the extra time and training needed for excellence in both disciplines, for instance Classics and GIS.  Kathleen Fitzpatrick and others have experimented with open review, and have discussed various options in a prepublication process (http://mcpress.media-commons.org/open-review/contextualizing-questions/why-open-review/).  NINES (www.nines.org) and the other specialist networks of digital scholarship such as http://www.18thconnect.org/ within ARC (Advanced Research Consortium) have long managed anonymous peer review in much the same manner as journals.  THATCamp offers a template for low-cost local unconference events.  I propose an amalgamation of all these approaches in a digital equivalent of an online “book event”: three or four designated mentor-reviewers (anonymous or not)[1]; open access to a project and one or more PIs or participants who volunteer for this event; a blog-like platform that could manage access (from open to private).  Such events, resembling a planning board’s charrette or a studio art review session, could consist of one-on-one consultations with the mentor-reviewers; more public hack-a-thons that try out usability and reliability; comment periods on a blog; virtual meetings or podcasts of the review panel.  The primary goal is enhancing the excellence and viability of the project.  When the professional goals of institutional evaluation and promotion are involved, the format can be adapted for confidentiality.

Alison Booth

University of Virginia

[1] Including a reviewer who does not practice advanced digital research would have a double effect of helping that reviewer to understand the ways of DH as well as encouraging the project to meet the standards of advanced traditional scholarship.