by Kristine M. Bartanen
Kristine Bartanen is academic vice president and dean of the University of Puget Sound in Tacoma, Washington, a position she has held since 2004. She has served Puget Sound as director of forensics, professor and chair of the Communication and Theatre Arts department, associate academic dean, and vice president for student affairs and dean of students. Dr. Bartanen’s work has included particular attention to development of academic-residential programs on the campus, including residential first-year seminars; growth of the interdisciplinary curriculum, most recently in neuroscience; and support of civic scholarship, such as the Sound Policy Institute and the Race and Pedagogy Initiative.
Many liberal arts college faculty members are interested in and increasing their use of digital resources in teaching and scholarly work. Some have been developing digital teaching resources for nearly two decades, some have begun to publish scholarship in on-line journals and other digital venues, and some are doing ground-breaking work in open source, collaborative scholarly projects. Others, particularly pre-tenure or pre-promotion faculty, are reticent to venture into digital work out of concern for how that work will be acknowledged, valued, and rewarded in existing faculty tenure, promotion, and merit award systems. That reticence lives in tension with recognition that advances in technology-enabled teaching and scholarship are progressing in other institutions – academic and non-academic alike – and that professional currency in the academy demands new or amended frameworks in the liberal arts college for evaluation of digital work.
For some, digital scholarship is considered difficult to evaluate because it more often crosses customary boundaries of teaching, scholarship, and service than does traditional scholarship. This territory is a “mixed-use landscape” that liberal arts college faculty members often cross in their careers on teaching-intensive, residential campuses where multiple forms of scholarship are evident in evaluation portfolios, where mentoring of undergraduate research produces co-authored scholarship with students, and where community-based teaching and civic scholarship transcend or elide those same traditional boundaries. My own experience – as a faculty member who always had off-site teaching and administrative components to her portfolio; who chaired a department with multiple programs in which laboratories for learning extended beyond the traditional classroom; and as a dean who has supported academic-residential initiatives, civic scholarship, field school projects, and interdisciplinary work and who has evaluated faculty members for more than a decade – suggests that faculty members on liberal arts campuses can succeed (indeed, in some cases may even be able to lead the way for others) in demonstrating effective use of evaluation guidelines for digital scholarship.
There are challenges, however, of at least two kinds. Evaluation of digital scholarship demands new conceptions of traditional measures of peer review and scholarly authority. Faculty and administrators are challenged to articulate, for example, what standards of peer review are appropriate for digital scholarship; what process revisions are needed to make visible and to communicate the rigor of digitally published or born digital work; how “open” components of digital work such as collaborative authoring, ongoing iteration, absence of page limits or page costs, or markedly different time-to-publication than print monographs intersect with assessments of quality; and what are credible markers of “impact” for digital scholarship. As Michael Jensen has observed:
For universities, the challenge will be ensuring that scholars who are making more and more of their material available online will be fairly judged in hiring and promotion decisions. It will mean being open to the widening context in which scholarship is published, and it will mean that faculty members will have to take the time to learn about — and give credit for — the new authority metrics, instead of relying on scholarly publishers to establish the importance of material for them.
Further, evaluation of digital scholarship demands new conceptions of what constitutes scholarly work. Digital scholars are not only moving work to new venues; nor, as William Thomas III argues, are they merely migrating analog data to digital form. Digital scholars, he summarizes, are engaged in “a humanistic scholarly endeavor, a process of encoding, editing, interpreting, and curating.” They are blurring traditional boundaries of teaching, scholarship, and service, and they are engaged in interdisciplinary, multidisciplinary, and transdisciplinary work. In order to support and reward appropriately such scholarship, evaluators need guidance on how to assess such boundary-crossing work and on what standards account for the scholarly work involved in learning or building new technological tools and archival resources, in analyzing and curating “big data,” as well as in using new tools and resources to extend scholarly arguments and create new knowledge.
The goal of this NITLE project is to provide access to resources that can assist liberal arts college faculties in crafting evaluation criteria and guidelines on their campuses that will appropriately support and encourage digital scholarship. While any given faculty, department, or faculty member might address this task on its own, the existence of framework guidelines – particularly those endorsed by professional groups – will add credibility to new or revised evaluation practices, serve as potential benchmarks for best processes, and reduce redundant efforts.
While the roots of digital scholarship extend back several decades, definitions and descriptions of the work continue to evolve. Collaborative definitions from Wikipedia provide:
Digital scholarship is the use of digital evidence, methods of inquiry, research, publication and preservation to achieve scholarly and research goals. Digital scholarship can encompass both scholarly communication and publication using digital media and research on digital media. . . . Digital scholarship may also include born-digital means of scholarly communication that are more traditional, like online journals and databases, e-mail correspondence and the digital or digitized collections of research and academic libraries.
The Digital Humanities are an area of research, teaching, and creation concerned with the intersection of computing and the disciplines of the humanities. Developing from the fields of humanities computing, humanistic computing, and digital humanities praxis . . . digital humanities embrace a variety of topics, from curating online collections to data mining large cultural data sets. Digital humanities (often abbreviated DH) currently incorporate both digitized and born-digital materials and combine the methodologies from traditional humanities disciplines (such as history, philosophy, linguistics, literature, art, archaeology, music, and cultural studies) and social sciences with tools provided by computing (such as data visualisation, information retrieval, data mining, statistics, text mining) and digital publishing.
Given that “DH” now encompasses areas of study beyond the humanities (and there is evidence that the sciences are wrestling with evolution of impact metrics in the digital age), this paper uses “digital scholarship” as the more inclusive designation, even while recognizing the term is subject to debate.
Need for Tenure and Promotion Guidelines
Literature that calls for, outlines, or discusses guidelines for evaluation of digital scholarship is robust, even as incorporation of such guidelines into campus tenure and promotion processes appears to remain inchoate. In advance of the 2014 MLA national meeting, Mark Sample reported that “digitally-inflected” sessions comprised nearly ten percent of approximately 800 convention slots, which is roughly double the number in 2012. To gain some indication of the degree to which, particularly at liberal arts colleges, faculty members are engaged in digital scholarship, this project included review of two selected membership groups: NITLE members and Annapolis Group institutions. A web survey used institutional home page search boxes to identify “digital scholarship”(DS) or “digital humanities” (DH) endeavors, with activity “counted” if the institution (1) had a dedicated DS/DH website, (2) participated in a consortia-level DS/DH program, (3) listed at least one faculty member with DH as a research interest or area of expertise, (4) offered at least one course with “digital humanities” in the title, (5) hosted a THATCamp, or (6) had been awarded a grant to fund DS/DH activities.
Results are contained in Table 1 and Table 2 and suggest that faculty involvement in digital scholarship is and will be growing in the years ahead. Yet a web search of the 27 institutions listed in Table 1 did not result in finding any evaluation, promotion, or tenure policies specific to digital humanities or digital scholarship. This finding aligns with the conclusion articulated by Sheila Cavanaugh, in the Fall 2012 special issue of the Journal of Digital Humanities devoted to faculty evaluation: “Written guidelines for digital assessment rarely exist and many tenured faculty members remain unable, unwilling, or blind to the need to adapt current promotion criteria to digital scholarship.” Eric Schnell concurs: “Review committees have a difficult time understanding the significance of digital scholarship, let alone knowing how to assess its impact. . . . As a result, emerging forms of digital scholarship are often not defined in criteria documents and therefore not fully valued [in] the faculty rewards system.” Fortunately, resources are available to provide guidance both for scholars engaged in exciting and vibrant digital scholarship and the colleagues called upon to evaluate them fairly and adequately.
The Modern Language Association has crafted evaluation guidelines for digital scholarship which have served as models for other associations and for colleges and departments.
- Guidelines for Evaluating Work in Digital Humanities and Digital Media http://www.mla.org/guidelines_evaluation_digital
The MLA Committee on Information Technology hosts a very comprehensive evaluation wiki (http://wiki.mla.org/index.php/Evaluation_Wiki) that includes materials from periodic MLA evaluation workshops, a “Short Guide to Evaluation of Digital Work,” suggestions for candidates preparing an evaluation file, example guidelines from selected universities, and a list of resource links and bibliography related to digital scholarship in the humanities. Other association guidelines linked there include:
- College Art Association: College Art Association: http://www.collegeart.org/guidelines/newmedia07.html
- Conference on College Composition and Communication:
Institutional evaluation guidelines linked on the MLA evaluation wiki include:
- University of Maine, New Media Department: http://newmedia.umaine.edu/interarchive/promotion_tenure_redefinitions.html
- University of Nebraska-Lincoln, Center for Digital Research in the Humanities: “Promotion & Tenure Criteria for Assessing Digital Research in the Humanities” http://cdrh.unl.edu/articles/promotion_and_tenure.php
- University of Victoria, Faculty of the Humanities: http://humanities.uvic.ca/faculty/evaluation_recognition.php
The American Historical Association adapted the MLA guidelines and, upon recommendation of its American Association for History and Computing, endorsed the following guidelines in January 2002:
- Suggested Guidelines for Evaluating Digital Media Activities in Tenure, Review, and Promotion: An AAHC Document
NINES (Networked Infrastructure for Nineteenth-Century Electronic Scholarship), through summer workshops supported by the National Endowment for the Humanities and involving faculty members (2011) and chairs of literature departments (2012), has drafted guidelines to support literature scholars:
- “Guidelines for Promotion and Tenure Committees in Judging Digital Work” http://institutes.nines.org/docs/2011-documents/guidelines-for-promotion-and-tenure-committees-in-judging-digital-work/
One of NINES’ primary functions is to provide scholarly peer review. These two documents provide additional guidance toward that end:
- Peer Review guidelines http://www.nines.org/about/scholarship/peer-review/
- General Guidelines and Peer Review Criteria for NINES Content http://www.nines.org/about/wp-content/uploads/2011/12/9s-guidelines.doc
In addition to sample guidelines, a summary of components of which is charted in Table 3, leaders in digital scholarship have provided useful presentations and articles that provide guidance in support of appropriate evaluation of this work. The Fall 2012 issue of Journal of Digital Humanities was published as a composite, single location for “proposals, guidelines, and experiences . . . and a living bibliography that will grow as additional examples are published across the web.” Included there is a particularly clear and thorough statement by Todd Presner et al. of the UCLA Digital Humanities Program, “How to Evaluate Digital Scholarship,” that is reproduced with permission in Appendix 1 of this paper. Additional resources are contained in Appendices 3-11.
Successful evaluation is both formative and summative. Evaluation candidates look forward to helpful and detailed feedback; they also expect full and fair consideration for change-of-status recommendations for tenure and promotion. Successful reviews can be supported through three important steps:
- As appropriate for the particular campus, draft departmental or college-wide evaluation guidelines for digital scholarship. On my campus, a faculty code governs faculty evaluation across the campus; in addition, each department and program articulates its own evaluation guidelines, which are subject to review and approval by the faculty’s professional standards committee. We have used addenda to departmental guidelines in cases where unique or distinctive appointments or roles (of which digital scholarship could be an example) require specifically- or specially-defined guidelines. Creation of the new guidelines or an addendum (such as the example in Appendix 2) also brings departmental and/or campus colleagues into conversation, well in advance of the review, which helps to build mutual understanding of expectations.
- Senior colleagues take responsibility for completing and forwarding the guidelines for approval to the faculty’s professional standards committee, faculty senate, or full faculty. Collaboration with educational technology and digital collections librarians in the process of completion can help to insure that system requirements are in place and also reinforces the collaborative nature of digital work. Again, discussion of guidelines or addenda in these broader forums raises awareness and understanding of digital work as a legitimate and valuable form of scholarship.
- Mentor evaluation candidates in the development and preparation of their portfolios. One form of mentorship is to encourage and make space for pre-evaluation digital scholars to share their work in campus seminars or workshops so that familiarity about their work is cultivated prior to the evaluation period. Another source of mentorship is sample evaluation statements made available by digital scholars such as Cheryl Ball, editor of Kairos: A Journal of Rhetoric, Technology and Pedagogy and Katherine D. Harris, who published her research statement in the Fall 2012 issue of Journal of Digital Humanities. A senior campus colleague or another digital scholar on campus, perhaps in another department, could review the clarity of a candidate’s draft evaluation statement and the intersection of the candidate’s presentation of evidence with campus guidelines; offer feedback on the wisdom of including in the file a reading, such as the Presner et al. document appended here; and provide pre-evaluation feedback well in advance of the due dates for submission of evaluation materials. Mentors can also help to temper candidate or colleague expectations that – as captured in a recent Chronicle Vitae article – digital scholars must work twice as hard as “traditional” scholars do to meet evaluation standards.
In the rapidly changing digital environment, this project is necessarily a work-in-progress designed to open conversation. Additional papers and articles, example documents, and approved evaluation guidelines will emerge to join the resource lists. The resources, I hope, will reduce faculty members’ reticence to engage digital scholarship and increase faculty members’ understanding of the nature of digital work. Those two changes will make space for the “larger possibilities” envisioned by Edward Ayers in his recent Educause Review essay:
Digital scholarship can reframe issues of enduring interest with broad arrays of information, it can integrate vast scholarly literature into more useful forms, and it can significantly broaden our temporal or spatial comprehension. In short, digital scholarship needs to do things that simply cannot be done on paper. . . . How can we advance digital scholarship? By thinking of larger possibilities. . . . As we try to foster digital scholarship in the years ahead, we need to begin by understanding the cultural, economic, personal, and institutional world in which the new scholarship will live. That scholarly world defines a problem within a meaningful conversation, it arrays evidence to address the problem, it makes the clearest case for a solution to the problem, and it conveys that case to every relevant audience. Digital scholarship must assume all of those responsibilities if it is to take its place as academic scholarship and if it is to align the core purpose of higher education with the possibilities of our time.
Appropriate and fair evaluation of digital scholarship in the promotion and tenure process is one small, but important step toward such a vision.
Kristine M. Bartanen (email@example.com) is academic vice president and dean, University of Puget Sound, Tacoma, WA, and a 2013-2014 NITLE Fellow. Credit for the literature review and website survey components of this paper belongs to Elizabeth Knight, MLIS, Seattle archivist and librarian, who contributed thorough and efficient research support as well as perceptive insights essential to the project.
Table 1: Digital Scholarship Activity among Annapolis Group and NITLE member Institutions
|Bowdoin College||Digital and Computational Studies Initiativehttp://www.bowdoin.edu/digital-computational-studies/index.shtml|
|Bryn Mawr College||Member of Tri-College DH project, plus visible local activity|
|City University of New York||Digital Humanities Initiative: http://cunydhi.commons.gc.cuny.edu/New faculty: http://www1.cuny.edu/mu/forum/2012/11/02/renowned-digital-humanities-expert-lev-manovich-joining-graduate-center-faculty|
|Bucknell University||Digital Scholarship Initiative https://www.bucknell.edu/newsevents/current-news/2013/july/rethinking-bucknell-in-the-21st-century.html|
|Davidson College||Digital Studies Initiative http://dsi.davidson.edu|
|Dickinson College||Digital Humanities Advisory Committee http://blogs.dickinson.edu/digitalhumanities/|
|Emory University||Center for Digital Scholarship http://digitalscholarship.emory.edu/|
|Hamilton College||Digital Humanities Initiative (DHi) http://www.dhinitiative.org/|
|Harvey Mudd College||Member of CCDH, hired UCLA Center for DH CIO, Joseph Vaughan|
|Haverford College||Member of Tri-College DHDigital Scholarship Program http://library.haverford.edu/services/digital-scholarship|
|Hope College||Mellon Scholars Program, Praxis Network member, William Pannapacker home institution|
|Lafayette College||Mellon DH grant recipient|
|Macalaster College||With Carleton and St. Olaf received Mellon DH grant|
|Middlebury College||DISH: Digital Scholarship Hub http://sites.middlebury.edu/futures/dish-digital-scholarship-hub/|
|Mt. Holyoke College||TEI Initiative and TAPAS involvement https://www.mtholyoke.edu/courses/smoss/TEI_Initiative_MHC.pdf|
|Occidental College||Center for Digital Learning + Research http://www.oxy.edu/center-digital-learning-research|
|Reed College||$800,000 Mellon/$150,000 Keck grants to advance student digital research|
|Richard Stockton College||Center for Digital Humanities https://dh.stockton.edu/|
|St. Olaf College||Digital Humanities on the Hill project receives $700k Mellon granthttp://wp.stolaf.edu/blog/st-olaf-receives-mellon-grant-for-digital-humanities-project/|
|Swarthmore College||Abundant DH activity|
|University of Chicago||Active DH English Department, Annual Chicago Colloquium on DH and Computer Science, ARTFL project|
|University of Richmond||Digital Scholarship Lab http://dsl.richmond.edu/|
|Virginia Tech University||Center for Digital Research and Scholarship http://www.cdrs.lib.vt.edu/|
|Washington and Lee University||DH Working Group http://digitalhumanities.wlu.edu/|
|Wellesley College||Digital Scholarship Initiatives Program http://www.wellesley.edu/lts/about/dsi/|
|Wheaton College (MA)||Digital History Project: http://wheatoncollege.edu/digital-history-project/
Digital Humanities: http://wheatoncollege.edu/technology/academic/technologies/digital-humanities-2/
|Whittier College||DigLibArts Center: http://www.whittier.edu/academics/diglibarts;Center received $750k Mellon grant|
Table 2: Digital Scholarship Activity among selected consortia
|Associated Colleges of the Midwest (14)||Enhancing Midwest Knowledge Eco-system initiative (EMKE), including collaboration in the digital humanities http://www.acm.edu/features/news/477, awarded Mellon Grant for liberal arts – research university collaboration|
|Claremont University Consortium (7)||Center for Digital Initiatives http://libraries.claremont.edu/CDI/default.asp|
|Five College Consortium (5)||Five College Digital Humanities Program https://www.fivecolleges.edu/dh|
|Five Colleges of Ohio Consortium (5)||Awarded $775,000 Mellon Grant for digital collections and programs: http://www.wooster.edu/news/releases/2013/may/mellon-grant|
|Great Lakes Colleges Association (12)||Digital Liberal Arts Initiative http://www.hope.edu/2013/10/01/william-pannapacker-direct-glca-digital-liberal-arts-initiative|
|New York Six Colleges Consortium (6)||New York Six Digital Humanities Consortium http://www.dhinitiative.org/|
|Tri College Digital Humanities (3)||Tri College Digital Humanities http://tdh.brynmawr.edu/|
Table 3: Chart of key components to support guideline development
|Organization or Institutional Guidelines:||MLA||AHA||Maine||Nebraska||NINES/NEH||Presner et al.|
|Department: Delineate and communicate responsibilities at hire||X||X|
|Department: Engage qualified peer reviewers||X||X||X||X||X|
|Faculty member: ask about evaluation and support||X||X|
|Faculty member: Negotiate and document your role||X||X|
|Faculty member: Document and explain your work||X||X||X||X|
|Evaluators: Review work in the medium in which it was developed||X||X||X||X||X|
|Evaluators: Recognize interdisciplinary dimensions of digital work||X||X||X|
|Evaluators: Recognize intrinsically collaborative nature of digital projects||X||X||X|
|Evaluators: Recognize iterative nature of projects and assess such work in the context of appropriate stage of development||X|
|Evaluators: Recognize multiple metrics/measures of impact and multiple dimensions of intellectual rigor||X||X||X|
|Evaluators: Recognize work crosses boundaries of teaching, research, and service||X|
|Evaluators: Recognize experimentation and risk-taking, including negative results||X|
|Evaluators: Avoid approximating equivalencies with print artifacts||X|
|All: Stay informed about accessibility issues||X||X|
|All: See also a checklist of questions included in these example guidelines||X||X||X|
Distributed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
Appendix 1: “How to Evaluate Digital Scholarship”
From “A Short Guide to Digital Humanities.” In the cooperative and open access book by Peter Lunenfeld, Anne Burdick, Johanna Drucker, Todd Presner, and Jeffrey Schnapp, Digital_Humanities. Cambridge, Mass.: MIT Press, 2012; pp. 128-129. Open access edition (PDF e-book) at: https://mitpress.mit.edu/books/digitalhumanities-0https://mitpress.mit.edu/books/digitalhumanities-0.
The document is also available in the Fall 2012 Journal of Digital Humanities (http://www.journalofdigitalhumanities.org). Lead author Todd Presner affirmed in personal correspondence (3/13/14) that while the document has not been formally adopted by any institutional body at UCLA, it has informed discussion at all levels, particularly at the departmental level where it has been used in tenure and promotion cases. Presner and collaborators are affiliated with the UCLA Digital Humanities program (http://www.digitalhumanities.ucla.edu).
This text provides a set of guidelines for the evaluation of digital scholarship in the humanities, social sciences, arts, and related disciplines. The guidelines are aimed, foremost, at academic review committees, chairs, deans, and provosts who want to know how to assess and evaluate digital scholarship in the hiring, tenure, and promotion process.
The list is also intended to inform the development of institution-wide policies for supporting and evaluating scholarship and creative work that reflects traditional values while incorporating specific understandings of new platforms and formats.
Fundamentals for initial review
The work must be evaluated in the medium in which it was produced and published. If it is a website, that means viewing it in a browser with the appropriate plug-ins necessary for the site to work. If it is a virtual simulation model, that may mean going to a laboratory outfitted with the necessary software and projection systems to view the model. Work that is time-based—such as videos—will often be represented by stills, but reviewers also need to devote attention to clips in order to fully evaluate the work. The same can be said for interface development, since still images cannot fully demonstrate the interactive nature of interface research. Authors of digital works should provide a list of system requirements (both hardware and software, including compatible browsers, versions, and plug-ins) for viewing the work. It is incumbent upon academic personnel offices to verify that the appropriate technologies are available and installed on the systems that will be used by the reviewers before they evaluate the digital work.
Digital projects are often collaborative in nature, involving teams of scholars who work together in different venues over various periods of time. Authors of digital works should provide a clear articulation of the role or roles that they have played in the genesis, development, and execution of the digital project. It is impractical—if not impossible—to separate out every micro-contribution made by team members since digital projects are often synergistic, iterative, experimental, and even dynamically generated through ongoing collaborations. Nevertheless, authors should indicate the roles that they played (and time commitments) at each phase of the project development. Who conceptualized the project and designed the initial specifications (functional and technical)? Who created the mock-ups? Who wrote the grant proposals or secured the funding that supported the project? What role did each contributor play in the development and execution of the project? Who authored the content? Who decided how that content would be accessed, displayed, and stored? What is the “public face” of the project and who represents it and how?
Digital projects vary tremendously and may not “look” like traditional academic scholarship; at the same time, scholarly rigor must be assessed by examining how the work contributes to and advances the state of knowledge in a given field or fields. What is the nature of the new knowledge created? What is the methodology used to create this knowledge? It is important for review committees to recognize that new knowledge is not just new content but also new ways of organizing, classifying, and interacting with content. This means that part of the intellectual contribution of a digital project is the design of the interface, the database, and the code, all of which govern the form of the content. Digital scholars are not only in the position of doing original research but also of inventing new scholarly platforms. Five hundred years of print have so fully naturalized the “look” of knowledge that it may be difficult for reviewers to fully understand these new forms of documentation and the intellectual effort that goes into developing them. This is the dual burden—and the dual opportunity—for creativity in the digital domain.
Crossing research, teaching, and service
Digital projects almost always have multiple applications and uses that enhance research, teaching, and service. Digital research projects can make transformative contributions in the classroom and sometimes even have an impact on the public-at-large. This ripple effect should not be diminished. Review committees need to be attentive to colleagues who dismiss the research contributions of digital work by cavalierly characterizing it as a mere “tool” for teaching or service. Tools shape knowledge, and knowledge shapes tools. But it is also important that review committees focus on the research contributions of the digital work by asking questions such as the following: How is the work engaged with a problem specific to a scholarly discipline or group of disciplines? How does the work reframe that problem or contribute to a new way of understanding the problem? How does the work advance an argument through both the content and the way the content is presented? How is the design of the platform an argument? To answer this last question, review committees might ask for documentation describing the development process and design of the platform or software, such as database schemata, interface designs, modules of code (and explanations of what they do), as well as sample data types. If the project is, in fact, primarily for teaching, how has it transformed the learning environment? What contributions has it made to learning and how have these contributions been assessed?
Digital projects should be peer-reviewed by scholars in fields who are able to assess the project’s contribution to knowledge and situate it within the relevant intellectual landscape. Peer review can happen formally through letters of solicitation but can also be assessed through online forums, citations, and discussions in scholarly venues, by grants received from foundations and other sources of funding, and through public presentations of the project at conferences and symposia. Has the project given rise to publications in peer-reviewed journals or won prizes by professional associations? How does it measure up to comparable projects in the field that use or develop similar technologies or similar kinds of data? Finally, grants received are often significant indicators of peer review. It is important that reviewers familiarize themselves with grant organizations across schools and disciplines, including the humanities, the social sciences, the arts, information studies and library sciences, and the natural sciences, since these are indicators of prestige and impact.
Digital projects can have an impact on numerous fields in the academy as well as across institutions and even the general public. They often cross the divide that arises among research, teaching, and service in innovative ways. Impact can be measured in many ways, including the following: support by granting agencies or foundations, number of viewers or contributors to a site and what they contribute, citations in both traditional literature and online (blogs, social media, links, and trackbacks), use or adoption of the project by other scholars and institutions, conferences and symposia featuring the project, and resonance in public and community outreach (such as museum exhibitions, public policy impact, adoption in curricula, and so forth).
Is a digital research project “equivalent” to a book published by a university press, an edited volume or a research article? These sorts of questions are often misguided since they are predicated on comparing fundamentally different knowledge artifacts and, perhaps more problematically, consider print publications as the norm and benchmark from which to measure all other work. Reviewers should be able to assess the significance of the digital work based on a number of factors: the quality and quantity of the research that contributed to the project; the length of time spent and the kind of intellectual investment of the creators and contributors; the range, depth, and forms of the content types and the ways in which this content is presented; and the nature of the authorship and publication process. Large-scale projects with major funding, multiple collaborators, and a wide-range of scholarly outputs may justifiably be given more weight in the review and promotion process than smaller-scale or short-term projects.
Development cycles, sustainability, and ethics
It is important that review committees recognize the iterative nature of digital projects, which may entail multiple reviews over several review cycles, as projects grow, change, and mature. Given that academic review cycles are generally several years apart (while digital advances occur more rapidly), reviewers should consider individual projects in their specific contexts. At what “stage” is the project in its current form? Is it considered “complete” by the creators, or will it continue in new iterations, perhaps through spin-off projects and further development? Has the project followed the best practices, as they have been established in the field, in terms of data collection and content production, the use of standards, and appropriate documentation? How will the project “live” and be accessible in the future, and what sort of infrastructure will be necessary to support it? Here, project specific needs and institutional obligations come together at the highest levels and should be discussed openly with deans and provosts, library and IT staff, and project leaders. Finally, digital projects may raise critical ethical issues about the nature and value of cultural preservation, public history, participatory culture and accessibility, digital diversity, and collection curation which should be thoughtfully considered by project leaders and review committees.
Experimentation and risk-taking
Digital projects in the humanities, social sciences, and arts share with experimental practices in the sciences a willingness to be open about iteration and negative results. As such, experimentation and trial-and-error are inherent parts of digital research and must be recognized. The processes of experimentation can be documented and can prove to be essential in the long-term development process of an idea or project. White papers, sets of best practices, new design environments, and publications can result from such projects, and these should be considered in the review process. Experimentation and risk-taking in scholarship represent the best of what the university, in all its many disciplines, has to offer society. To treat scholarship that takes on risk and the challenge of experimentation as an activity of secondary (or no) value for promotion and advancement can only serve to reduce innovation, reward mediocrity, and retard the development of research.
Appendix 2: Sample addendum for evaluating a digital scholar
ADDENDUM TO THE [INSERT NAME] DEPARTMENT EVALUATION STANDARDS
FOR FACULTY MEMBERS ENGAGED IN DIGITAL SCHOLARSHIP
Approved by the [insert name] Department: ______ (date)
Approved by the Professional Standards Committee: _____ (date)
The faculty member engaged in digital scholarship should satisfy each evaluation criterion – teaching effectiveness, scholarship, and service to the department, college/university, and community – at the same level of quality expected of colleagues. Due to distinctive features of digital scholarship, however, evidence of scholarship and markers of quality may distinguish the digital scholar from colleagues. Evaluation of the digital scholar offers unique opportunities and requires unique delineations because, by definition and practice, the digital scholar’s role is one that may often challenge traditional academic categories and metrics for evaluation. The existence of these distinctions, however, does not lessen the excellence of digital scholarship. The faculty member under review must provide evidence that her or his work as a digital scholar meets standards of quality prevalent among the communities of digital scholars in his or her disciplinary or interdisciplinary field.
This evaluation addendum is informed by digital scholarship evaluation guidelines endorsed by the [insert relevant professional association(s)].
Colleagues who participate in the review of a faculty member engaged in digital scholarship, whether at the department or college/university level, should be prepared to review digital work in the medium in which it was created and published. This means both that the faculty member being reviewed will be expected to provide accessible links to digital work and information on system requirements for viewing it. The review committee chair will be expected to work with the Office of the Dean to insure that appropriate technologies are available and in working order for reviewers before they evaluate the digital work.
If external peer review of scholarship is required for the evaluation or, if not required, such peer review is desired by the faculty member being evaluated, those reviewers should similarly be expected to access digital work in its medium with appropriate systems needs in place. Furthermore, if external review is required, at least [#] of the standard number of such reviewers should be peers engaged in digital scholarship.
Definition of work
The faculty member engaged in digital scholarship may demonstrate excellence through such traditional vehicles as publication in print journals or monographs; presentations at conventions, conferences, workshops or similar forums; publication of instructional materials; and book of materials reviews. The digital scholar may also demonstrate excellence by publishing in digital media, such as online journals and databases; conducting research or creating digital media, such as digitized collections of materials; curating online collections; or mining large cultural data sets. The faculty member engaged in digital scholarship should be expected to explain and document the work being pursued, to discuss its distinctive components and stage(s) of development, and to both articulate appropriate metrics for and evidence of the impact of the work.
The reviewers of the digital scholar’s portfolio should expect to see:
- work that is collaborative (often with colleagues who may be in staff roles in library or IT organizations);
- work that may be interdisciplinary, multidisciplinary, or trans-disciplinary;
- work that may involve experimentation and risk-taking; and
- the faculty member and peer reviewers discuss these dimensions of the work.
The reviewers of digital scholarship should avoid:
- Attempting to equate digital products with traditional print products;
- Expecting the digital scholar to do more work than other colleagues, but rather expect that quantity of work may differ even as quality of work is expected to be high;
- Dismissing digital work that includes teaching about new forms of scholarship, or service to the development of new scholarly communication organizations, as not also scholarly in its contribution to the academy.
Questions and notice
Departmental or college/university-level questions about this addendum should be resolved prior to the initiation of the review process. The addendum itself should be completed and approved as early as possible in a digital scholar’s appointment, preferably by the end of the first-year and no later than two years prior to a scheduled promotion or tenure review.
Appendix 3: Selected resources for evaluation guideline development
Braun, Catherine C. Cultivating Ecologies for Digital Media Work: The Case of English Studies. 2014. Southern Illinois University Press.
Note especially, Chapter 3, “Scholarship through a New Lens: Digital Production and New Models of Evaluation,” pp. 91-131which contains questions to guide departmental considerations of evaluation of digital scholarship.
Byerly, Alison. “Evaluating Digital Scholarship.” NITLE Seminar, October 10, 2012. http://www.slideshare.net/nitle/byerly-nitle-digital-scholarship
Slides 4-6 summarize that “Good evaluation depends on shared context: clear expectations, defined roles, recognized measures of success” and outline essential tasks for scholars and evaluators.
Fitzpatrick, Kathleen. “Beyond Metrics: Community Authorization and Open Peer Review.” In Debates in the Digital Humanities. 2012. University of Minnesota Press. http://dhdebates.gc.cuny.edu/debates#text/7
Discusses ways both scholars engaged in digital work and evaluation committees can come to understand and articulate the values of open vs. traditional peer review.
Fitzpatrick, Kathleen. “Peer Review, Judgment, and Reading.” Profession. Modern Language Association of America. 2011. http://humanities.case.edu/digital/evaluatingDigital_6.pdf
Discusses traditional and new forms of review of scholarship in tenure and promotion evaluations, with emphasis on the responsibility of reading and on how to read open peer review of digital scholarship.
Flanders, Julia. “Time, Labor, and ‘Alternate Careers’ in Digital Humanities Knowledge Work.” In Debates in the Digital Humanities. 2012. University of Minnesota Press. http://dhdebates.gc.cuny.edu/debates/text/26
Discusses the “para-academic” roles involved in digital humanities work (e.g., graduate assistant, consultant, freelancer, technology services staff, adjunct faculty member), how those roles intersect with and are valued relative to the traditional “faculty” paradigm, and future evolution likely in collaborative digital humanities work.
The Journal of American History. Website Reviews. http://www.journalofamericanhistory.org/submit/websitereviews.html
Outlines types of websites and provides general guidelines for review of them:
- Content: Is the scholarship sound and current? What is the interpretation or point of view?
- Form: Is it clear? Easy to navigate? Does it function effectively? Does it have a clear, effective, and original design? Does it have a coherent structure?
- Audience/Use: Is it directed at a clear audience? Will it serve the needs of that audience?
- New Media: Does it make effective use of new media and new technology? Does it do something that could not be done in other media—print, exhibition, film?
Lunenfeld, Peter, Anne Burdick, Johanna Drucker, Todd Presner and Jeffrey Schnapp. Digital Humanities. Cambridge, Mass.: MIT Press, 2012. PDF eBook http://mitpress.mit.edu/books/digitalhumanities-0
“… [a] report on the state of contemporary knowledge production… explores methodologies and techniques unfamiliar to traditional modes of humanistic inquiry – including geospatial analysis, data mining, corpus linguistics, visualization, and simulation… the authors argue that the digital humanities offers a revitalization of the liberal arts tradition…”
Williford, Christa and Charles Henry. 2012. One Culture: Computationally Intensive Research in the Humanities and Social Sciences. http://www.clir.org/pubs/reports/pub151
Based on eight international projects involving computational analysis of large data sets, this resource offers recommendations to enable universities “to adapt to, support, or sustain this emerging research over time.”
Appendix 4: Key Digital Humanities Organizations
Alliance of Digital Humanities Organizations: http://adho.org/
American Academy of Arts and Sciences, Humanities Indicators: http://www.humanitiesindicators.org/
Association for Computers and the Humanities, hosted by Boston College: http://ach.org/
CenterNet: International Network of Digital Humanities Centers: http://digitalhumanities.org/centernet/
North America: http://digitalhumanities.org/centernet/centers
Council on Library and Information Resources, Digging into Data Challenge (NEH): http://www.diggingintodata.org/
June 2012 report: http://www.clir.org/pubs/reports/pub151
Recommended readings: http://devdh.org/recommended-readings/
Digital Humanities Now: http://digitalhumanitiesnow.org/
A search for “evaluating digital scholarship” provides a chronological list of resources.
DH Commons: http://dhcommons.org/
Harvard MetaLab http://metalab.harvard.edu/
Institute for Computing in Humanities, Arts, and Social Science (iCHASS): http://chass.illinois.edu/
Institutes for Advanced Topics in the Digital Humanities (NEH): http://www.neh.gov/grants/odh/institutes-advanced-topics-in-the-digital-humanities
NEH Office of Digital Humanities: http://www.neh.gov/divisions/odh
NITLE Digital Humanities Council:
Open Scholar Foundation: http://www.force11.org/node/4383
Open Scholar platform: http://openscholar.harvard.edu/
Princeton Digital Humanities Initiative (2013) http://digitalhumanities.princeton.edu/
Appendix 5: ITHAKA and ITHAKA S+R resources on digital humanities
These sites do not appear to contain content focused specifically or primarily on faculty evaluation. Below are reports pertaining to digital humanities, faculty research support, or sustaining local digital scholarship.
- Sustaining the Digital Humanities. 2013. http://www.sr.ithaka.org/research-publications/sustaining-digital-humanities
NOTE: A final report will be issued in Spring 2014, and will be accompanied by a toolkit to support campus administrators in developing a digital strategy for supporting digital humanities projects at their institution.
- ARL Appraising our Digital Investment. 2013. http://www.sr.ithaka.org/research-publications/appraising-our-digital-investment
- Research Support Services for Scholars. 2013. http://www.sr.ithaka.org/research-publications/research-support-services-scholars
- Host Institution Strategies for Sustaining Digital Resources. 2012. http://www.sr.ithaka.org/research-publications/host-institution-strategies-sustaining-digital-resources
- Supporting the Changing Research Practices of Historians. 2012. http://www.sr.ithaka.org/research-publications/supporting-changing-research-practices-historians
- U.S. Faculty Survey 2012. http://www.sr.ithaka.org/research-publications/us-faculty-survey-2012
- Local Faculty Survey tool. 2012. http://sr.ithaka.org/local-faculty-survey
- RLUK: UK Survey of Academics 2012. http://www.sr.ithaka.org/research-publications/ithaka-sr-jisc-rluk-uk-surveyacademics-2012
- Department of Digital Humanities at King’s College London 2011. http://www.sr.ithaka.org/research-publications/ddh-kings-college-london
- Sustaining Digital Resources… 2009. http://www.sr.ithaka.org/research-publications/sustaining-digital-resources-ground-view-projects-today
Appendix 6: Selected Digital Scholarship Tools
- Altmetrics http://altmetrics.org/manifesto/
- Anthologize http://anthologize.org/
- Bamboo DiRT Digital Research Toolshttp://dirt.projectbamboo.org/
- Center for History and New Media Tools http://chnm.gmu.edu/research-and-tools/
- Cohen, Daniel and Roy Rosenzweig. 2005. Digital History: A Guide to Gathering, Preserving and Presenting the Past on the Web. University of Pennsylvania Press. http://chnm.gmu.edu/digitalhistory/
- DM Digital Annotation Tool http://dm.drew.edu/dmproject/
- Tapas Project http://tapasproject.org/
- Text Encoding Initiative http://www.tei-c.org/index.xml
Appendix 7: Additional literature on faculty evaluation of digital scholarship
Andersen, D. L. (ed.) Digital Scholarship in the Tenure, Promotion and Review Process. 2003. http://www.mesharpe.com/mall/resultsa.asp?Title=Digital+Scholarship+in+the+Tenure%2C+Promotion%2C+and+Review+Process
“…examines the evolution of nontraditional scholarship, analyzes the various formats, and suggests guidelines for assessment on a scholarly level…” Part III Section 11.: The Development of Criteria for the Inclusion of Digital Publications in the Tenure Process: A Case Study of Washington State University Libraries Ryan Johnson…”
Clement, Tanya E. “Half Baked: The State of Evaluation in the Digital Humanities.” American Literary History. vol. 24, no. 4, October 2012, pp. 876–890. doi:10.1093/alh/ajs051. Review of:
The American Literature Scholar in the Digital Age. 2011. Amy E. Earheart and Andy Jewell, eds. University of Michigan Press and University of Michigan Library.
Switching Codes: Thinking Through Digital Technology in the Humanities and the Arts. 2011.Edited by Thomas Bartscherer and Roderick Coover, eds. University of Chicago Press.
“… The American Literature Scholar in the Digital Age and Switching Codes are unique and valuable as collections whose historicizing illuminates current fissures in discussions about the ways that the academic community can evaluate digital scholarship in the humanities…”
Cheverie, Joan F., Jennifer Boettcher and John Buschman. 2009. “Digital Scholarship in the University Tenure and Promotion Process: A Report on the Sixth Scholarly Communication Symposium at Georgetown University Library.” Journal of Scholarly Publishing. April 2009. doi: 10.3138/jsp.40.3.219.
“… Four notable scholars who have done significant work in digital scholarly projects were invited to speak on the theme of ‘digital scholarship in the university tenure and promotion process’ at Georgetown University Library to explore the scholarship and the continuing problems with evaluating it – particularly for promotion and tenure…”
Liu, Alan. December 2011. “The state of the digital humanities: A report and a critique.” Arts and Humanities in Higher Education. DOI: 10.1177/1474022211427364
“… the scholarly field of the digital humanities has recently expanded and integrated its fundamental concepts, historical coverage, relationship to social experience, scale of projects, and range of interpretive approaches…[the field] has the potential not just to facilitate the work of the humanities but to represent the state of the humanities at large in its changing relation to higher education in the postindustrial state…”
MLA Profession. 2011.Evaluating Digital Scholarship (contains six articles, pp. 123-196, focused on the topic). http://www.mlajournals.org/toc/prof/2011/1
“There is a growing consensus that humanities disciplines must find ways not simply of evaluating but also of valuing digital scholarship as part of hiring, promotion, and tenure decisions. National scholarly organizations such as the Modern Language Association and the American Council of Learned Societies have called for departments and institutions to “recognize the legitimacy of scholarship produced in new media, whether by individuals or in collaboration, and create procedures for evaluating these forms of scholarship” (Report of the MLA Task Force).”
Purdy, James P. and Joyce R. Walker. 2010. “Valuing Digital Scholarship: Exploring the Changing Realities of Intellectual Work.” Profession. 19. pp. 177-195. https://docs.google.com/viewer?url=http%3A%2F%2Fdmp.osu.edu%2Fdmac%2Freadings%2FPurdyWalker.pdf
“…many tenure guidelines . . . label research as either creative or scholarly,” counting only the scholarly…. This narrow binary is perhaps the most significant (and problematic) aspect of current attitudes regarding the value of digital scholarship… One way to evaluate scholarly production that avoids a simple print-digital binary opposition is to think … more about what it produces, participates in, or does. To develop a more robust, complex evaluation framework, we might ask these questions:…Such an approach to assessment would not look very much like the tenure-and-promotion activities now in place at most institutions.”
Richardson, James. 2013. “Establishing a New Paradigm: the Call to Reform the Tenure and Promotion Standards for Digital Media Faculty.” Journal of Interactive Technology and Pedagogy. Issue 3.
“… in the case of new and evolving fields of study, there are alternative criteria that would be better suited for the digital disciplines, and would serve as a more accurate assessment on the quality of faculty scholarship as they march towards tenure, promotion and reappointment….”
Schreibman, Susan and Ann M. Hanlon. 2010. “Determining Value for Digital Humanities Tools: A Report on a Survey of Tool Developers.” Digital Humanities Quarterly. Volume 4 Number 2. http://digitalhumanities.org:8080/dhq/vol/4/2/000083/000083.html
“… the authors conducted an online survey of developers of digital humanities tools in March 2008. The survey focused on their perceptions of their work, how they felt their tool development fit into a structure of academic rewards, and the value of tool development as a scholarly pursuit. Survey results indicate that tool development is indeed considered a scholarly activity by developers, but recognition of this work and rewards for it lag behind rewards for traditional scholarly pursuits…”
Zorich, Diane M. 2012. “Transitioning to a Digital World: Art History, Its Research Centers, and Digital Scholarship.” A Report to The Samuel H. Kress Foundation and The Roy Rosenzweig Center for History and New Media, George Mason University. http://www.academia.edu/2973600
Appendix 8: Selected faculty perspectives on evaluation of digital scholarship
Davidson, Cathy. 2009. “Respecting the Meaning of Tenure.”
Davidson, Cathy. September 2012. “How Can a Digital Humanist Get Tenure?” http://www.hastac.org/blogs/cathy-davidson/2012/09/17/how-can-digital-humanist-get-tenure
Harley, Diane, et al. 2010. Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Values and Needs in Seven Disciplines. Center for Studies in Higher Education. http://escholarship.org/uc/cshe_fsc
Includes faculty responses from fields of archeology, astrophysics, biology, economics, history, music, and political science.
Jaschik, Scott. 2009. “Tenure in the Digital Era.” http://www.insidehighered.com/news/2009/05/26/digital
Kolowich, Steve. 2012. “The Promotion That Matters.”
Koh, A. 2012. “The Challenges of Digital Scholarship: A Report on the MLA Preconference on Evaluating Digital Work for Promotion andTenure.”http://chronicle.com/blogs/profhacker/the-challenges-of-digital-scholarship/38103
Lee, Valerie and Cynthia L. Selfe. 2008. “Our Capacious Caper: Exposing Print-Culture Bias in Departmental Tenure Documents. ADE Bulletin, No 145. http://susandelagrange.com/cccc/Lee&Selfe.pdf
Rhee, Jennifer. 2012. “More Ammo: Digital Scholarship and Activity in Tenure and Promotion.”
Richardson, James. “Redefining the Standards of Tenure and Promotion for Multi-Media and Digital Arts Faculty. ”http://www.academia.edu/1684630/Redefining_the_Standards_of_Tenure_and_Promotion_For_Multimedia_and_Digital_Arts_Faculty
Roy, Michael. 2013. “Is Linking Thinking? Addressing and Assessing Scholarship in the Digital Era.”
Schnell, Eric. 2013. “Digital Scholarship and the Faculty Reward System.”http://library.osu.edu/blogs/digitalscholarship/2013/04/29/digital-scholarship-and-the-faculty-rewards-system/#more-405
Schreibman, Susan, Laura Mandell, Stephen Olsen. 2011. “Evaluating Digital Scholarship: A Case Study in the Field of Literature.” Digital Humanities. http://dh2011abstracts.stanford.edu/xtf/view?docId=tei/ab-142.xml;query=;brand=default
Starkman, Ruth. 2013. “What Counts?” http://www.insidehighered.com/advice/2013/02/20/essay-issues-related-what-digital-scholarship-counts-tenure-and-promotion
Takats, Sean. 2013. “A Digital Humanities Tenure Case, Part 2: Letters and Committees.”
Appendix 9: Literature on evaluation metrics in science
Boardman, P. Craig and Branco L. Ponomariov. 2007. “Reward Systems and NSF University Research Centers: The Impact of Tenure on University Scientists’ Valuation of Applied and Commercially Relevant Research.” The Journal of Higher Education. 78(1).
“…We performed a principal component analysis of the rankings produced by 39 existing and proposed measures of scholarly impact that were calculated on the basis of both citation and usage log data. Our results indicate that the notion of scientific impact is a multi-dimensional construct that cannot be adequately measured by any single indicator, although some measures are more suitable than others. The commonly used citation Impact Factor is not positioned at the core of this construct, but at its periphery, and should thus be used with caution…”
Cacioppo, John, T. 2013. Metrics of Science. Association for Psychological Science.
Howard, Jennifer. 2013. “Rise of ‘Altmetrics’ Revives Questions About How to Measure Impact of Research.” Chronicle of Higher Education. http://chronicle.com/article/Rise-of-Altmetrics-Revives/139557/
Lozano, G.A., Lariviè, V., & Gingras, Y. 2012. “The weakening relationship between the Impact Factor and papers’ citations in the digital age.” Journal of the American Society for Information Sciences and Technology. 63(11): 2140-2145. doi: 10.1002/asi.22731
NSF Workshop on “Scholarly Evaluation Metrics: Opportunities and Challenges.” 2009. http://informatics.indiana.edu/scholmet09/announcement.html
“… we have seen a rapid expansion of proposed metrics to evaluate scientific impact. This expansion has been driven by interdisciplinary work in web, network and social network science, e.g. citation PageRank, h-index, and various other social network metrics. Second, new data sets such as usage and query data, which represent aspects of scholarly dynamics other than citation, have been investigated as the basis for novel metrics. The COUNTER and MESUR projects are examples in this realm. And, third, an interest in applying Web reputation concepts in the realm of scholarly evaluation has emerged and is generally referred to a Webometrics….”
Piwowar, H. 2013. “Altmetrics: Value all research products.” Nature. 493(159). doi:10.1038/493159a.
“Science Metrics.” 2010. Nature. http://www.nature.com/news/specials/metrics/index.html
Appendix 10: Additional articles related to digital scholarship and faculty evaluation
Arbaugh, J. Ben. 2009. “Digital Scholarship in the Tenure, Promotion, and Review Process.” Academy of Management Learning and Education. 8:3 460-462.
Borgman, Christine L. 2008. “Supporting the ‘Scholarship’ in E-Scholarship.” Educause Review 43, no. 6: 32-33. http://www.educause.edu/ero/article/supporting-%E2%80%9Cscholarship%E2%80%9D-e-scholarship
Buller, Jeffrey L. 2007. “Improving Documentation for Promotion and Tenure.” Academic Leader 23, no. 11: 7-8.
Collins, Ellen; Monica E. Bulger and Eric T. Meyer. 2011. “Discipline matters: Technology use in the humanities.” Arts and Humanities in Higher Education 2012 11: 76. DOI: 10.1177/1474022211427421
Burgess, Helen J. and Jeanne Hamming. 2011. “New Media in the Academy: Labor and the Production of Knowledge in Scholarly Multimedia.” Digital Humanities Quarterly. 5(3).
Cross, Jeanne G. 2008. “Reviewing Digital Scholarship: The Need for Discipline-Based Peer Review.” Journal Of Web Librarianship 2, no. 4: 549-566. doi: 10.1080/19322900802473936
Delagrange, Susan H. 2009. “When Revision Is Redesign: Key Questions for Digital Scholarship.” Kairos: A Journal Of Rhetoric, Technology, And Pedagogy 14, no. 1
Diamond, Robert M. and Bronwyn E. Adam. 2004. “Balancing Institutional, Disciplinary and Faculty Priorities with Public and Social Needs: Defining scholarship for the 21st century.” Arts & Humanities in Higher Education. vol 3(1) 29–40 doi: 10.1177/147402204039643
Filetti, Jean S. 2009. “Assessing service in faculty reviews: mentoring faculty and developing transparency.” Mentoring & Tutoring: Partnership In Learning 17, no. 4: 343-352.
Hardré, Patricia, and Michelle Cox. 2009. “Evaluating faculty work: expectations and standards of faculty performance in research universities.” Research Papers in Education 24, no. 4: 383-419.
Howard, Jennifer. 2010. “Hot Type: No Reviews of Digital Scholarship = No Respect.”
Howard, Jennifer. 2008. “A New Field Study Identifies Eight Major Types of Digital Scholarship.” Chronicle Of Higher Education 55, no. 13: A11.
NEH Grantees Experiment with New Kinds of Peer Review. 2010. http://www.neh.gov/divisions/odh/featured-project/neh-grantees-experiment-new-kinds-peer-review
O’Meara, Kerry Ann. 2005. “Encouraging multiple forms of scholarship in faculty reward systems: Does It Make a Difference?” Research in Higher Education 46, no. 5: 479-510.
“This article presents findings from a national study of Chief Academic Officers of 4-year institutions on the impact of policy efforts to encourage multiple forms of scholarship in faculty roles and rewards….CAOs at campuses that initiated reforms reported a greater congruence between faculty priorities and institutional mission, and greater improvement in attention to undergraduate learning over the last decade.”
Peer Review in Academic Promotion and Publishing: Its Meaning, Locus, and Future. 2011. Center for Studies in Higher Education.
Full text: http://escholarship.org/uc/item/1xv148c8#page-1
“… This report includes (1) an overview of the state of peer review in the Academy at large, (2) a set of recommendations for moving forward, (3) a proposed research agenda to examine in depth the effects of academic status-seeking on the entire academic enterprise… The document explores, in particular, the tightly intertwined phenomena of peer review in publication and academic promotion, the values and associated costs to the Academy of the current system, experimental forms of peer review in various disciplinary areas, the effects of scholarly practices on the publishing system…”
Raben, Joseph. 2007. “Tenure, Promotion and Digital Publication.” Digital Humanities Quarterly. Volume 1, Number 1. http://www.digitalhumanities.org/dhq/vol/001/1/000006/000006.html
Zorich, D.M. 2008. A Survey of Digital Humanities Centers in the United States. https://docs.google.com/viewer?url=http%3A%2F%2Fwww.clir.org%2Fpubs%2Freports%2Fpub143%2Fpub143.pdf Washington D.C.: Council on Library and Information Resources.
Discusses criteria for project selection by digital humanities centers (see Section 4.4.3) as well as means for measure project success (see: Section 4.4.4).
Appendix 11: Additional literature on digital scholarship
The Credibility of Electronic Publishing: A Report to the Humanities and Social Sciences. Federation of Canada. 2001. Raymond Siemens, Project Coordinator.
Fitzpatrick, Kathleen. 2011. Planned Obsolescence. http://simon.ups.edu/record=b1807771~S0
Fitzpatrick, Kathleen. 2011. “The Humanities, Done Digitally.” Chronicle of Higher Education 57, no. 36: B26.
Heap, Tania, and Shailey Minocha. 2012. “An Empirically Grounded Framework to Guide Blogging for Digital Scholarship.” Research in Learning Technology 20, 176-188.
Kuhn, Virginia. 2013. “Embrace and Ambivalence.” Academe 99, no. 1: 8-13.
Littlejohn, A., H. Beetham, and L. McGill. 2012. “Learning at the digital frontier: a review of digital literacies in theory and practice.” Journal of Computer Assisted Learning 28, no. 6: 547-556.
Losoff, Barbara, and Harry E. Pence. 2010. “Digital Scholarship and Open Access.” Journal of Educational Technology Systems 38, no. 2: 95-101.
Magnan, Sally Sieloff. 2007. “Commentary: The Promise of Digital Scholarship in SLA Research and Language Pedagogy.” Language Learning & Technology 11, no. 3: 152-155.
Maron, Nancy L. and K. Kirby Smith. 2008. Current Models of Digital Scholarly Communication: Results of an Investigation Conducted by Ithaka for the Association of Research Libraries. Washington D.C.: Association of Research Libraries.
Pannacker, William. 2013. “Stop Calling It ‘Digital Humanities’.” Chronicle Of Higher Education 59, no. 24: A21-A22.
Pannapacker, William. 2012. “No DH, No Interview.” Chronicle of Higher Education. Vol. 58, Issue 42: A25-A26.
Pannapacker, William. 2011. “Big-Tent Digital Humanities: a View From the Edge, Part 2.” Chronicle Of Higher Education 58, no. 5: A32.
Prescott, Andrew. 2011. “Consumers, creators or commentators?: Problems of audience and mission in the digital humanities.” Arts and Humanities in Higher Education.2012 11: 61. DOI: 10.1177/1474022211428215.
Scanlon, Eileen. 2012. “Digital Futures: Changes in Scholarship, Open Educational Resources and the Inevitability of Interdisciplinarity.” Arts and Humanities In Higher Education: An International Journal Of Theory, Research And Practice 11, no. 1-2: 177-184.
Seaman, John T., and Margaret B. W. Graham. 2012. “Sustainability and the Scholarly Enterprise.” Journal of Scholarly Publishing 43, no. 3: 257-293.
Svensson, Patrik. 2012. “The digital humanities as a humanities project.” Arts and Humanities in Higher Education 2012 11: 42 originally published online 1 December 2011. DOI: 10.1177/1474022211427367.
Veletsianos, G. 2012. “Higher education scholars’ participation and practices on Twitter.” Journal of Computer Assisted Learning 28, no. 4: 336-349.
Veletsianos, George, and Royce Kimmons. 2012. “Networked Participatory Scholarship: Emergent techno-cultural pressures toward open and digital scholarship in online networks.” Computers & Education 58, no. 2: 766-774.
“Working Together or Apart: Promoting the Next Generation of Digital Scholarship. CLIR Publication No.145.” 2009. Washington, DC. Council on Library and Information Resources, and Washington, DC. National Endowment for the Humanities. https://docs.google.com/viewer?url=http%3A%2F%2Fwww.clir.org%2Fpubs%2Freports%2Fpub145%2Fpub145.pdf
 For example, some know well the impact of Ernest Boyer’s 1990 Scholarship Reconsidered: Priorities of the Professoriate, andthe ways in which his definitions of the scholarship ofdiscovery, integration, application, and teaching have been debated and integrated into evaluation guidelines across the academy. For a concise summary, see http://en.wikipedia.org/wiki/Ernest_L._Boyer.
 See, for example, the Clearinghouse and National Review Board for the Scholarship of Engagement at http://schoe.coe.uga.edu/evaluation/evaluation_criteria.html.
 I served as Director of Forensics, leading and coaching the college’s intercollegiate speech and debate program. We wrote and used departmental guidelines based on resources for forensics educators provided by the American Forensic Association; contact the author for a copy of current guidelines.
 The department was then a combined Communication and Theatre Arts program. For an example of evaluation guidelines for the distinct types of teaching, research, and service involved in multiple positions in theatre, see Association of Theatre in Higher Education materials http://www.athe.org/?page=tp_guide&terms=%22evaluation+and+guidelines%22.
 Michael Jensen, “The New Metrics of Scholarly Authority,” The Chronicle Review – The Chronicle of Higher Education, June 15, 2007.
 William G. Thomas III, “The Promise of Digital Humanities and the Future of the Liberal Arts,” http://railroads.unl.edu/blog/?p=1094, posted October 26, 2013.
 See for example: John T. Cacioppo, “Metrics of Science,” Association for Psychological Science, 2013, http://www.psychologicalscience.org/index.php/publications/observer/2008/january-08/metrics-of-science.html; a 2009 National Science Foundation workshop on scholarly evaluation metrics, http://informatics.indiana.edu/scholmet09/announcement.html; and a 2010 special issue of the journal Nature, http://www.nature.com/news/specials/metrics/index.html.
 See, for example, Alison Byerly, “Everything Old is New Again: The Digital Past and the Humanistic Future,” Modern Language Association Conference, January 2012, and William Pannapacker, “Stop Calling It ‘Digital Humanities’,” Chronicle of Higher Education 59, no. 24: A21-22.
 Mark Sample, “Digital Humanities at MLA 2014,” at http://www.samplereality.com/2013/09/19/digital-humanities-at-MLA-2014/.
 THATCamp (The Humanities And Technology Camp) is a user-generated unconference for technologists and humanities professionals, including university and college faculty, librarians and archivists, and museum staff; see http://en.wikipedia.org/wiki/THATCamp or www.thatcamp.org.
 The author recognizes that faculty evaluation guidelines may not be visible beyond a campus intranet, so both welcomes correction of this data and need for additional survey work.
 Sheila Cavanaugh, “Living in a Digital World: Rethinking Peer Review, Collaboration, and Open Access,” Journal of Digital Humanities, Vol 1, No. 4, Fall 2012), www.journalofdigitalhumanities.org.
 Eric Schnell, “Digital Scholarship and the Faculty Rewards System,” www.library.osu.edu, April 29, 2013.
 Two additional sets of campus guidelines from a decade prior are: TLT Group. Sample guidelines: Duquesne University. 1996. http://www.tltgroup.org/resources/rduqten.html
and University of Virginia. Evaluating Digital Scholarship, Promotion & Tenure Cases. 2001. http://artsandsciences.virginia.edu/dean/facultyemployment/evaluating_digital_scholarship.html
 For a paper by University of Maine faculty members that “argues for redefining evaluation criteria for faculty working in new media research and makes specific recommendations for promotion and tenure committees in U.S. universities,” see: Ippolito, J., et al. 2009, “New Criteria for New Media” Leonardo, Vol. 42, No. 1, pp.71-75. http://www.mitpressjournals.org/doi/pdf/10.1162/leon.2009.42.1.71
 “Closing the Evaluation Gap,” Journal of Digital Humanities,Vol. 1, No. 4, Fall 2012. http://journalofdigitalhumanities.org/1-4/
 Sydni Dunn, “Digital Humanists: If You Want Tenure, Do Double the Work,” Chronicle of Higher Education, 2014 January 6. https://chroniclevitae.com/news/249-digital-humanists-if-you-want-tenure-do-double-the-work/
 Ayers, Edward L. “Does Digital Scholarship Have a Future?” Educause Review. July/August 2013, pp. 24-34. http://www.educause.edu/ero/article/does-digital-scholarship-have-future
 Several of the guidelines reference MLA as an important influence.
Written by the author based on other addenda in use in campus evaluations; a model for digital scholarship has not been used or adopted yet at Puget Sound.