Exploring AI assisted writing using structured process techniques
An initial experiment with ChatGPT
To envision how generative AI might support students in the writing process, I used the latest version of ChatGPT with the GPT-4 model enabled to create an academic argument essay. My goal was to explore an enacted writing process that integrates evidence-based composing techniques that have been shown to improve the quality of written products. Specifically, I created a series of process-based prompts to build on a pre-defined set of sources with selected quotations. In this particular experiment, I was interested in seeing how far I could get with just pre-selected sources and only minimal human drafting and revising. Hence, the primary focus for the human role is structured process techniques and AI prompting.
Prewriting
Generating Research Questions
I started by exploring research questions based on my own established interest in research on “psychological capital.” Previous experiments have demonstrated that generating ideas is one clear strength of AI tools.
Generating ‘Smart’ Research Notes
Next, I prompted ChatGPT to help me create research notes based on a set of sources and quotations organized by a broad research question and topical headings.
I created a prompt that asked ChatGPT to suggest claims that could be made with the quotation as evidence. This aligns with research on Taking Smart Notes in ways that facilitate future writing and the creation of connections across sources.
Then, I prompted ChatGPT to work with the resulting list of claims for each source and re-process and analyze them using Bloom’s taxonomy. This approach builds on a previous study that used Bloom’s taxonomy verbs to assess the use of simple “knowledge telling” versus higher-order “knowledge transformation” sentences in academic writing (Raković et al., 2021).
After that, I sought to apply insights from research on the SOAR method, “select, organize, associate, regulate” which has been demonstrated to support strong source-based writing (Luo & Kiewra, 2019). In particular, I tried adapting the strategy of matrix organization of selected ideas (Kauffman & Kiewra, 2010).
Generating an Argument Diagram
I wanted to create an outline and argument diagram for the research essay. I created a prompt to integrate the “Matrix 1” that was generated by ChatGPT in the subsequent step with this proto-thesis: “argue for the ‘best self’ visioning exercises and self-directed plan for undergraduate students in courses.” I also requested that argument followed the Toulmin method as an outline guide.
Drafting
Generating an Essay Draft
I created a prompt that to generate a draft based on the outline and research notes. I specified that the draft should integrate parenthetical citations to signal the use of specific research notes.
Revising
Generating a Content Revision
I noticed that the draft didn’t directly address the matrix table and I decided that I wanted to directly integrate the matrix as a Table into the essay, so I prompted to add a new explanatory paragraph.
Generating Formative Feedback
Next, I created a prompt that generated formative feedback and specific suggestions for improvement based on a synthesis of previous research on undergraduate writing rubrics (Aull, 2015; Clark, 2018; Dryer, 2013; Elander et al., 2006). I repeated this step for three dimensions: argument; critical thinking and evaluation; and organization and coherence.
Generating Revisions based on Feedback
Finally, I created prompts that asked ChatGPT to implement the specific recommendations that it had generated for revision. I did this in three separate passes to conserve word-count (token) bandwidth and so that I could review each set of revisions separately. The recommendations led to one important improvement in the comprehensiveness of the thesis. The other suggestions led to additional modest improvements in the essay, though I did choose to overrule a suggestion to cut a paragraph to streamline the argument.
Results
You can read the results for yourself here: Cultivating Psychological Capital in Undergraduates. It has the stilted style of ChatGPT doing formal writing, but this approach led to an essay that accurately represents the ideas and relationships from the source material and offers a logical and coherent argument. For a comparison, you can read this essay using a simple “one-shot” prompt based on the same proto-thesis: The Transformative Power of Psychological Capital. You’ll see that this second essay lacks the depth and detail of the first essay. These results suggest that structured process techniques can help support source-based writing applications of AI tools. I look forward to continuing to explore and refine this idea in future experiments.
References
Aull, L. (2015). Connecting writing and language in assessment: Examining style, tone, and argument in the U.S. Common Core standards and in exemplary student writing. Assessing Writing.
Clark, I. (2018). Argument Essays Written in the 1st and 3rd Years of College: Assessing Differences in Performance. Journal of Writing Assessment, 11(1).
Dryer, D. B. (2013). Scaling Writing Ability: A Corpus-Driven Inquiry. Written Communication, 30(1), 3–35. https://doi.org/10.1177/0741088312466992
Elander, J., Harrington, K., Norton, L., Robinson, H., & Reddy, P. (2006). Complex skills and academic writing: A review of evidence about the types of learning required to meet core assessment criteria. Assessment & Evaluation in Higher Education, 31(1), 71–90. https://doi.org/10.1080/02602930500262379
Kauffman, D. F., & Kiewra, K. A. (2010). What makes a matrix so effective? An empirical test of the relative benefits of signaling, extraction, and localization. Instructional Science, 38(6), 679–705. https://doi.org/10.1007/s11251-009-9095-8
Luo, L., & Kiewra, K. A. (2019). Soaring to successful synthesis writing. Journal of Writing Research, 11(1), Article 1. https://doi.org/10.17239/jowr-2019.11.01.06
Raković, Mladen, Philip H. Winne, Zahia Marzouk, and Daniel Chang. “Automatic Identification of Knowledge‐transforming Content in Argument Essays Developed from Multiple Sources.” Journal of Computer Assisted Learning 37, no. 4 (August 2021): 903–24. https://doi.org/10.1111/jcal.12531.
Great write-up and very exciting work! Thanks for including the screenshots of your prompts!