Skip to main content
To the Editor,
We have carefully read the original editorial in the March 2023 issue of the Annals of Family Medicine, authored by ChatGPT itself [1], likely version 3.5 given the publication date.
In our view, there are more reasons than those humbly acknowledged by ChatGPT regarding its own limitations. Firstly, it fails to adhere to the journal authorship policies (updated in January of the same year), as indicated in the disclaimer of the editorial, which aligns with the recommendations of the International Committee of Medical Journal Editors (ICMJE).
As highlighted in other letters to the editor, such as the one written by Md. Christina Marie Roberts, the way in which the prompt question is phrased influences the response content. Therefore, the negative tone of the question in the editorial prompt leads ChatGPT to only present arguments against its capability to write an academic article.
Of course, utilizing this tool for authoring a paper is extremely risky, as it suffers from hallucinations [2-3]. While this is a characteristic of its design, allowing for responses that are somewhat creative, it is dangerous because it cannot discern the limits of its knowledge and what it just "imagines".
Moreover, a significant portion of the scientific references it cites are entirely fictional [4].
However, perhaps an even bigger issue is the lack of coherence in its responses. If asked the same question by repeating the prompt 100 times, not all responses will align, and at least in the current version, this Large Language Model (LLM) is not able to distinguish reliable responses from unreliable ones. We encourage readers to conduct this same experiment, in which we obtained the following results [5]:
Prompt
Write a short editorial about why chatGPT should not be used to write academic scientific manuscripts for publication.
Response
"The integration of artificial intelligence (AI) tools like ChatGPT into various facets of academia has sparked considerable debate, particularly regarding their use in composing academic scientific manuscripts for publication. While AI can offer significant advantages in data analysis, literature review, and preliminary drafting, there are compelling reasons why ChatGPT should not be used to write academic scientific manuscripts for publication. These reasons revolve around issues of originality, reliability, ethical considerations, and the development of critical thinking skills.
Originality and Intellectual Contribution:
Academic writing, especially in the sciences, is not just about the dissemination of information but also about the contribution of new knowledge and original thought to a field of study. The essence of scientific inquiry is the generation of novel ideas, hypotheses, and interpretations of data. ChatGPT, by its design, generates content based on patterns and information it has been trained on up to its last update, without the ability to produce or conceptualize new knowledge or original research findings. Relying on it for manuscript writing could lead to a homogenization of thought and a lack of innovation in academic literature, as the tool may recycle existing ideas rather than propose new ones.
Reliability and Accuracy:
The reliability of the content produced by ChatGPT is another critical issue. While it can provide information that appears accurate and well-structured, there's no guarantee that this information is error-free or up-to-date. Scientific writing demands a high level of accuracy, as the implications of disseminating incorrect information can be far-reaching in fields like medicine, engineering, and environmental science. The potential for ChatGPT to inadvertently include outdated or incorrect data in manuscripts could mislead research and hinder scientific progress.
Ethical Considerations:
The use of AI in writing scientific manuscripts also raises ethical questions. Authorship entails accountability for the content presented, reflecting an understanding and assurance of the work's validity and integrity. When ChatGPT is used to compose manuscripts, it blurs the lines of authorship and accountability. Additionally, the potential for plagiarism or the unauthorized use of copyrighted material without proper attribution could compromise the ethical standards of academic publishing.
Development of Critical Thinking and Writing Skills:
The process of writing is intrinsically linked to the development of critical thinking and analytical skills. For researchers, especially students and early-career scientists, the act of writing is not only about communicating findings but also about refining their understanding of their work and its significance. By outsourcing the writing process to AI, there's a risk of diminishing these essential skills, potentially stunting the intellectual growth of researchers and weakening the overall quality of scientific inquiry.
Conclusion:
While ChatGPT and similar AI tools hold promise for assisting with certain aspects of academic work, their limitations in generating original research contributions, ensuring accuracy, upholding ethical standards, and fostering intellectual development make them unsuitable for writing scientific manuscripts for publication. The scientific community must critically evaluate the role of AI in research and maintain a balance that leverages technology's benefits without compromising the integrity and advancement of science."
Furthermore, when compared to other LLMs, such as Google's Gemeni (which was made available to the public just this week), the arguments in the response differ. However, in this case, it does attempt to provide potential real sources of information on the internet that complement its response:
Prompt:
Write a short editorial about why Bard should not be used to write academic scientific manuscripts for publication.
Response (Gemini):
"While Bard can be a helpful tool, it's not ready to ghostwrite your science paper (yet).
The allure of large language models like Bard to expedite the academic writing process is understandable. With our ability to synthesize information and generate text, it might seem tempting to delegate manuscript writing to AI. However, before you hit the "write" button and submit your Bard-authored paper, consider these critical limitations:
1. Lack of Originality and Understanding:
Bard excels at mimicry, not genuine scientific understanding. We can produce text that sounds scientific, but we don't grasp the nuances of research design, methodology, or data interpretation. This lack of depth can lead to factual errors, superficial analysis, and ultimately, meaningless content.
2. Ethical Concerns:
Plagiarism is a serious academic offense. While Bard doesn't directly plagiarize, its generated text might unknowingly mirror existing published work. Furthermore, relying solely on AI for authorship disregards the crucial role of human researchers in conceptualizing, conducting, and interpreting research.
3. Inability to Address Reviewer Feedback:
The scientific publishing process thrives on peer review, an iterative cycle of feedback and improvement. Bard is currently unable to understand and respond to such nuanced critiques, leaving you stuck with a potentially flawed manuscript.
4. Risks to Scientific Integrity:
Widespread use of AI-written manuscripts could erode the foundation of scientific trust. The inability to verify authorship, potential biases in AI algorithms, and the difficulty in detecting AI-generated text all pose significant risks to the integrity of the scientific record.
While Bard holds immense potential to assist researchers in tasks like literature review and data analysis, it should not be viewed as a replacement for human authorship in scientific manuscripts. Remember, scientific publishing is not just about producing text; it's about contributing original, well-founded knowledge to the advancement of a field. That responsibility ultimately lies with you, the researcher, not with an AI tool.
Instead of seeking a shortcut, use Bard's capabilities to enhance your own writing and critical thinking skills. Let AI be your collaborator, not your ghostwriter. The future of scientific communication lies in a thoughtful partnership between human expertise and the potential of language models like Bard."
Technology is advancing rapidly, but as Gemini indicates, it is still far from replacing doctors who write academic articles. And, of course, this letter to the editor has not been written by ChatGPT or Gemini, except for the excerpt we have quoted within quotation marks.
[1] American Academy of Family Physicians. (2023). Why ChatGPT Should Not Be Used to Write Academic Scientific Manuscripts for Publication. The Annals of Family Medicine.
[2] Alkaissi, H., & McFarlane, S. I. (2023). Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus, 15(2).
[3] Giuffrè, M., You, K., & Shung, D. L. (2023). Evaluating ChatGPT in Medical Contexts: The Imperative to Guard Against Hallucinations and Partial Accuracies. Clinical Gastroenterology and Hepatology.
[4] Metze, K., Morandin-Reis, R. C., Lorand-Metze, I., & Florindo, J. B. (2024). Bibliographic research with ChatGPT may be misleading: the problem of hallucination. Journal of Pediatric Surgery, 59(1), 158.
[5] https://chat.openai.com/share/5c523a3b-9ca8-4ca3-b4e3-eca43edbc80b
[6] https://g.co/gemini/share/292fb27e131c