26 September 2025: our work, coi, standards, ai, acute, bfa, methods-series

Up and down (and up?) the AI hype cycle

In spring 2023, the hype surrounding generative AI models — particularly OpenAI's ChatGPT — arrived at ACCORD. As in many other professional environments, we asked ourselves what this meant for us, including the uncomfortable question: “Are we now obsolete?” ;-) This was followed by a period of exploration and testing. AI for text-based tasks such as summarising and paraphrasing turned out to work well. Eventually, the models gained internet access, meaning they were no longer limited to the information contained in their two-year-old training data. Gradually, the tools were able to incorporate current search results from browsing, and to receive some sort of assistance in finding sources.

What we find particularly significant about all these developments is the contrast we are experiencing. A few weeks ago, after a team meeting at which the use or potential use of AI applications was discussed, we realised how much AI, or at least the topic of AI, has already become part of our professional everyday life over the past two years. While two years ago, early adopters were trying to convince some of their colleagues to try ChatGPT, sometimes encountering hesitation or even resistance, those same early adopters now sometimes feel like big “AI sceptics” when talking with people who express strong confidence in AI’s future capabilities. Among these capabilities might be the elimination of hallucinations, AI becoming a primary source for discovering useful information. Others suggest that our target groups need more summaries generated by AI, as this is reflected by what we are increasingly experiencing with the products of various major AI companies in recent weeks and months — particularly with developments like Gemini-generated summaries now appearing by default at the top of Google search results.

The attitudes of many of our colleagues towards AI have changed significantly over the last two years, probably due to increased contact with AI applications in their personal lives. Attitudes have shifted from reservation to acceptance. And yet, amidst all the AI developments around “reasoning” models, etc., we have noticed that our own caution towards AI has increased rather than decreased. While two years ago we could not (and still cannot estimate where the generative AI language model journey would lead!) we now find ourselves in an era where “Large Language Models” (LLMs) are celebrated for their reasoning abilities. We can now, we are told, delve into the “chain of thought” of the model to understand its thought process.

In this blog post, we will outline the reasons why we are now more cautious than before and find that a thorough look at what possibilities and risks AI can bring to the field of COI is essential.

Reasoning?

Since we've already mentioned AI “reasoning” models, let's start there. These days, all major AI companies offer so-called “reasoning models”. Researchers have developed techniques such as “Chain-of-Thought” (CoT) and “Tree-of-Thought” (ToT) to improve the performance of LLMs and create the impression that these models engage in logical thinking. In its original sense, “reasoning” describes “a complex cognitive process that involves causal understanding, mental modelling, intentionality, and abstraction“(White, 2 March 2025). While generating their output, these models inform us that they are “thinking” or “getting their ideas in order”, and guide us through all their “reasoning” steps. However, this perception is misleading, as these models merely “statistically predict textual continuations based on patterns learned from extensive training data containing examples of human-written logical processes and problem decomposition”. When we come across text that seems to reflect our own thought processes, we presume that “similar cognitive mechanisms” must be at work. “But this is a fundamental attribution error — attributing human-like cognition to what is essentially sophisticated pattern matching” (White, 2 March 2025). Some scientific publications explicitly warn against this anthropomorphising, as a research paper by Kambhampati et al. (2025) with the title, “Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!“ clearly indicates.

For example, Kambhampati et al. (2025) and Stechly et al. (2025) specifically warn against the “reasoning” and “thinking” metaphors, describing them as “actively harmful” because they create false confidence in these systems' capabilities (Kambhampati et al., 2025, p. 9). Given that such clear language is used in scientific articles, it is not unreasonable to suggest that even experienced COI researchers, who work with information daily but do not necessarily understand the technical details of various AI technologies, could be impressed by the 'thought processes' of these 'reasoning' models. After all it is not currently expected that all COI researchers must familiarise themselves with the technical details of new AI technologies (let alone newly hired researchers), much less different types of COI users.

Hallucinating?

Similarly, AI hallucinations remain a significant problem. We assume that nowadays most users are familiar with these hallucinations, which occur when an AI model generates content that is (entirely) fabricated. Such content is often presented in a polished and convincing manner, making it difficult to detect inaccuracies. However, are you aware that with newer models released by major AI providers the risk of hallucinations has increased rather than decreased? While their mathematical abilities have notably improved, their grasp of facts has weakened. The reason for this is not entirely clear (NYT, 6 May 2025a). After all, AI models cannot distinguish between truth and falsehood.

Contrary to the widespread assumption that the hallucination problem will eventually be solved, some researchers argue that hallucinated output is an inherent and unavoidable aspect of generative AI models' underlying functionality (NYT, 6 May 2025a).

“Those hallucinations may not be a big problem for many people, but it is a serious issue for anyone using the technology with court documents, medical information or sensitive business data”, according to a recent New York Times article on this topic (NYT, 6 May 2025a). In the field of COI, we are likely to operate in a similarly sensitive area. In addition, it isn't always easy to distinguish between factual and fictional AI output using the techniques we normally use to assess truthfulness (NYT, 17 May 2025).

The New York Times and other media outlets are reporting that fake, AI-generated documents are flooding US and British courtrooms (NYT, 6 May 2025b; The Conversation, 12 March 2025; The Guardian, 6 June 2025), often going unnoticed by the opposing side. Not everyone is aware of the hallucination problem, nor is everyone sufficiently motivated or thorough when reviewing AI-generated output. Some scientific papers even raise the question of whether “we are about to enter a world where the truth isn’t obvious to anyone, where we are all swimming in falsehoods. Some people think we are already there” (NYT, 6 May 2025b).

These thoughts become even more crucial when we consider that leading AI companies have now used almost all the English text on the internet to train their systems. For years, companies like OpenAI have relied on a simple concept: The more internet data they feed into their AI systems, the better they work (NYT, 16 May 2025).

Web content

And while we're on the subject of English-language content on the internet: Current estimates suggest that the amount of AI-generated content on the Internet is increasing significantly (OeAW, 14 February 2025). Some forecasts predict that this figure will reach 90% by 2026 (Ohio University, 11 December 2024). Bearing in mind that buzzwords like “fake news” and “post-truth era” were commonly used even before the big hype around generative AI almost three years ago, one might feel uneasy looking at these forecasts, both as a private individual and all the more as a COI researcher. And in May 2025, Google introduced its latest planned developments, revealing a radical AI-supported overhaul of its search function that will provide answers without users needing to visit any websites (Google, 20 May 2025; Google, 5 March 2025). The underlying sources of the information found thus move even further into the background. This may seem like a practical development for private Google use: If I want to quickly find a pancake recipe, I don't really care where the Gemini summary gets its information from. I'm happy about the quick help. And, to be honest, not much is at stake with the pancake recipe search query. However, a completely different scenario arises when it comes to research in our professional field.

Whether quality journalism will be the big saviour in this complex situation is also questionable, given that AI is reportedly already widely used in newsrooms, yet the ethical discussion surrounding it has largely been absent so far. Canadian researchers working on this topic (see, Misri et al., 2025) reached the following scathing conclusion: “it became clear that many news organizations are still operating in the ethical equivalent of the Wild West.” One example of insufficient AI supervision is an article in the Los Angeles Times that was reportedly criticised for softening the image of the Ku Klux Klan (The Conversation, 13 May 2025).

What to do?

To recapitulate: we are facing a situation where generative AI models are increasingly trying to convince us of their human-like abilities, even though they are actually becoming more and more prone to errors; at a time when the internet is increasingly flooded with potentially inaccurate, AI-generated content, the industries responsible for processing meaningful information seem to be caught off guard by fundamental ethical questions, and Google would like to provide us with AI-generated summaries for all our search queries without citing sources and promises “expert-level” reports created by its language model (Google, 20 May 2025), while OpenAI promises “PhD-level intelligence” of its newst model (OpenAI, 7 August 2025).

In light of these developments, it seems that new challenges are emerging for COI research in the coming years. While the biggest difficulty in the past, especially pre-internet, was often gaining access to certain information, the opposite now seems to be happening. We are flooded with information, and determining whether it is factually correct is difficult amid such vast (AI-generated) quantities. Thus, over time, the challenge of gathering information has switched to the opposite extreme. This leaves us with some unanswered questions: What is the future of information and what is the future of COI?

In summary, we believe that it remains undisputed that AI models can provide us with enormous support in our work, and we are happy to accept this support in our field fraught with time pressure. However, this requires critical examination and consideration of many different facets, ranging from technical issues to legal and ethical aspects.

Working on it

Right now, and for precisely these reasons, we are all the more pleased to have the opportunity to tackle these challenges in the course of a project dedicated to sharing experiences and jointly examine theoretical and practical aspects of the application of AI in COI. The COI unit of the Austrian Federal Office for Immigration and Asylum has commissioned ACCORD with the project “ACUTE – Artificial Intelligence in Country of Origin Information: Understanding technical and ethical implications”. ACUTE will bring together experts and representatives of COI units from several countries and will run for four years. We believe that joint observation of developments in AI and their specific impact on the field of COI are essential. Together, we can develop a common approach and establish guidelines for addressing these challenges.

By obtaining external expertise in ethics, law and technology, developing common guidelines for the use of AI in COI research and working on AI-based software solutions tailored to our field, the ACUTE project can further help keep people working with COI up to date with specific training. This will enable the field of COI to navigate the AI jungle and ultimately look to a somewhat uncertain future with confidence, while trying our best to ensure the quality of COI.

We will publish relevant methodological updates on the use and capabilities of AI in our field in this blog’s methods series.

You can read a project description on ACUTE here.

Sources

Google: Expanding AI Overviews and introducing AI Mode, 5 March 2025
https://blog.google/products/search/ai-mode-search

Google: AI in Search: Going beyond information to intelligence, 20 May 2025
https://blog.google/products/search/google-search-ai-mode-update/#deep-search

Kambhampati, S. et al.: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces! [Preprint], 2025
https://www.arxiv.org/pdf/2504.09762

Misri, A. et al.: “There’sa Rule Book in my Head”: Journalism Ethics Meet AI in the Newsroom. In: Digital Journalism, 2025
https://www.tandfonline.com/doi/full/10.1080/21670811.2025.2495693

NYT – New York Times: A.I. Is Getting More Powerful, but Its Hallucinations Are Getting Worse, 6 May 2025a
https://www.nytimes.com/2025/05/05/technology/ai-hallucinations-chatgpt-google.html

NYT – New York Times: What happens when A.I. hallucinates? [Audio], 6 May 2025b
https://www.nytimes.com/2025/05/05/technology/ai-hallucinations-chatgpt-google.html

NYT – New York Times: Why We’re Unlikely to Get Artificial General Intelligence Anytime Soon, 16 May 2025
https://www.nytimes.com/2025/05/16/technology/what-is-agi.html

NYT – New York Times: For One Hilarious, Terrifying Day, Elon Musk’s Chatbot Lost Its Mind, 17 May 2025
https://www.nytimes.com/2025/05/17/opinion/grok-ai-musk-x-south-africa.html

OeAW – Österreichische Akademie der Wissenschaft: KI-Content: Ein Fluch im Netz?, 14 February 2025
https://www.oeaw.ac.at/news/ki-content-ein-fluch-fuer-das-netz

Ohio University: AI content: Ethics, identification and regulation, 11 December 2024
https://www.ohio.edu/news/2024/12/ai-content-ethics-identification-regulation

OpenAI: Introducing GPT-5, 7 August 2025
https://openai.com/index/introducing-gpt-5/

Stechly, K. et al.: Beyond semantics: The unreasonable effectiveness of reasonless intermediate tokens [Preprint], 2025
https://arxiv.org/pdf/2505.13775

The Conversation: AI is creating fake legal cases and making its way into real courtrooms, with disastrous results, 12 March 2025
https://theconversation.com/ai-is-creating-fake-legal-cases-and-making-its-way-into-real-courtrooms-with-disastrous-results-225080

The Conversation: Have journalists skipped the ethics conversation when it comes to using AI?, 13 May 2025
https://theconversation.com/have-journalists-skipped-the-ethics-conversation-when-it-comes-to-using-ai-255485

The Guadian: High court tells UK lawyers to stop misuse of AI after fake case-law citations, 6 June 2025
https://www.theguardian.com/technology/2025/jun/06/high-court-tells-uk-lawyers-to-urgently-stop-misuse-of-ai-in-legal-work

White, Matt: I Think Therefore I am: No, LLMs Cannot Reason, 2 March 2025
https://matthewdwhite.medium.com/i-think-therefore-i-am-no-llms-cannot-reason-a89e9b00754