By Alessandro Meregaglia
Introduction
With the meteoric rise in popularity of artificial intelligence (AI) and large language models (LLM), many students have turned to ChatGPT, Gemini, and other platforms for their research. Even if students aren’t using AI to write their papers, they might be using it to gather primary sources and summarize content– erroneously thinking it checks everything on the internet. As a primary source instructor, I have encountered specific limitations of AI to locate and analyze primary sources. For example, AI cannot easily ingest and process PDFs, web archive (WARC) files, paywalled content, and even open-access structured databases.
This post will discuss the current limitations of using AI to search digitized and born digital primary sources––like newspaper databases, the Wayback Machine, and archival digital collections. I will show examples of how AI fails to identify information in these primary sources, even though they are available online and navigable by humans. My goal is to equip other primary source instructors with a general overview so they, in turn, can inform their students, who might be tempted to think AI is an exhaustive tool for primary source research.
What Primary Sources Can AI NOT search?
Each year, I teach around 8-10 one-shot instruction sessions on primary source research. These sessions primarily focus on using Boise State’s Special Collections and Archives, but I also cover digital collections outside of Boise State. Beginning in Spring 2025, I started talking with those students about the limitations of using AI specifically for primary source In Fall 2024, I acquired a ChatGPT Edu license through my institution, Boise State University. Since then, I have become an avid user and find ChatGPT helpful for a range of applications in my job as an archivist and librarian: writing code to extract metadata from image files, comparing spreadsheets for small differences, and troubleshooting audiovisual digitization software. I also began to extensively investigate how ChatGPT searches different types of internet sources.
PDFs
Like many universities, Boise State’s Special Collections has digitized all of its yearbooks and has made them available online and searchable for free in our institutional repository (IR). However, these yearbooks are only available as PDFs (albeit with OCRed text). If a researcher were interested, for example, in the list of Boise Junior College sophomores from 1949, that patron would not be able to use AI to get that information immediately. Instead, they would need to know that Boise State’s yearbooks were digitized, navigate to our IR, and then open up the PDF. Here is ChatGPT’s response to that inquiry directing users to our digitized yearbooks, followed by a page from the 1949 Les Bois showing the start of the sophomore listing:
ChatGPT cannot interpret PDFs because ChatGPT is not a search engine; it cannot independently search the internet for PDFs. However, if PDFs are uploaded directly to ChatGPT, it can then run an analysis of those files (this functionality is discussed further below).
Open-Access Structured Databases
Newspapers, of course, are commonly used in primary source research. Yet many digitized historic newspapers are not searchable via AI because of the way those databases are formatted. Although one of the preeminent databases for American newspapers, Chronicling America, is freely available, ChatGPT and other AI platforms cannot search the content of the digitized newspapers for the same reasons ChatGPT cannot directly access PDFs: these databases are structured in such a way that are geared towards human use and not AI use. These databases understandably prioritize presentation and preservation over machine learning ingestion. Moreover, the search capabilities for these types of databases rely on internal indexes that are not public or exposed in a way that AI platforms can harvest.
The same is true for many content management systems (CMS) and databases used by special collections and libraries. The content of digital objects in CMS databases—both locally created or offered by third parties like ContentDM, Islandora, Preservica—suffer the same issue as Chronicling America for the reasons explained above: the content is essentially invisible to AI. For example, Boise State’s digital collections website (using Preservica as the platform) includes the monthly magazine for Morrison-Knudsen, a large construction company founded in Boise. Those magazines are full-text searchable thanks to OCR, but their contents are not available to AI models. One of those magazine issues gives the name of the project manager for the Aliamanu military housing project in Hawaii (Russell Stanley Sr.), but ChatGPT says that “the specific individual who served as the project manager . . of the Aliamanu Military Reservation (AMR) housing project in the 1970s is not publicly documented in available sources.”
Open-access structured databases––relatively intuitive and easy for humans to navigate––pose challenges for AI to interpret.
WARC Files
The Wayback Machine, a digital archive of the internet that captures websites in the web archive (WARC) format, is an incredible primary source resource. I always talk about it when giving instruction sessions––even sessions only on general library research. “What did company X say about itself during its first year, and how does it describe itself now?” “Who sat on the board of governing body Y twenty years ago?” These are questions the Wayback Machine can answer. But AI can’t answer them.
Humans who are aware of the target website can enter that URL into the Wayback Machine, navigate to the relevant date range, and answer their question. For example, here is the ChatGPT response when I queried it to give me the names of the National Museum and Library Services Board board members from 2005, followed by a screenshot of the IMLS website from 2005 showing the availability of that very information in the Wayback Machine:
So why can’t AI accomplish this? As ChatGPT puts it: “ChatGPT is optimized for language processing, not for snapshot navigation or time-based URL scraping, which requires specialized tooling.” In simplest terms, web archive (WARC) files are not designed for easy search or analysis without preprocessing (such as indexing them or transforming the files into plain text or another format), making direct access to WARC files by AI systems or search engines inefficient.
Paywalled Content
The vast majority of primary sources are open access and not locked behind paywalls. However, there are a number of proprietary databases offered by information-content companies that contain curated sets of primary sources. (Examples range from ProQuest’s History Vault to Gale’s Slavery and Anti-Slavery: A Transnational Archive to Adam Matthew Digital’s Medical Services and Warfare.) Not surprisingly, ChatGPT cannot scrape those databases or anything behind a paywall.
How to Use AI to Search Primary Sources
As is evident from those restrictions, ChatGPT and AI in general does best with explaining concepts, summarizing sources, and guiding researchers in their methodology. It is not an in-depth, all-encompassing research tool. However, despite the limitations outlined above, there are several ways to leverage AI to search for and within primary sources that are not part of AI’s training model.
Briefly, here are some workarounds:
- Primary Source Wayfinding: AI does a decent job of directing users to archival repositories and databases. Ask ChatGPT “where are Ernest Hemingway’s papers?” and it will be able to answer accurately. Ask ChatGPT “Who was in the Sophomore class at Boise Junior college in 1949?” and, although it will not be able to answer, it will recommend contacting Boise State’s Special Collections and even provide a link to our digitized yearbooks. In other words, it often knows where sets of primary sources are located; it just cannot search in them.
- WARC-GPT: Developed by Harvard and released in early 2024, WARC-GPT is a tool that allows users to point to a specific group of WARC files and then interact with that collection in a natural language style like other LLMs. This means archivists and librarians can use WARC-GPT to create a mini ChatGPT that is trained on archived websites you have access to.
- Application Programming Interfaces (APIs): APIs act as an intermediary between software applications (often websites) and another software application or coding script to allow the two exchange information or work together. In the context of primary sources, here are two use cases of APIs:
- Digital Public Library of America: Civil Rights Movement researchers interested in primary sources from 1954-1968 can write a script to query DPLA’s API for a specific set of primary sources on their specific topic and date range, and include relevant metadata.
- Chronicling America: researchers can write a script to query Chronicling America’s API to produce a list of all the issues of a given newspaper (this is that list for the New York Sun). Then a script can be written to extract the OCR text for a specific page of a specific issue (this is the OCR text of the April 18, 1911 front page; this is the digital image of that page). That second link can be put into ChatGPT for analysis––summarizing, cleaning it up, or providing suggestions for further research.
- Direct Links to PDFs: A tedious but effective option to analyze primary sources is to locate the specific PDF of a primary source, give that link (or multiple links) or upload the file to ChatGPT, and instruct it to summarize the content. For example, when I upload the 1949 Les Bois yearbook discussed earlier, ChatGPT can successfully give me a list of the sophomore students:
Of course, this essay and these examples only address AI’s relationship to teaching with primary sources. There are myriad ways that AI can be used to improve accessibility in special collections libraries and archives (such as automating transcription of oral histories, cleaning up OCR text, or writing biographical notes in finding aids).
The common thread among these limitations is that AI can search and analyze primary sources that are:
-
- Freely available on an internet webpage, and
-
- In machine-readable text format, and
-
- Written in a recognized natural language (archaic scripts or obscure dialects are not supported), or
- Loaded directly into an AI platform (either via direct link or file uploading)
Conclusion
This article may be obsolete in two years – or less. But, for the time being, it is evident that students (or anyone working primary sources) still need to know where to find digitized primary sources and how to search those digital repositories. They need to know what types of primary sources are available in archives and online so that, if they use AI, they are aware of the gaps in the responses. I encourage all primary source instructors to play around with ChatGPT and other platforms to understand for themselves how AI searches and summarizes primary sources. Users can query ChatGPT for free without even having to sign up for an account. Knowing what AI can and cannot do will make us better teachers.
Alessandro Meregaglia is an Archivist/Librarian and Associate Professor at Boise State University in Idaho, where he has worked since 2016. He earned a Master of Arts in History and a Master of Library Science at Indiana University.