Have you ever wished you could snap your fingers and have a vast amount research done for you? Insert your research question, and get a detailed summary in minutes. That’s the idea behind GPT Researcher.
HOW DID THIS PROJECT BEGIN?
Rotem Weiss is one of the earliest developers of Assaf Elovic's open-source project, GPT Researcher. He has a little bit of a different path than the traditional Silicon Valley tech developer. He grew up in Israel, where at the age of 18 he was required to do military service. After that, he journeyed to the United States and worked for two years. Shortly after, he found himself at Columbia University, where he remains a student. It was here he really began to develop a growing interest in computers and Artificial Intelligence (AI). He thinks of himself as not only a programmer but also a entrepreneur in the tech space.
Weiss was one of the earliest developers and serves as a main developer with creator Assaf Elovic. GPT Researcher began by using AutoGPT, but they found that—for their specific needs—the technology needed to be further developed due to never-ending loops which required a lot of human intervention. Weiss said this process “allowed them to see the potential of AI to conduct online research.”
Weiss points out that it is very difficult for a human to explore and examine all of the available data on a given subject—there is a lot of content—so this is just not feasible. “This is where computers can step in … They can collect data super fast, they can bring the data you need, and they can build on top of it much faster.” GPT Researcher can allow the researcher to compile a summary of a vast array of data which can save the researcher time and also allow them to pull from resources they might not have come across using their own research methods.
Weiss does not see GPT Researcher as a replacement for human research. Rather, it is to be used as a data-gathering tool to aid in further research, to make research more efficient and intentional.
A SIMPLE PROCESS
The process for GPT Researcher is quite simple. The user begins by creating a research question or prompt. GPT Researcher then uses that question to create four sub-questions, using GPT-4. For each sub-question, it will conduct research on Duck-Duck-Go using five URLs, meaning the user is able to access 20 websites simultaneously. The results will then be compiled together and summarized by the large language model (LLM) with in-text citations. The entire research process takes less than three minutes.
WHAT ABOUT BIAS AND HALLUCINATIONS?
GPT Researcher’s approach to bias is in the law of numbers. One webpage, for example, might be strongly bias. However, exploring 20 webpages will hopefully garner a more comprehensive analysis which encompasses a variety of viewpoints. Of course, this will not eliminate bias entirely, but 20 webpages allows for a more comprehensive and fair analysis of the field than simply one webpage. As Weiss says, “There will always be bias, but I do believe we are able to reduce it significantly.”
GPT Researcher is also programmed to only pull from the data that is collected in its summary. In this way, the program does not allow the LLMs to have freedom to create content out of thin air, as is sometimes a worry when people use models like ChatGPT for research purposes. As Weiss said, “We try to use LLMs for what they are best at: summarization and creating the report, not providing data.”
WHAT’S NEXT FOR GPT RESEARCHER?
GPT Researcher only launched a month ago and already has 10 active contributors (their code has been integrated) and a Discord community of over 800 members. There are no indications of slowing down. Some features of the tool currently being enhanced are as follows:
Word limits: The current limit is 2,000 words for research due to the bottleneck of ChatGPT’s limitations on summaries, but the developers are currently working on a master agent that will be able to produce up to 10-page research papers using the context of previous and future paragraphs to expand the research content.
Database improvements: The GPT Researcher team is working on a whitelist which will give rankings to certain domains as well as a scholarly database which will allow the system to access less publicly available materials. Currently, they use Google’s popularity as an indicator for reliable websites, but as they improve the scope of available domains, they hope to provide more available customization for the user to choose which type of sources they are pulling from as well as how many.
LLMs: OpenAI’s ChatGPT is known for its apt ability to summarize data, so this is the main LLM used by GPT Researcher. However, Weiss said, “We do get a lot of requests for open-source LLMs from users, so we are obviously going to listen to them … and are working on integrating multiple LLMs for a user to choose from.”
Web scraping: GPT Researcher will soon be able to access much more than text. Weiss describes, “Imagine a web scraper that is much smarter and is able to understand video, audio, images, and graphs along with text … It will be like a person collecting the data. That is what we are aiming for.”
RESEARCH IS GOING FORWARD …
This project has massive potential for growth and a clear vision for the future. We can imagine a slew of potential applications for this project:
Academic Research Assistance: Students, researchers, and academics can use GPT Researcher to quickly gather information, explore different perspectives, and summarize relevant content for their research projects and papers.
Content Creation: Content creators, such as bloggers and journalists, can leverage GPT Researcher to efficiently collect information and sources for their articles, ensuring well-rounded coverage of topics.
Market Research and Decision Support: Businesses can utilize GPT Researcher to gather insights from various sources, helping them analyze market trends, consumer opinions, and competitor activities to aid in decision-making and marketing efforts.
Educational Material Generation: Teachers and educators can use GPT Researcher to curate materials for lesson plans, presentations, and educational resources, enhancing the learning experience for students.
Policy and Governance Analysis: Policymakers and government officials can employ GPT Researcher to gather a wide range of perspectives on policy-related issues, leading to well-informed decision-making.
Scientific Exploration: Scientists and researchers can benefit from GPT Researcher's ability to quickly scan a large volume of scientific literature and extract key insights, aiding in hypothesis formulation and experimental design.
Healthcare and Medical Research: Medical professionals and researchers can use GPT Researcher to aggregate medical studies, clinical trial results, and treatment approaches for evidence-based decision-making.
Environmental Analysis: Environmentalists and researchers can employ GPT Researcher to gather data and insights related to environmental issues, helping to inform sustainable practices and policies.
Cultural and Social Studies: Sociologists, anthropologists, and cultural researchers can use GPT Researcher to explore diverse cultural perspectives and social phenomena, enhancing their qualitative research.
Historical Research: Historians and researchers can use GPT Researcher to access and summarize historical documents, enhancing their understanding of past events and contexts.
Public Relations and Communication: PR professionals and communicators can utilize GPT Researcher to gather insights on public sentiment, news coverage, and relevant opinions for strategic communication planning.
Personal Knowledge Expansion: Individuals interested in self-learning and personal development can use GPT Researcher to explore a wide range of topics and expand their intellectual horizons.
… and much more! The sky is the limit with GPT Researcher, and the passionate and knowledgeable community will continually drive this project forward. In a world where AI is revolutionizing almost everything we do, it is clear that GPT Researcher will play a significant role in transforming the way humans conduct research.
