Mastering ChatGPT: Effective Summarization with LLMs
Are you part of the population that leaves reviews in Google maps everytime you visit to a new restaurant?
Or perhaps you are the type to share your opinion on Amazon purchases, especially when you get triggered by a low-quality product?
Don't worry, I won't blame you – we all have our moments!
In today's data world, we all contribute to the data deluge in multiple ways. One data type that I find particularly interesting due to its diversity and difficulty of interpretation is textual data, such as the countless reviews that are posted over the Internet every day. Have you ever stopped to consider the importance of standardizing and condensing textual data?Welcome to the world of summarization agents!
Summarization agents have seamlessly integrated into our daily lives condensing information and providing quick access to relevant content across a multitude of applications and platforms.
In this article, we will explore the utilization of ChatGPT as a powerful summarization agent for our custom applications. Thanks to the ability of Large Language Models (LLM) to process and understand texts, they can assist in reading texts and generating accurate summaries or standarizing information. However, it is important to know how to extract their potential in doing such a task, as well as to acknowledge their limitations.
The biggest limitation for summarization? LLMs often fall short when it comes to adhering to specific character or word limitations in their summaries.
Let's explore the best practices for generating summaries with ChatGPT for our custom application, as well as the reasons behind its limitations and how to overcome them!
Effective Summarization with ChatGPT
Summarization agents are used all over the Internet. For instance, websites use summarization agents to offer concise summaries of articles, enabling users to gain a quick overview of the news without diving into the entire content. Social media platforms and seach engines do this too.
From news aggregators and social media platforms to e-commerce websites, summarization agents have become an integral part of our digital landscape. And with the raise of LLMs, some of these agents are now using AI for more effective summarization results.
ChatGPT can be a good ally when building an application using summarization agents for speeding up reading tasks and classifying texts. For example, imagine we have an e-commerce and we are interested in processing all our costumer reviews. ChatGPT could help us in summarizing any given review in a few sentences, standarizing it to a generic format, determine the sentiment of the review and classify it accordingly.
While it is true that we could simply feed the review to ChatGPT, there are a list of best practices — and things to avoid – to leverage the power of ChatGPT in this concrete task.
Let's explore the options by bring this example to life!
Example: E-commerce Reviews
Consider the example above in which we are interested in processing all the reviews for a given product on our e-commerce website. We would be interested in processing reviews such as the following one about our star product: a first computer for children!
In this case, we would like that ChatGPT to:
- Classify the review into positive or negative.
- Provide a summary of the review of 20 words.
- Output the response with a concrete structure to standardize all the reviews into one single format.
Implementation Notes
Here is the basic code structure we colud use to prompt ChatGPT from our custom application. I also provide a link to a Jupyter Notebook with all the examples used in this article.
The function get_completion() calls the ChatGPT API with a given prompt. If the prompt contains additional user text, such as the review itself in our case, it is separated from the rest of the code by triple quotes.
Let's use the get_completion() function to prompt ChatGPT !
Here is a prompt fulfilling the requirements described above:
⚠️ The prompting guidelines used in this example such as using delimiters to separate the input text from the rest of the prompt and asking for a structured output are fully explained at What I Learned from OpenAI's Course on Prompt Engineering – Prompting Guidelines.
Here is ChatGPT's answer:
As we can observe from the output, the review is accurate and well structured, although it misses some information we could be interested on as the owners of the e-commerce, such as information about the delivery of the product.
Summarize with a Focus on
We can iteratively improve our prompt asking to ChatGPT some focus to include in the summary. In this case, we are interested in any details given about the shipping and delivery:
This time, ChatGPT's answer is the following one:
Now the review is much more complete. Giving details on the important focus of the original review is crucial to avoid ChatGPT skipping some information that might be valuable for our use case.
Have you noticed that although this second trial includes information on the delivery, it skipped the only negative aspect of the original review?
Let's fix that!
"Extract" instead of "Summarize"
By investigating summarization tasks, I found out that that summarization can be a tricky task for LLMs if the user prompt is not accurate enough.
When asking ChatGPT to provide a summary of a given text, it can skip information that might be relevant for us — as we have recently experienced – , or it will give the same importance to all the topics on the text, only providing an overview of the main points.
Experts in LLMs use the term extract and additional information on their focuses instead of summarize when doing such tasks assisted by these type of models.
While summarization aims to provide a concise overview of the text's main points including topics non-related to the topic of focus, information extraction focuses on retrieving specific details and can give us what we are exactly looking for. Let's try then with extraction!
In this case, by using extraction, we only get information about our topic of focus: Shipping: Arrived a day earlier than expected.
Automatization
This system works for one single review. Neverthtless, when designing a prompt for a concrete application, it is important to test it in a batch of examples so that we can catch any outliers or misbehavior in the model.
In case of processing multiple reviews, here is a sample Python code structure that can help.
Here is the summaries of our batch of reviews:
⚠️ Note that although the word restriction of our summaries was clear enough in our prompts, we can easily see that this word limitation is not accomplished in any of the iterations.
This mismatch in the word counting happens because LLMs do not have a precise understanding of word or character count. The reason behind relies on one of the main important components of their architecture: the tokenizer.
Tokenizer
LLMs like ChatGPT are designed to generate text based on statistical patterns learned from vast amounts of language data. While they are highly effective at generating fluent and coherent text, they lack precise control over the word count.
In the examples above, when we have given instructions about a very precise word count, ChatGPT has struggle to meet that requirements. Instead, it has generated text that is actually shorter than the specified word count.
In other cases, it may generate longer texts or simply text that is overly verbose or lacking in detail. Additionally, ChatGPT may prioritize other factors such as coherence and relevance, over strict adherence to the word count. This can result in text that is high-quality in terms of its content and coherence, but which does not precisely match the word count requirement.
The tokenizer is the key element in the architecture of ChatGPT that clearly influences the number of words of the generated output.
Tokenizer Architecture
The tokenizer is the first step in the process of text generation. It is responsible for breaking down the piece of text that we input to ChatGPT into individual elements — tokens – , which are then processed by the language model to generate new text.
When the tokenizer breaks down a piece of text into tokens, it does so based on a set of rules that are designed to identify the meaningful units of the target language. However, these rules are not always perfect, and there can be cases where the tokenizer splits or merges tokens in a way that affects the overall word count of the text.
For example, consider the following sentence: "I want to eat a peanut butter sandwich". If the tokenizer is configured to split tokens based on spaces and punctuation, it may break this sentence down into the following tokens with a total word count of 8, equal to the token count.
However, if the tokenizer is configured to treat "peanut butter" as a compound word, it may break the sentence down into the following tokens, with a total word count of 8, but a token count of 7.
Thus, the way the tokenizer is configured can affect the overall word count of the text, and this can impact the ability of the LLM to follow instructions about precise word counts. While some tokenizers offer options to customize how text is tokenized, this is not always sufficient to ensure precise adherence to word count requirements. For ChatGPT in this case, we cannot control this part of its architecture.
This makes ChatGPT not good accomplishing character or word limitations, but one can try with sentences instead since the tokenizer do not affect on the number of sentences, but their length.
Being aware of this restriction can help you to build the best suitable prompt for your application in mind. Having this knowledge about how word count works on ChatGPT, let's do a final iteration with our prompt for the e-commerce application!
Wrapping up: E-commerce Reviews
Let's combine our learnings from this article into a final prompt! In this case, we will be asking for the results in HTML format for a nicer output:
And here is the final output from ChatGPT:
Summary
In this article, we have discussed the best practices for using ChatGPT as a summarization agent for our custom application.
We have seen that when building an application, it is extremely difficult to come up with the perfect prompt that matches your application requirements in the first trial. I think a nice take-home message is to think about prompting as an iterative process where you refine and model your prompt until you get exactly the desired output.
By iteratively refining your prompt and applying it to a batch of examples before deploying it into production, you can make sure the output is consistent across multiple examples and cover outlier responses. In our example, it could happen that someone provides a random text instead of a review. We can instruct ChatGPT to also have an standardize output to exclude this outlier responses.
In addition, when using ChatGPT for a specific task, it is also a good practice to learn about the pros and cons of using LLMs for our target task. That is how we came across with the fact that extraction tasks are more effective than summarization when we want a common human-like summary of an input text. We have also learn that providing the focus of the summary can be game-changer regarding the generated content.
Finally, while LLMs can be highly effective at generating text, they are not ideal for following precise instructions about word count or other specific formatting requirements. To achieve these goals, it may be necessary to stick to sentence counting or use other tools or methods, such as manual editing or more specialized software.
And that is all! Many thanks for reading!
I hope this article helps when building custom applications with ChatGPT!
You can also subscribe to my Newsletter to stay tuned for new content. Especially, if you are interested in articles about ChatGPT:
Unlocking a New Dimension of ChatGPT: Text-to-Speech Integration
What I Learned from OpenAI's Course on Prompt Engineering – Prompting Guidelines
What ChatGPT Knows about You: OpenAI's Journey Towards Data Privacy
Improve ChatGPT Performance with Prompt Engineering
Feel free to forward any questions you may have to [email protected]
