Text written in Basque and translated automatically by Elia without any subsequent editing. SEE ORIGINAL

Provide chatbots with updated and personalized knowledge

It is undeniable that chatbots and the large LLMs or language models on which they are based have a great knowledge of a wide variety of topics and are able to adequately answer a large number of questions. And it is also true that they often make mistakes in their answers, especially in very specialized or current issues. But there is a way to get better answers using the RAG technique.

Igor Leturia Azkarate

In two articles previously written in this section on creative artificial intelligence,LLM or great language models,chatbot we have explained to you that ChatGPT, Gemini, Copilot and Claude, among others, are not really intelligent, that is, they do not understand the facts, reason about them and draw conclusions. They are mere probabilistic machines that return the next most probable word in a sequence of words and link it back to the input, and “learn” those probabilities from the texts. However, by using extraordinarily large and complex structures to store the probabilities learned and to recalculate with them, the use of extraordinarily large collections of texts in the “learning process”, which is long, difficult and expensive, and which can be given very long sequences of words, leads to a good assimilation of the syntax and semantics of the language (of several languages) and of a great general knowledge of the world, being able to imitate those that seem to have a certain logical capacity. In this way, they can perfectly answer or solve many questions or tasks, as we have all seen.

Error, especially in very specialized or current issues

However (and this we have all been able to verify), they often give wrong answers. In the technical terminology of AI, these are called “hallucinations” and it is not possible to predict when or why they will occur. Sometimes it is possible that there is too much information on the subject, that there is contradictory information, that fiction and reality coexist... (I mean, too many things in my head ;-) Or the way you ask has triggered a link to a wrong text. I don't know.

However, there are cases in which these large linguistic models are particularly poorly adapted. One of them is when the subject of the question is very specialized. It may happen that there are no texts of such a specialized subject or that precisely answer this question in the texts that have been given to them to “learn”, or that they exist but rarely appear and “lose” that knowledge among all others or “forget” the cause of everything that has later had to “learn”. Again, I don't know. This is often the case with questions about Basque culture, for example.

Something similar happens in questions that are very related to current affairs. As we have said before, the process of “learning” or training large language models is usually very long and expensive, even in very powerful data centers it can take several months. That’s why, by the time a new version of one of these chatbots comes out, the world that he knows is usually quite a few months earlier, and he doesn’t know anything about recent events, and so it will be for a few more months, until the next version comes out. It is therefore absolutely impossible for an LLM alone to respond well to recent events.

Finally, among the questions that a chatbot cannot answer is, of course, about information that is not publicly available. We will not be able to ask you about the information contained in any internal document of our company, association or organization, since this private document has not been seen by the chatbot during the training.

The possibility of giving incorrect answers in the cases mentioned (or sometimes certainty, as we have seen) is an even greater problem, taking into account that LLMs always respond with some certainty. In reality, they don’t say they don’t know something or are not sure, for this they are preceded by filters with predefined answers, but they are not perfect and they often fail too.

Solution: RAG technique

We have stated above that it is impossible for an LLM alone to respond to recent events. And the key to the solution lies in the accuracy of “just him.” Although the main component of a chatbot is its large language model, many other components are also needed. The filters mentioned above are an example. And another, very useful in all the sources of error that have been mentioned, is the RAG technique.

RAG stands for Retrieval Augmented Generation, or “search-assisted creation,” and this is basically what the technique consists of, taking advantage of the searches that are performed on a set of documents. Before passing the same question to the LLM, a search engine is asked to return documents that may be related to the question, and then the question and documents are passed to the LLM, telling the LLM to answer the question of using the information in those documents.

This greatly improves the chances of large language models responding well to the difficult types of questions listed in the previous section. General hallucinations are less likely to occur if you give him the document with the answer or a small set of documents than if he has to extract the answer probabilistically from all the information that he has coded in a diffuse way. For the same reason, it will also respond better to highly specialized consultations, especially considering that in these cases it often does not have this information. And the same in queries related to current affairs: although the LLM does not know anything about new events because it has never “seen” them, it will be able to respond well by passing the documentation related to them.

“Not only can you increase the accuracy of the answers, but you can also show the source on which the answer is based.”

In addition to increasing the accuracy of the responses, the RAG technique has another great advantage: it is possible to show the source on which the response is based. In this way, the user can be sure that the answer is correct or not (since the answers can also be erroneous with the RAG) and receive information beyond the answer and increase knowledge.

The RAG technique is not perfect either. To be useful, the search must work correctly. If the information that answers the question is not among the first search results, or if there is incorrect or contradictory information, then the LLM will have difficulty in providing an adequate answer based on them.

Although at first it was not, today the most popular chatbots allow you to take advantage of the RAG. ChatGPT, for example, launched the RAG in October 2024. When asking the chatbot, if we think that searching on the web will give us a better answer, we can mark the “Web Search” option; and even without us saying it, sometimes he decides that the search can help, and he does. In these cases, it initially displays a message like “Searching on the web...” and displays the sources along with the response. On the other hand, Google’s well-known NotebookLM tool is based on a concept similar to the RAG, but without searching: we upload directly the documents that it will use to answer the questions.

The RAG technique also serves to solve the last of the cases listed above, that of questions related to information within the organization itself. If we build a search engine that searches our internal documents, we can make a chatbot for internal use that will make our work easier. However, in this case we should also have the LLM within the organization, because it is not a good idea to upload our private documents to the chatbots of the technological giants... Fortunately, there are options for this, such as the chat developed by Orai: Kimu. It is a small SLM or language model (and therefore cheaper and more sustainable to implement) that works very well in routine work tasks and knows Basque perfectly, designed to be installed on its own servers. And with the same methodology (our search engine for publishable documents to perform the RAG and our own LLM or SLM), we can also build a specialized chatbot for our website.