Bookmark and Share

Alex Bäcker's Wiki / How to teach LLMs to know when they don't know
View
 

How to teach LLMs to know when they don't know

Page history last edited by Alex Backer, Ph.D. 2 years, 3 months ago

Transformers, introduced in the seminal Attention is all You Need paper, use the learned linear transformation and softmax function to convert the decoder output to predicted next-token probabilities. I propose here using the gradient of this function across the first successive tokens sorted from highest to lowest to determine whether the LLM knows the answer or not. Where a correct answer is known by the LLM, there should be a sharp decrease in the probabilities from the correct answer(s) to others. In contrast, where the LLM/AI does not know the correct answer, probabilities are expected to decrease more gradually or less sharply. Training a model on correct and invented answers could be used to find the right cutoff/threshold.

 

Comments (0)

You don't have permission to comment on this page.