Hey there, fellow tech enthusiasts! Have you ever wondered how to evaluate the quality of document summaries generated by Large Language Models (LLMs)? I’m currently working on a project that involves building a simple document summarization platform, and I’m stuck on finding the perfect scoring method to analyze these summaries.
I’ve tried various techniques like BERTScore, MoverScore, G-eval, ROGUE, and BLEU, but the scores themselves don’t tell me much. I mean, what does a cosine similarity score of 0.7 really mean in terms of the summary’s quality? It’s hard to put these numbers into context.
I’ve also experimented with sending the summary to another decoder-only model to extract key facts or questions and then running them through a BERT NLI model against chunks of the source material. This approach seems promising, but I’m not thrilled with the results.
My question is: Does anyone have experience with evaluating document summaries generated by LLMs? Do you have any suggestions for methods to try or experiment with? I feel like this might be an area of ongoing research, but at this point, we’re just looking for something simple.
If you have any insights or ideas, please share them with me. I’d love to hear your thoughts on this topic.