LLMs have a “lost in the middle” problem – they focus on the start and end of documents but miss key info in between. (Adam Zewe, MIT News)
Summary
- Researchers uncover mechanism behind position bias in large language models.
- Certain design choices like attention masking can amplify bias toward start/end of text.
- Adjusting model architecture and training data can mitigate this "lost-in-the-middle" phenomenon.
