LongRope This is related to Google Gemini 1.5 Pro which can achive context lengths of up to 1 million tokens (and perhaps more). Paper: https://arxiv.org/pdf/2402.13753.pdf