How to Break the Context Barrier: Leveraging 12-Million-Token Windows with Subquadratic

Introduction

Today's frontier AI models boast context windows of a million tokens or more, but making full use of that information remains a challenge. The bottleneck? Attention cost in transformer models scales quadratically with input length—double your tokens, and you quadruple computational work. This is why many models, like Claude Opus 4.7, achieve only 32.2% on the MRCR v2 retrieval benchmark, while GPT-5.5 leads at 74.0%—still far from ideal. Workarounds like RAG, agentic decomposition, and hybrid architectures all trade off key capabilities.

Enter Subquadratic, a Miami-based startup. Its new model features a 12-million-token context window—the largest available—and claims to scale linearly in both compute and memory. The company's Subquadratic Selective Attention (SSA) architecture runs 52 times faster than dense attention at a million tokens, achieves 92.1% on needle-in-a-haystack retrieval at 12 million tokens, scores 83 on MRCR v2 (beating OpenAI by 9 points), and hits 82.4% on SWE-bench, outperforming Anthropic's Opus 4.6 (81.42%) and Google's Gemini 3.1 Pro (80.6%). All at a significantly lower cost. This guide walks you through understanding and leveraging this breakthrough.

What You Need

• Basic understanding of transformer models and the attention mechanism
• Knowledge of current context-window limitations and workarounds (RAG, agentic decomposition)
• Access to Subquadratic's API (available via their platform)
• Familiarity with performance benchmarks (needle-in-a-haystack, MRCR v2, SWE-bench)
• Interest in applications requiring massive context (e.g., long document analysis, code generation, deep research)

Step-by-Step Guide

Step 1: Understand the Quadratic Attention Bottleneck

Every transformer-based model since 2017 faces the same fundamental issue: attention cost scales quadratically with context length. If you double your input, the computational work quadruples. This is why frontier labs cap context windows at around a million tokens—going further becomes impractical without massive infrastructure. Recognizing this limitation is the first step toward appreciating the solution.

Step 2: Recognize Current Workarounds and Their Trade-Offs

To get around quadratic scaling, the industry relies on techniques like RAG (Retrieval-Augmented Generation), agentic decomposition, and hybrid model architectures. Each makes trade-offs: RAG loses some context coherence, agentic approaches require complex orchestration, and hybrids may sacrifice performance in narrow tasks. Subquadratic's approach aims to replace these workarounds entirely with a new architecture that scales linearly.

Step 3: Discover Subquadratic Selective Attention (SSA)

Subquadratic's 11 Ph.D. researchers developed SSA, which achieves linear scaling in both compute and memory relative to context length. At a million tokens, it runs 52 times faster than dense attention. This means you can process 12 million tokens in roughly the same time it takes a standard model to handle far fewer. The architecture is designed to maintain retrieval accuracy even at extreme lengths.

Step 4: Evaluate Performance Benchmarks

Before adopting any model, verify its claims. Subquadratic reports:

• Needle-in-a-haystack retrieval: 92.1% at 12 million tokens—no frontier model currently attempts this length.
• MRCR v2: Score of 83, beating GPT-5.5 (74%) by 9 points.
• SWE-bench: 82.4%, outperforming Anthropic's Opus 4.6 (81.42%) and Google's Gemini 3.1 Pro (80.6%).

These benchmarks demonstrate that SSA not only handles long contexts but also excels at reasoning and retrieval tasks where others falter.

Step 5: Access Subquadratic's API and Tools

Subquadratic makes its model available through an API featuring a 12-million-token context window. Additionally, they offer two specialized tools:

• SubQ Code: A coding agent that can analyze entire codebases in a single pass.
• SubQ Search: A deep research tool for comprehensive document analysis.

You can sign up via Subquadratic's website to get API keys and start integrating the model into your applications.

Step 6: Implement Ultra-Long Context Applications

With a 12-million-token context, you can perform tasks previously impossible: analyze entire legal contracts, review full code repositories, process months of customer support logs, or conduct deep research on massive corpora. Start by testing small-scale use cases, then gradually increase context length. Monitor retrieval quality and latency—Subquadratic claims linear scaling, but real-world performance may vary based on your infrastructure.

Tips for Success

• Plan for the future: Subquadratic plans a 50-million-context window soon. Design your applications with even larger contexts in mind to future-proof them.
• Optimize cost: Because SSA runs significantly cheaper than dense attention at scale, you can afford to process larger inputs without breaking your budget.
• Combine with other tools: While SSA reduces the need for RAG and agentic decomposition, you may still want to use them in hybrid configurations for specific tasks.
• Validate on your data: Always run your own benchmarks on domain-specific data to ensure the model meets your accuracy and speed requirements.
• Stay updated: Follow Subquadratic's research publications and API updates to leverage new improvements as they are released.