InvestigationMachine Learning / Language Model Architectures

Beyond the Next Token: Are We Teaching AI to Generate Language or Organize Thought?

Language models traditionally generate text one token at a time. But must intelligence remain tied to that sequence? This investigation examines Next-Concept Modeling, latent reasoning, and hybrid AI architectures through published experiments. It evaluates whether hierarchical representations and internal computation can improve generation efficiency and problem-solving, while questioning whether these advances represent deeper reasoning or simply more efficient ways of processing information.

Type: Investigation
Field: Machine Learning / Language Model Architectures
Exhibits: 6
Published: October 22, 2026
Version: v1

01Research Question

Most language models are built around a simple objective: predict what comes next.

Given a sequence of tokens, the model learns to produce the next likely token. Repeating this process allows it to generate sentences, explanations, code, and complex responses.

But I think there is a more fundamental question worth investigating.

If the final answer must be expressed in language, does the process of arriving at that answer also need to happen through language?

Consider how people solve difficult problems. We do not always think through complete sentences. Sometimes we form a rough idea, examine its structure, consider several possibilities, and only afterward find the words to explain it.

This observation does not establish how human cognition works internally, but it motivates an important question about artificial intelligence.

Could a model become more efficient by organizing intermediate representations before producing its final sequence of words?

This investigation examines three related approaches:

Next-Concept Modeling, which generates hierarchical representations from coarse structure to finer detail.
Coconut, which explores computation in continuous latent states.
Hybrid Reasoning Agents, which combine internal latent computation with language-based interaction.

The central hypothesis is that models can improve computational efficiency by separating some internal processing from token-by-token language generation.

The competing hypothesis is equally important: these improvements may result primarily from better computational scheduling, compression, or training techniques rather than from a fundamentally superior form of reasoning.

The objective is to determine what the available experiments actually establish about generation efficiency, reasoning performance, and the relationship between internal representation and language.

→Next research

The Architecture of Stagnation

Why do dysfunctional systems persist even when their failures are widely recognized? The Architecture of Stagnation investigates how institutional inertia, entrenched incentives, concentrated power, and coordination failures sustain ineffective systems. Drawing on institutional economics, political theory, and historical case studies, it examines why recognizing failure rarely produces meaningful reform and explores the conditions under which lasting systemic change becomes possible.

The dominance of next-token prediction

Autoregressive language models produce output sequentially. Each new token is conditioned on the tokens that came before it.

This approach has demonstrated remarkable capabilities. Modern language models can write software, explain scientific concepts, solve mathematical problems, and interact with external tools.

It would therefore be incorrect to suggest that next-token prediction is incapable of supporting complex reasoning.

The more specific concern is computational.

Sequential generation creates dependencies between output positions. Producing a long response can require many successive model evaluations, even when parts of that response could potentially be planned or generated together.

Reasoning models can also allocate additional computation to difficult problems. OpenAI's API documentation describes internal reasoning tokens that allow models to consider intermediate steps before producing a response.

This demonstrates that visible output and internal computation are already treated differently in modern reasoning systems.

However, internal reasoning tokens remain part of a token-based computational process. They are not equivalent to continuous latent reasoning.

Next-Concept Modeling

In October 2026, Kumbong and colleagues introduced Next-Concept Modeling, or NCM.

The architecture uses a multiscale residual vector-quantized variational autoencoder to represent text through several levels of discrete codes.

A transformer then generates these representations from coarse to fine.

Rather than producing every output token sequentially, it predicts the codes within each level in parallel before eventually decoding the representation into text.

The authors report substantial speed improvements compared with selected diffusion language-model baselines.

The key innovation is not that the system has discovered human-like concepts. It is that the model learns hierarchical compressed representations and generates them through a shorter sequential process.

Coconut and continuous latent reasoning

Coconut, introduced by Hao and colleagues, approaches the problem differently.

Instead of converting every intermediate reasoning state into text, the model can reuse a continuous hidden state as the representation of its next reasoning step.

The authors investigate whether this enables more effective exploration of alternative reasoning paths.

Their results on selected logical reasoning tasks suggest that continuous latent computation can improve the balance between reasoning accuracy and efficiency.

Hybrid Reasoning Agents

HybridAgents extends this direction into environments requiring tool use and external interaction.

The architecture allows the system to transition between latent computation and text-based actions.

This distinction matters because an agent cannot operate entirely through private internal representations. It still needs to communicate with users, call tools, interpret observations, and provide understandable results.

Together, these approaches raise an architectural question:

Should language remain the default medium for every stage of AI computation, or should it primarily serve as one interface between internal representations and the external world?

03Methodology

This investigation uses a comparative secondary-research methodology.

I examine published architecture descriptions, experimental results, benchmark evaluations, and technical documentation to assess whether language generation and internal computation can be organized more efficiently.

The analysis focuses on three primary research systems:

1. Next-Concept Modeling (NCM)

Examined as an approach to hierarchical, coarse-to-fine text generation.

Its reported results are evaluated using generation quality, function evaluations, inference speed, and reasoning-related benchmark performance.

2. Coconut

Examined as an approach to continuous latent reasoning.

Its experiments are used to assess the potential value of reasoning without explicitly generating every intermediate step as language.

3. Hybrid Reasoning Agents

Examined as an architecture combining latent computation, text generation, and external tool interaction.

Its evaluation provides evidence about performance, token consumption, inference latency, and computational cost in agentic settings.

Supporting sources include OpenAI's reasoning-token documentation and Inception Labs' description of the Mercury diffusion language-model family.

Analytical framework

I evaluate the research through five dimensions:

Representation: What information does the model generate or manipulate internally?
Generation dependency: Which operations must happen sequentially, and which can happen in parallel?
Reasoning quality: Does the architecture improve performance on tasks requiring planning, constraints, or multiple reasoning steps?
Computational efficiency: How much time, computation, or token usage is required to achieve a particular result?
Generalization: Do the observed advantages extend beyond the original benchmarks and model scales?

I distinguish direct experimental measurements from theoretical explanations and interpretations.

Reported performance improvements are attributed to the original researchers. I do not present their experiments as independently reproduced results.

An important methodological restriction is that the systems are evaluated on different tasks, model sizes, and hardware configurations.

Their reported speedups and accuracy improvements therefore cannot be treated as results from one controlled experiment.

The investigation aims to identify plausible architectural principles, evaluate supporting evidence, and develop testable hypotheses for further research.

04Evidence

Evidence I — Next-Concept Modeling

Kumbong and colleagues introduced Next-Concept Modeling in October 2026.

Their architecture represents text through a hierarchy of discrete latent codes. A multiscale residual VQ-VAE produces the representation, while a transformer predicts each level in sequence, generating codes within the same level in parallel.

This reduces the number of sequential generation stages compared with producing every output token independently.

The authors evaluated NCM on language generation, translation, summarization, and Sudoku.

On the One Billion Word (LM1B) benchmark, NCM achieved better generative perplexity than the strongest diffusion baseline used in the reported comparison while generating samples 21 times faster.

On OpenWebText, the authors reported competitive generation quality at approximately 10 times the speed of the selected diffusion baseline.

On XSum summarization, NCM matched the strongest diffusion baseline's ROUGE-L performance while achieving a reported 31-fold speedup.

The research also evaluated Sudoku, a task requiring outputs to satisfy multiple simultaneous constraints. NCM outperformed the evaluated autoregressive and diffusion baselines across the reported difficulty levels while using fewer function evaluations.

These results support the possibility that hierarchical generation can reduce sequential computation without necessarily sacrificing output quality.

They do not establish that NCM is universally faster than modern autoregressive systems or that its discrete representations correspond to human-like concepts.

Evidence II — Coconut

Hao and colleagues investigated reasoning in a continuous latent space through their Coconut architecture.

In conventional chain-of-thought approaches, intermediate reasoning is represented through generated text.

Coconut instead uses the model's last hidden state as a continuous reasoning representation and feeds that state back into subsequent computation.

This allows the system to perform intermediate operations without decoding every step into words.

The authors report that Coconut outperforms textual chain-of-thought approaches on selected logical reasoning tasks requiring substantial search and planning, while achieving a favorable accuracy-efficiency trade-off.

They also describe the ability of continuous representations to preserve multiple possible next steps before committing to a particular reasoning path.

This result supports investigating alternatives to fully verbalized intermediate reasoning.

However, improved performance on selected tasks does not establish that continuous latent representations reliably preserve multiple hypotheses in all settings or that the approach will generalize to every class of reasoning problem.

Evidence III — Hybrid Reasoning Agents

Zou and colleagues introduced Hybrid Reasoning Agents in October 2026.

Their architecture combines continuous latent computation with text-based interaction, allowing an agent to perform internal processing while retaining the ability to communicate and use external tools.

The researchers evaluated HybridAgent at three model scales across eleven benchmarks involving mathematics, science, medicine, search, and agentic reasoning.

They reported an average performance improvement of 10.4% across their comparisons.

The experiments also reported:

Token-use reductions ranging from 35.5% to 54.7%.
Inference speed improvements ranging from 1.8 to 2.6 times.
Computational cost reductions ranging from 2.7 to 3.7 times.

These results suggest that assigning different computational roles to latent states and language-based interaction can improve efficiency while maintaining or improving benchmark performance.

The reported results remain specific to the authors' training procedures, selected baselines, model configurations, and evaluation settings.

Evidence IV — Commercial Diffusion Models

Inception Labs' Mercury announcement provides an industry example of non-autoregressive language generation moving beyond academic experiments.

The company described Mercury as a diffusion-based language-model family and reported generation speeds exceeding 1,000 tokens per second on NVIDIA H100 hardware for its code-generation system.

It also claimed substantial speed advantages over selected conventional language models.

These numbers demonstrate the performance targets being pursued by commercial diffusion systems, but the announcement is vendor-reported evidence rather than an independent comparison.

Mercury also uses diffusion-based generation, not NCM's multiscale discrete-code architecture. It should not be used as direct experimental validation of NCM.

Evidence V — Existing Internal Reasoning

OpenAI's reasoning-model documentation describes systems that perform internal reasoning before or between visible responses.

The documentation distinguishes input tokens, visible output tokens, and internal reasoning tokens.

This matters because it demonstrates that an AI system's computation is not limited to the language ultimately shown to the user.

However, hidden reasoning tokens remain tokens. Their existence does not establish that the model reasons in a continuous latent space like Coconut.

Evidence Synthesis

The five sources support three observations.

First, generating text sequentially is not the only viable architecture for language production.

Second, some reasoning operations can be performed using representations other than explicitly generated natural-language steps.

Third, coordinating different computational modes can improve efficiency on selected tasks.

The evidence does not establish that these architectures are universally superior or that they reproduce human thought.

What it establishes is a credible research direction: the structure of internal computation can be redesigned independently of the structure of the final language output.

06Findings

After comparing the research, I arrive at five findings.

1. The next token is an output decision, not necessarily the smallest useful unit of computation.

Autoregressive generation organizes output one token at a time.

NCM demonstrates that text can instead be represented through multiple levels of compressed information, allowing several lower-level components to be produced in parallel.

This suggests that the best unit for generating language may differ from the best unit for organizing the information behind it.

It does not mean that NCM has identified genuine semantic concepts as humans understand them.

2. Reasoning and explanation are different operations.

Coconut provides evidence that some intermediate computations can occur without generating a natural-language explanation at every step.

A model may perform useful operations on internal states before expressing its final answer.

This challenges the assumption that a model must verbalize every reasoning step to perform useful reasoning.

However, it does not establish that latent reasoning is inherently more accurate or reliable.

3. Internal computation and external interaction may require different representations.

HybridAgent is particularly important because it investigates how a system can combine different computational modes.

Internal processing can occur in latent space, while language remains available for tools, observations, and communication.

This suggests that future AI systems may benefit from separating private computation from externally interpretable actions.

4. Efficiency should be measured against the quality of the result.

The objective is not simply to generate fewer tokens or run faster.

An efficient system must still produce correct, useful, and reliable outputs.

A model that generates an answer twice as quickly but makes substantially more mistakes may provide little practical benefit.

The meaningful comparison is the cost and latency required to achieve a defined level of task performance.

5. The deeper research question is computational organization.

My interpretation is that these studies are exploring different ways to organize computation before producing an answer.

NCM investigates hierarchical generation.

Coconut investigates continuous intermediate reasoning.

HybridAgent investigates movement between internal and external computational modes.

Their architectures differ, but they challenge the assumption that every computational step must be organized around visible language generation.

This leads to a working hypothesis:

An effective AI architecture should allocate computation according to the structure and difficulty of the problem, rather than requiring every task to follow the same sequence of intermediate representations.

This is a proposed synthesis, not an experimentally validated new architecture.

07Counterarguments

The argument for moving beyond token-by-token generation has several weaknesses that must be addressed.

Next-token prediction already works remarkably well.

Modern autoregressive models demonstrate strong capabilities in reasoning, programming, mathematics, and tool use.

A system does not need an explicit hierarchy of human-readable concepts to learn useful internal representations.

Transformers already construct rich hidden states while predicting tokens.

Therefore, the existence of NCM or latent reasoning does not prove that next-token prediction is fundamentally inadequate.

Faster generation does not necessarily mean better reasoning.

NCM's reported speed improvements are substantial, but many involve comparisons against diffusion language models.

They cannot automatically be extended to highly optimized autoregressive inference systems.

Likewise, improvements on Sudoku do not establish superiority across scientific reasoning, software engineering, or real-world decision-making.

Continuous representations can reduce interpretability.

Text-based reasoning is imperfect, but intermediate text can be inspected and evaluated.

Continuous hidden states are more difficult to interpret directly.

A model might perform useful computation internally without providing a faithful account of how it reached a conclusion.

That creates challenges for auditing decisions, identifying errors, and establishing accountability.

Hierarchical representations introduce additional complexity.

NCM requires a separately trained multiscale tokenizer and a generative model.

The quality of the compressed representation may constrain what the generator can ultimately produce.

Fixed hierarchical structures can also impose unnecessary computation on simple tasks.

The original NCM paper acknowledges limitations involving fixed hierarchy depth, separate tokenizer training, and relatively limited model scaling.

Results are not directly comparable.

NCM, Coconut, and HybridAgent use different model architectures, datasets, training procedures, and evaluation criteria.

Their reported improvements cannot be combined to infer that a single hybrid architecture would automatically inherit every advantage.

Language itself may be computationally useful.

Words provide structure, abstraction, and a relatively interpretable way to represent complex relationships.

For some problems, explicitly expressing intermediate reasoning may improve verification and coordination.

The strongest conclusion is therefore not that language should disappear from reasoning.

It is that language should not be assumed to be the most efficient representation for every intermediate operation.

01Research Question

The Architecture of Stagnation

02Background

The dominance of next-token prediction

Next-Concept Modeling

Coconut and continuous latent reasoning

Hybrid Reasoning Agents

03Methodology

Analytical framework

04Evidence

Evidence I — Next-Concept Modeling

Evidence II — Coconut

Evidence III — Hybrid Reasoning Agents

Evidence IV — Commercial Diffusion Models

Evidence V — Existing Internal Reasoning

Evidence Synthesis

05Exhibits

NCM — LM1B Generation Speedup

NCM — XSum Summarization Speedup

HybridAgent — Lower Bound of Reported Token Reduction

HybridAgent — Reported Average Performance Improvement

Coconut — Reasoning in Continuous Latent Space

Next-Concept Modeling — Hierarchical Text Generation

06Findings

1. The next token is an output decision, not necessarily the smallest useful unit of computation.

2. Reasoning and explanation are different operations.

3. Internal computation and external interaction may require different representations.

4. Efficiency should be measured against the quality of the result.

5. The deeper research question is computational organization.

07Counterarguments

Next-token prediction already works remarkably well.

Faster generation does not necessarily mean better reasoning.

Continuous representations can reduce interpretability.

Hierarchical representations introduce additional complexity.

Results are not directly comparable.

Language itself may be computationally useful.

08Limitations

09Conclusion

10Sources

Primary Research

Implementation Repositories

Supporting Technical Sources

Research Attribution