Position paper · October 2026

Order, time and memory in sequence models

Spiking, associative and transformer approaches, 2007 to 2026 · Joy Bose, independent researcher, Bengaluru

A plain-language narrative version of this work is on the research thread page.

Abstract

This paper summarises a research programme that began with a spiking sequence machine built for a 2007 doctoral thesis at the University of Manchester. The system encoded symbols as rank order spike codes, held recent context in a running state, and retrieved continuations from a Sparse Distributed Memory. From 2025 the design was re-examined using vocabulary that postdates it: transformers, attention, and modern associative memory. The programme asks whether sequence processing can be organised around sparse, temporally ordered representations, a persistent state, and associative retrieval, and what such an organisation would offer in memory, continual learning, temporal structure and energy use. Five recent studies are summarised and placed against current literature, followed by open problems and predictions that could falsify the approach. No claim is made that spiking networks outperform transformers.

Keywords: spiking neural networks, rank order coding, sparse distributed memory, associative memory, attention, positional encoding, neuromorphic computing, hierarchical temporal memory

Section 1

Background

1.1 The 2007 sequence machine

The thesis, Engineering a sequence machine through spiking neurons employing rank order codes, addressed a problem that remains current: representing a sequence, remembering what has just occurred, using recent history to disambiguate otherwise identical inputs, retrieving the correct continuation, and continuing to learn online.

The design stored information in which neurons fired and in what order, rather than only in dense numeric activations. Four components worked together: rank order codes in a sparse "N of M" form; a running context state; an associative store (Sparse Distributed Memory); and a leaky integrate and fire variant, the Rate Driven LIF, tuned so that firing order within a burst, rather than firing rate or absolute spike time, carried the identity of each symbol.

Order is needed because the same input has different meaning in different contexts. An input X following the sequence A, B is a different situation from X following C, D. The input is identical and the context differs, so the representation of order must carry information (Figure 1).

A C B D

one firing order

C A D B

same neurons, different order, different meaning

Figure 1. In a rank order code the set of active units is the same in both rows. Only the firing order differs, and that difference carries the symbol.

1.2 Reinterpretation after 2017

The system predates the transformer and is not one. A functional comparison is nevertheless possible, because both must perform the same set of operations: encode the input, hold context, compare the current state against stored states, retrieve what is relevant, and produce the next output. The reconstruction used for this comparison is termed STSM, the Spiking Transformer Sequence Machine. Table 1 gives the correspondences used. They are functional analogies, not equivalences.

**Table 1.** Functional correspondence between the 2007 machine and a transformer.
2007 component	Transformer role	Limit of the analogy
Rank order spike code	Token embedding	Sparse and ordinal, not a dense real vector.
Spike timing	Positional information	Order is intrinsic to the code, not added to it.
Running context state	Query	A single decaying state, not a per layer, per head projection.
Stored patterns	Keys and values	Written by a Hebbian rule, not learned by gradient descent.
Similarity based retrieval in memory	Attention	Retrieves from a stored set, not over the current sequence.
Winner take all readout	Decoding	A hard selection, not a softmax over a vocabulary.

Section 2

Contributions of the programme

Four studies address different aspects of the question, and a fifth, related study applies the same family of ideas to streaming data. Each is summarised by its question, method, findings and limitations.

theory

2.1 Spiking sequence machines and transformers

Question. Why do a neuroscience derived sequence machine (2007) and an engineering derived transformer (2017) perform equivalent operations?

Method. Both are decomposed into five functions: encoding, context, associative retrieval, storage and readout. The shared primitive is comparison of high dimensional vectors by similarity (cosine or dot product). A "phase latency isomorphism" is derived: sinusoidal position signals and spike latencies are two coordinate systems for the same ordered quantity, and the attention dot product is indifferent to which is used, up to a scale factor.

Findings. On a copy task, compressing the position signal until positions are no longer distinguishable stops convergence. A learned rank based position performs as well as the sinusoid, and in some settings better. The relevant property is therefore that positions remain separable under similarity lookup, not any particular formula. Spiking implementations of the underlying memory match conventional ones in capacity across several neuron models (Ajwani et al., 2021, on the Nengo framework).

Limitations. The experiment is a copy task. Longer sequences, natural language data and a wider baseline set are not yet covered.

arXiv 2605.00662 code stsmpaper

memory

2.2 Rank order N of M codes for Sparse Distributed Memory

Question. Which component of the original memory is responsible for its robustness to noise: rank order coding, sparsity, the memory itself, the learning rule, or spiking dynamics?

Method. The components are separated and compared under matched conditions, using modern embeddings (GloVe, sentence embeddings) as well as binary patterns, and compared against recent memory augmented and state space systems including CALM and SpikingMamba.

Findings. Most of the robustness arises from the interaction of rank order coding with a MAX Hebbian learning rule. Rank order coding alone helps, with a smaller effect. Performance depends on representation, memory, learning rule and retrieval jointly rather than on one ingredient.

Limitations. Whether the advantage persists on a downstream retrieval task at equal memory and compute cost is not established.

arXiv 2607.02967 code sequence-machine-revisited

substrate

2.3 Pendulum model of spiking neurons

Question. Can a neuron model generate phase and timing from its own dynamics, instead of acting as a leaky counter that thresholds summed input?

Method. The neuron is modelled as a damped, driven pendulum, a second order nonlinear system. The paper analyses single neuron behaviour, extends to layers with spike timing dependent plasticity (STDP), provides Python and Brian2 code, and outlines an approximation for neuromorphic hardware.

Findings. Oscillation and phase arise from the neuron dynamics. This gives a possible intrinsic clock and ordering signal in place of an externally added position vector.

Limitations. This is the most exploratory component. No task is yet shown on which it outperforms a standard LIF neuron, and the hardware mapping is a sketch.

arXiv 2507.22146 code pendulum-spiking-model

space

2.4 Temporal coding for the Thousand Brains architecture

Question. Does the order in which a sensor encounters features carry spatial information that a dense, unordered representation discards?

Method. In Monty, the open Thousand Brains framework, the per contact dense vector is replaced with rank order spike packets. The most strongly responding unit fires first, and the interval between bursts encodes displacement without explicit coordinates. STDP learns sweep direction, and a learnable parameter weights earlier against recent contacts. The change is about 450 lines of code.

Findings. On synthetic objects composed of identical features in different arrangements, temporal coding separates objects that the unordered representation cannot.

Limitations. Evaluation uses synthetic objects only. Real sensor data has not been tested.

arXiv 2605.22206 code temporal-coding-tbt

streaming

2.5 Related study: HTM for drift detection

Context. Hierarchical Temporal Memory (Hawkins, Numenta) and the Thousand Brains theory model cortical columns using sparse distributed representations and a sequence memory that predicts the next input. They share with the spiking work sparse codes, associative sequence memory and online learning from a live stream without a separate training phase. The open htm.core library is the usual implementation.

Method and findings. HTM monitors a stream of telecom performance metrics, and a Sequential Probability Ratio Test decides when a change in pattern is genuine. On live data the hybrid detected drift with fewer false alarms than Kolmogorov Smirnov and Wasserstein based tests.

Link to the spiking line. The two lines meet in the Thousand Brains study (2.4), where rank order spikes are placed inside Numenta's Monty framework.

arXiv 2504.18599 code htm_drift_detection

Section 3

State of the field

As of October 2026, six lines of work bear directly on the questions above. Citations are listed in Section 9.

3.1 Attention as associative memory

Ramsauer et al. (2020) showed that the update rule of a continuous modern Hopfield network is equivalent to transformer attention, and that such a network can store a number of patterns that grows exponentially with dimension. This is the principal formal link between the associative memory literature and current models. It also establishes that the identification of attention with memory retrieval is prior work. The open question is narrower: whether the memory must be dense and softmax based, or whether sparse, rank coded memory can perform part of the function.

3.2 Spiking transformers and language models

Spiking language models include SpikeBERT, SpikeGPT and SpikeLLM. Sorbet (Tang, Yan and Wong, ICML 2025) derives a binary spiking model from BERT and replaces softmax and layer normalisation with shift based operations, since neuromorphic hardware handles division and square roots poorly. The authors report 27.16 times lower energy than BERT on GLUE. The result demonstrates feasibility at BERT scale on classification tasks. The energy figure is as reported by the authors, and evidence at frontier language model scale is not yet available.

3.3 Neuromorphic hardware

Intel's Hala Point integrates 1,152 Loihi 2 chips with 1.15 billion neurons and was delivered to Sandia National Laboratories in 2024. Intel describes it as a research system. Sandia also operates a SpiNNaker 2 system, the second generation of the architecture originating at Manchester and developed with TU Dresden, and reports fitting a complete fruit fly connectome onto 12 Loihi 2 chips. System scale is no longer the main limit. Software tooling, training methods and workloads on which the hardware clearly wins are the more commonly cited constraints.

3.4 Cost aware model routing

FrugalGPT (Chen, Zaharia and Zou, 2023) formalised LLM cascades, which query cheaper models first and escalate only when needed. On the tasks studied it matched the best single model with up to 98 percent lower cost. The approach operates within one family of system, since every tier is a language model accessed through an API.

3.5 History as state

State space models, Mamba style architectures and recurrent memory transformers carry the past forward as a compact state instead of recomputing attention over the full context at each step. The 2007 context layer, with a single forgetting parameter, is an early and much simpler instance of the same design choice.

3.6 Memory outside the weights

Retrieval augmented generation, vector stores and agent memory place knowledge outside the parameters so that it can be written and updated without retraining. Most implementations use dense embeddings with nearest neighbour search, and the coding of the stored memory is rarely varied. Whether rank order codes with a Hebbian store offer advantages in robustness, capacity per bit or cost remains an open empirical question.

3.7 Summary

These lines converge on four themes reached from different directions: sparse activity, explicit memory, state in place of recomputation, and allocation of compute according to need. They originate respectively in hardware, retrieval, sequence modelling and neuroscience research, and are only partly integrated.

Section 4

Positioning of the contributions

Table 2 separates what is established in the literature from what each study adds and what remains open.

**Table 2.** Contributions relative to prior work.
Study	Related literature	Established	Contribution	Open
2.1 Spiking machines and transformers	Attention as a Hopfield read (3.1); positional encoding	Attention equals a modern Hopfield update. Sinusoids are one position code among many.	Five function decomposition; phase latency view; position judged by separability under similarity lookup; copy task result.	Longer sequences, natural language, wider baselines.
2.2 Rank order N of M memory	Explicit memory (3.6)	Hebbian associative stores are long established. Dense retrieval is the default.	Ablation attributing robustness to rank coding combined with the MAX Hebbian rule.	Downstream retrieval at equal memory and cost.
2.3 Pendulum neuron	Neuron models; oscillatory models	LIF and variants are standard. Oscillatory neurons have a long history.	Damped, driven pendulum neuron with phase from intrinsic dynamics; STDP layers; code.	A task where it beats LIF; a tested hardware mapping.
2.4 Thousand Brains temporal coding	Sensorimotor learning	Monty treats features at each contact as an unordered set.	Rank order packets encoding displacement, in about 450 lines.	Real sensor data.
2.5 HTM drift detection	Streaming and online learning	HTM and sequential probability ratio tests are established.	Hybrid evaluated on live telecom metrics with fewer false alarms than KS and Wasserstein tests.	Other streams and drift types.

The contributions are small and checkable findings about which ingredient of a design is responsible for an effect. None is a new architecture shown to exceed a standard one on a standard benchmark.

Section 5

Open problems and research directions

This section states directions and hypotheses. They are not results, and each is labelled with its evidential status.

5.1 Position encoding judged by retrieval geometry

If positional encoding serves to keep order separable under the similarity operation, position codes can be designed for the memory they feed rather than adopted by convention. Rank based, phase based, latency based and learned codes can be compared on equal terms. Gains would be expected first in length generalisation and robustness to timing noise, where built in order has a mechanistic reason to help, and less so in headline accuracy on standard language benchmarks. Hypothesis, testable

5.2 Writable explicit memory

A system that learns an item and uses it moments later without retraining requires cheap writes, graceful forgetting and retrieval tolerant of noisy cues. Sparse Hebbian stores are well suited to the first and third by construction. Deciding what to write and when to overwrite is unresolved. Need established, design open

5.3 Allocation of computation across heterogeneous substrates

The cascade idea of Section 3.4 can be generalised from a sequence of language models to a sequence of different computational mechanisms. One illustrative ordering, cheapest first, is shown below.

Exact lookup or cache (near zero cost)
Deterministic rules (regular expressions, parsers)
Classical machine learning (tree ensembles, linear models)
Associative memory or an event driven network (streams, pattern recall)
A small language model
A large language model (only if lower tiers fall short)

The selection problem can be stated as constrained minimisation:

minimise E(a, x) energy of pathway a on input x subject to Q(a, x) >= Q_min required quality L(a, x) <= L_max allowed latency

The cascade component is prior work (FrugalGPT). Less examined is the inclusion of non language and non transformer tiers in the same ladder and the reporting of measured energy rather than operation counts. The distinction matters because fewer operations do not imply proportionally less energy, and activity sparsity reduces energy only on hardware that can skip inactive units. Framing sound, benefit of non transformer tiers untested

5.4 Domains of applicability

The following assignment is a working hypothesis, not a measured result.

Likely favourable	Likely unfavourable at present
Always on sensing under tight power budgets; event camera and audio streams; streaming anomaly and drift detection; control loops with timing constraints; memory written online.	Large scale dense text generation; settings that depend on mature GPU tooling and large pretraining recipes; tasks where accuracy, not energy, is the binding constraint; problems with no natural event or time structure.

5.5 Algorithm and hardware co-design

Work such as Sorbet adapts algorithms to the operations neuromorphic chips support. The converse question is which memory designs map most directly onto event driven hardware. Retrieval based on rank codes may map more naturally than softmax attention, but this has not been measured. Conjecture

Section 6

Falsifiable predictions and planned work

6.1 Results that would count against the programme

A rank or phase based position code that underperforms the sinusoid on long sequences of real language.
A rank coded explicit memory that shows no gain over standard retrieval at equal memory and cost.
An energy advantage that disappears when measured on hardware, or with a faithful hardware model, instead of estimated from operation counts.
A five function correspondence between the 2007 machine and the transformer that holds only on toy tasks.
A routing ladder in which non transformer tiers are rarely selected because a small language model already suffices.

6.2 Planned experiments, in order

Position codes at scale. Extend the copy task to longer sequences and real text, comparing rank, phase, learned and sinusoidal positions under one similarity lookup.
Memory against retrieval. Compare the rank coded memory with dense retrieval on one downstream task at a fixed memory and cost budget.
Measured energy. On one small task, report joules for each candidate pathway.
Streaming benchmark. Compare an HTM style memory, a small spiking network and a small state space model on sensor like streams.
Routing. Once the preceding components exist, test whether routing across them beats always invoking the largest model on a mixed task set.

6.3 Scope and non-claims

The programme does not claim that spiking networks outperform transformers.
The links between attention, Hopfield networks and associative memory are prior work (Ramsauer et al., 2020).
Combining spiking computation with language models is an active area with many contributors.
Biological plausibility is a source of hypotheses and is not treated as evidence for a design.

Section 7

Chronology

2007

Doctoral thesis, University of Manchester: sequence learning from spiking neurons, rank order codes, running context and Sparse Distributed Memory. Supervisors: Steve Furber and Jon Shapiro.

2021

Spiking Sparse Distributed Memory implemented on Nengo (Ajwani, Lalan, Sen Bhattacharya, Bose).

2025

Re-examination of the 2007 design in transformer and associative memory terminology (STSM). Pendulum model of spiking neurons. HTM based drift detection.

2026

Studies on spiking sequence machines and transformers, temporal coding for the Thousand Brains architecture, and rank order memory.

October 2026

Programme placed against the current literature, with open problems and falsifiable predictions (this paper).

Section 8

Companion essays

Longer, general audience treatments are published on Medium.

Mar 2026 · 15 min read

What I Built in 2007, and Why It Looks a Bit Like a Transformer

Layer by layer comparison: the N of M encoder against token embeddings, the context layer and its forgetting parameter against the KV cache and Mamba style gating, the memory address decoder against the query key dot product, the max outer product Hebbian rule as a one shot local alternative to backpropagation, and burst stability analysis as a spiking analogue of vanishing and exploding gradients.

Jul 2025 · 14 min read

Novel transformer architectures with RNNs, spiking and shallow ML models

Seven hybrid designs, each addressing one transformer limitation: LSTM attention for temporal order, spiking transformers for energy, HTM transformers for online learning, binary associative memory for caching lookups, Hopfield memory for content addressable retrieval, shallow early exit gates for adaptive compute, and state space hybrids for linear cost long context.

Jul 2025 · 7 min read

References and resources

Bose, J. (2007). Engineering a sequence machine through spiking neurons employing rank order codes. PhD thesis, University of Manchester.
Ajwani, Lalan, Sen Bhattacharya and Bose (2021). Spiking Sparse Distributed Memory on the Nengo framework. arXiv:2109.03111.
Ramsauer et al. (2020). Hopfield Networks is All You Need. arXiv:2008.02217.
Tang, K., Yan, Z. and Wong, W.-F. (2025). Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model. ICML 2025. arXiv:2409.15298.
Chen, L., Zaharia, M. and Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176.
Intel Labs (2024), Hala Point announcement; Sandia National Laboratories, Advancing neuromorphic computing at the Neural Exploration and Research Lab.

Studies summarised above: 2605.00662 (spiking machines and transformers), 2607.02967 (rank order memory), 2507.22146 (pendulum neuron), 2605.22206 (Thousand Brains temporal coding), 2504.18599 (HTM drift detection).