Joy Bose

Spiking neurons · sequence memory · 2007 to 2026

The same idea, twenty years apart.

In 2007 I built a machine that learned sequences out of spiking neurons. It sat quiet for almost twenty years. Then transformers, attention, and modern memory research handed me a new vocabulary for things I had already tried. This page is the whole thread, in plain words, with every paper and repo linked.

1st 2nd 3rd 4th 5th time →
Spike time, phase, and rank are three ways of writing the same order. identity + order + time
1
PhD thesis, 2007
4
recent papers
1
idea underneath
19
years between

Where it started

A machine that learned sequences out of spikes

My PhD at Manchester, finished in 2007, was a machine that learned sequences using spiking neurons. The title was long and the problem, looking back, was oddly modern: how can a machine represent a sequence, remember what just happened, use that recent history to tell apart otherwise identical moments, fetch the right continuation, and keep learning as it goes.

The trick was to stop storing information only as dense numbers. Instead, information lived in which neurons fired and the order in which they fired. Rank order codes, a sparse "N of M" representation, a running memory of recent context, and an associative store called Sparse Distributed Memory did the work together. The neurons were a leaky integrate and fire variant I called the Rate Driven LIF, tuned so that the firing order within a burst, not the firing rate and not the absolute timing, carried the meaning of each symbol.

Why does order matter so much? Because the same input means different things in different company. Seeing X after A then B is not the same situation as seeing X after C then D. The input is identical. The context is not. So the order itself has to carry information.

A C B D
one situation
C A D B
same neurons, different meaning

This was not a transformer, and I would never claim I invented attention in 2007. The interesting and defensible thing came much later: the old machine and the transformer seem to solve several core sequence problems using surprisingly similar operations. That observation is what the recent work is really about.

The turn, around 2025

Rereading the old design in the language invented after 2017

Around 2025 I started asking a simple question. If I described that old system using words that only exist because of transformers, attention, and modern associative memory, what would it look like? A rough mapping fell out almost on its own.

A rank order spike code plays the role of a token embedding. Spike timing plays the role of positional information. The running context state plays the role of the query. Stored patterns play the roles of keys and values. Similarity based retrieval in the memory plays the role of attention. Winner take all readout plays the role of decoding.

None of these map perfectly, and pretending they do would weaken the point. What actually holds up is that both systems have to do the same handful of jobs: encode the input, hold context, compare the current state against stored states, pull back what is relevant, and produce the next output. We started calling the reconstructed version STSM, the Spiking Transformer Sequence Machine, partly as an architecture and partly as a bridge for thinking. I worked through the full layer by layer mapping, down to the neuron model and the learning rule, in a companion essay.

The four papers

Four papers, one program

Each one takes a different slice of the same question. They are numbered in the order they fire, because here, order is the whole point. Every card links to the arXiv paper and the code.

theory

Spiking sequence machines and transformers

Attention may just be similarity based memory lookup, wearing a different hat

The question. My 2007 spiking machine and the 2017 transformer were built for completely different reasons, one from neuroscience and one from engineering. Why do they end up doing the same thing?

What it shows. Both perform the same five jobs: encoding, holding context, associative retrieval, storage, and readout. The shared move underneath is comparing high dimensional vectors by similarity, the cosine or dot product. I also work out a "Phase Latency Isomorphism": the sine and cosine position signal a transformer uses and the spike timing a neuron uses are just two coordinate systems for the same ordered quantity, and the dot product in attention does not care which one you feed it, up to a single scale factor.

The experiment. When I squash the position signal until positions stop being distinguishable, a copy task stops converging. A plain learned rank based position does as well as the sinusoid, sometimes better. So the sine waves are not the magic. What matters is that positions stay far enough apart under the similarity lookup.

Why it matters. If attention is a special case of similarity based retrieval, and position encoding is really about keeping order distinguishable rather than about any particular formula, then the design space for both is much wider than it looks.

Not just theory. Spiking versions of this memory have already been shown to match conventional ones in capacity, across several neuron models, in Ajwani, Lalan, Sen Bhattacharya, and Bose (2021), which built a spiking Sparse Distributed Memory on the Nengo framework. So the retrieval mechanism at the heart of the comparison is a working system, not a thought experiment.

arXiv 2605.00662 code stsmpaper
memory

Rank order N of M codes for Sparse Distributed Memory

Taking the old memory apart to see which piece was actually doing the work

The question. My old rank order memory shrugged off noise, but the system bundled several ideas together. Was it the rank order coding, the sparse representation, the memory, the learning rule, or the spiking hardware that made it robust?

What it found. I pulled the pieces apart and tested them under matched conditions. The big robustness did not come from rank order coding on its own. Most of it came from rank order coding working together with a MAX Hebbian learning rule. Rank order coding by itself still helped, but the effect was smaller. The honest conclusion is that performance depends on representation, memory, learning rule, and retrieval as a set, not on one fashionable ingredient.

The context. I tested with modern embeddings like GloVe and sentence embeddings, not just toy binary patterns, and set the old memory next to recent systems such as CALM and SpikingMamba. There is a neat loop here: modern AI is rediscovering explicit associative memory bolted onto a transformer, which is close to what the spiking work was already doing.

Why it matters. Rank order memory deserves a second look as a design space for memory in modern AI, as long as you isolate which old mechanism is really carrying the load instead of reviving the whole thing out of nostalgia.

arXiv 2607.02967 code sequence-machine-revisited
substrate

Pendulum model of spiking neurons

What if the neuron produced phase and timing on its own?

The question. Why do we model artificial spiking neurons as simple leaky counters that add up input and fire at a threshold? That throws away a lot of the temporal richness a real neuron has.

What it does. I model the neuron as a damped, driven pendulum, a second order nonlinear system. Oscillation and phase fall out of the neuron's own dynamics instead of being pasted on from outside. The paper works through single neuron behaviour, extends to layers governed by STDP learning, gives runnable Python and Brian2 code, and sketches how to approximate the equations on neuromorphic hardware.

Why it matters. Since the transformer work already made phase and timing the interesting quantity, a neuron that generates phase natively could give a system its own built in sense of order and clock, rather than a position vector added after the fact. This is the most speculative branch, and I say so plainly.

arXiv 2507.22146 code pendulum-spiking-model
space

Temporal coding for the Thousand Brains architecture

The order you touch things in tells you where they are

The question. The Thousand Brains model, and its open framework Monty, recognises an object by moving a sensor across it and gathering features one contact at a time. But it treats the features at each contact as an unordered bag. The order in which you meet features as you sweep across a surface carries spatial information, and a dense vector throws that away.

What it does. I replace the dense vector with rank order spike packets. At each contact, the neuron that responds most strongly fires first, and the gap between bursts quietly encodes how far the sensor moved, with no explicit coordinates. STDP learns the direction of the sweep. A learnable parameter sets how much earlier contacts matter versus the most recent one, so the system adapts to each object's shape. The whole thing is about 450 lines inside Monty.

The result. On synthetic objects built from the same features arranged differently, temporal coding tells them apart where the unordered version cannot.

Why it matters. The same principle from the language and sequence work shows up in touch and space. Order is information. Do not discard it and try to rebuild it later.

arXiv 2605.22206 code temporal-coding-tbt

A related family

The cousins: HTM and the cortical column

Not every neighbour is spiking in the strict sense. Hierarchical Temporal Memory, from Jeff Hawkins and Numenta, and its successor the Thousand Brains theory, model the cortex as columns of neurons using sparse distributed representations and a sequence memory that keeps predicting the next input. I treat them as close cousins of my spiking work, because they run on the same genes: sparse codes, associative sequence memory, and learning that happens online from a live stream with no separate training and testing split. The open htm.core library is where a lot of this lives in practice.

With two colleagues I put that online property to work on a real problem. HTM watches a stream of telecom performance metrics and notices when the pattern starts to shift, and a Sequential Probability Ratio Test decides when a shift is a genuine change rather than noise. The hybrid caught data drift with fewer false alarms than standard statistical tests such as Kolmogorov Smirnov and Wasserstein. It is the cousin family earning its keep on live data, not just on paper.

arXiv 2504.18599 code htm_drift_detection

The point where the two lineages actually touch is the Thousand Brains paper above, where I drop rank order spikes into Numenta's Monty framework. That is the spiking line and the HTM line meeting at a single spot.

What this suggests for modern AI

Five ways this could make transformers better

Some of these are close to ready and some are still ideas. The point is not to replace transformers or to resurrect old neural networks unchanged. It is to find the functional job each part does and ask whether a sparser, more temporal, more memory based version could do it too. I sketch concrete hybrid designs along these lines in a separate essay.

A Position encoding judged by retrieval geometry, not by formula

Stop asking which sinusoidal formula is best. Ask which ordered representation keeps positions most distinguishable under the similarity operation attention uses. That opens the door to rank based, phase based, latency based, and learned order representations on equal footing.

B Separate memory from model weights

Transformers cram enormous amounts of information implicitly into their parameters. An explicit associative memory sitting alongside the model lets you write a new experience immediately without retraining billions of weights. That helps with continual learning, episodic memory, personalisation, and forgetting less.

C Replace some attention with associative retrieval

Full self attention compares every token with every other token. A memory asks a narrower question: which stored patterns are similar enough to my current state? That points toward a sparse lookup that touches a small relevant set instead of the whole sequence.

D Make order part of the representation, not an add on

Transformers start with a content vector and then add position. A spike already carries identity and time together. A rank order code already carries feature and relative order together. So a representation could encode content, order, and context jointly from the start, instead of content plus a separate position term.

E Let history become a persistent state

My old machine kept an evolving context state. Much modern work, from state space models to recurrent memory transformers to Mamba like systems, is moving the same way for a similar reason: the past should turn into state, not get recomputed as a static block every step.

The one idea underneath

Order, time, and memory are the primitives, not the metadata

If I compress all of this into a single sentence, it is not that spiking neurons beat transformers. The papers do not show that, and framing them that way would be dishonest. The deeper claim is quieter and, I think, more useful.

Modern deep learning tends to start with a dense vector and then add machinery around it: a vector plus position, a vector plus attention, a vector plus external memory. The spiking view starts somewhere else. An event already has both what happened and when it happened. A run of spikes already carries identity, order, and timing. Give those patterns a place to persist and you have memory.

Classic transformer recipe

  • dense vectors
  • position added on
  • global attention
  • knowledge baked into weights

The spiking alternative

  • sparse events
  • order built in
  • similarity based retrieval
  • persistent state and explicit memory

The real research question behind everything on this page is whether sequence intelligence can be rebuilt around sparse, temporally ordered representations, a persistent state, and associative retrieval, and whether that buys some of the transformer's power together with better continual learning, memory, temporal structure, and maybe efficiency. Not either or. The strongest ideas from both.

How it unfolded

The chronology, in plain terms

2007 · PhD, Manchester

Can spikes learn sequences? A sequence learner built from spiking neurons, rank order codes, running context, and Sparse Distributed Memory.

2025 · the rereading

Why does this old thing look so familiar now? I start mapping the old design onto transformer, attention, and memory language. This becomes the working STSM idea.

2025 · pendulum neuron

Could the neuron itself carry phase? A second order pendulum model where oscillation and timing are intrinsic to the neuron.

2026 · spiking machines and transformers

What exactly do they share? Five common jobs, cosine similarity as the shared primitive, the Phase Latency Isomorphism, and experiments on position encoding.

2026 · temporal coding for Thousand Brains

Does order carry space too? The "order is information" principle extended from sequences to sensorimotor object recognition.

2026 · rank order memory, modernised

Which old mechanisms actually survive? The old memory taken apart experimentally and placed against contemporary neuromorphic and memory systems.

The long reads

Companion essays on Medium

Everything above is the short version. These are the essays where I take the same ideas slowly, plus the wider landscape they sit in. All written for a general reader.

Mar 2026 · 15 min read

What I Built in 2007, and Why It Looks a Bit Like a Transformer

The full retrospective behind this page. I lay the 2007 sequence machine next to a transformer, layer by layer: the rank ordered N of M encoder as token embedding, the context layer with its forgetting parameter as a scalar ancestor of both the KV cache and Mamba style gating, the Sparse Distributed Memory address decoder as the QK dot product, the max outer product Hebbian rule as a one shot local alternative to backpropagation, and the burst stability analysis as a spiking version of the vanishing and exploding gradient problem. This is the plain language source for the origin and the first paper above.

Jul 2025 · 14 min read

Novel transformer architectures with RNNs, spiking and shallow ML models

The fuller version of the five directions above. A thought experiment walking through seven hybrids, each aimed at one transformer weakness: attention over an LSTM for temporal order, a spiking transformer for energy, an HTM transformer for online learning and anomaly gated attention, a binary associative memory to cache repeated lookups, a Hopfield memory as principled content addressable retrieval, a shallow ML early exit gate for adaptive compute, and a Mamba style state space hybrid for linear cost long context.

Jul 2025 · 7 min read

Neuromorphic AI and Generative AI: Two approaches to Intelligence

The big picture framing. Generative AI mimics the output. Neuromorphic AI tries to mimic how the brain gets there, with spikes, event driven compute, and memory fused into processing. I cover the hardware (Loihi, TrueNorth, and Manchester's SpiNNaker), HTM and the Thousand Brains theory, the energy gap, and where the two approaches are likely to converge rather than compete.

Apr 2025 · 3 min read

Recent papers related to spiking neurons and LLMs

A short field guide to what other people are building at this intersection, so my own work has a map to sit on: BrainTransformers, SpikeGPT, SpikeBERT, SpikeLLM, Meta SpikeFormer, the Neuro LIFT drone framework, and a Loihi reinforcement learning controller, among others.

Jun 2026 · 20 min read · members only

The Hardware Lottery: Why Transformers Won, What Could Replace Them, and How the AI Bubble Could Burst

The economic and hardware backdrop for why any of this matters. Transformers won partly because their math happened to fit GPUs, not because they were the only good answer. I go through the efficient alternatives (xLSTM, Mamba, RWKV, state space models), custom silicon and neuromorphic hardware including SpiNNaker, the cheap Chinese models reshaping the market, and the financial bet riding on the current setup. This is the context that makes sparse, memory based, temporally aware designs worth taking seriously.