Brain inspired computing with SpiNNaker

Why neuromorphic computing?

"Don't simulate a brain, build one!"

• Motivation: design and build from the ground (e.g. transistor) up with a practical engineered system in mind

• Dedicated hardware places the brain in the physical world → embodied (neuro)science (e.g. robotics)

  • towards hardware/brain-in-the-loop ?

• Huge discrepancy between the efficiency of a biological brain and artificial brains

  • Biological brain communication is event-driven → asynchronous, local computing
  • Yet, over the shortest distances, communication is best described by analog processes (e.g., chemical process underlying synaptic dynamics)
  • von Neumann architecture bottleneck on conventional computers
  • while GPU and HPC are great at handling large data structures "linearly", brain-like communication involves transmitting small data packets (i.e. spikes) to many targets

∴ VLSI philosophy: combine IC fabrication and system-level architecture to produce truly integrated systems

• Computational neuroscience: neuromorphic hardware becomes a tool to validate hypotheses

"No neuromorphic system attempts to reproduce all of the biological detail, but all adhere to the idea that computation is highly distributed across small computing elements analogous in some way to neurons, connected into networks, with some degree of flexibility in the way connections are formed. That much is common; the details vary greatly." - Steve Furber

Early beginnings

"In working on the ACE I am more interested in the possibility of producing models of the action of the brain than in the practical applications to computing." - Alan Turing ca. 1946

• 1948: Alan Turing presents a connectionist approach to computing in Intelligent Machinery (Turing, 1948), with his concept of unorganized machine.

• 1948: Alan Turing works on the Manchester computers (Manchester Mark 1)

Fast forward 30 years ...

"All neuromorphic roads lead to Carver Mead." - Chris Eliasmith

• 1980: Carver Mead and Lynn Conway publish Introduction to VLSI Systems

• 1981: Carver Mead, John Hopfield and Richard Feynman collaborate on the course Physics of Computation

  • 1982: Hopfield popularizes a neural network model of memory which we now know as the Hopfield network
  • 1983: Feynman teaches "Potentialities and Limitations of Computing Machines" at Caltech, now known as the Feynman Lectures on Computation (1996) and eventually looked at quantum computing. and eventually looked at quantum computing.
  • 1986-1991: Mead and his students create the first neural-chip models to mimic touch (1986), hearing (1988) and vision (1991). Any working system made it to Mead's book Analog VLSI and neural systems (1989)

• 1991: Misha Mahowald creates the first silicon retina, precursor to the dynamic vision sensor (DVS) or event-camera

• 1990s: Mead and his students popularize the address-event representation (AER) protocol for transferring spikes between bio-inspired chips
  (high-level: allocating individual wires to each spike source, or neuron, is not feasible as it becomes increasingly more difficult to scale; instead, associate each spike source with a unique address and use a shared bus)

Meanwhile ...

• 1980s: Steve Furber is working for Acorn Computers Limited, which eventually became (in part) ARM
• 1990s: Steve Furber joins the University of Manchester, where he works on novel microprocessors and asynchronous computing.
  • While working on associative memory, the Advanced Processor Technologies group recognizes that the architecture they're after resembles that of a neural network
  • The group's expertise lies in digital architectures and processors, not so much in analog circuit design
• 2000s: Given the importance of scalability, Furber et al. revisit AER which has its limitations for large scale, multicore systems. They transform the bus-based protocol into a packet-switched system, with local routing table on each chips.

Today

Steve Furber et al., September 2020:

IBM TrueNorth

Primary use: real-time cognitive applications

Notable features:

• digital
• purely deterministic (i.e. software model is replicated exactly and software emulator may be used to predict performance of hardware)
• simulates leaky integrate-and-fire (LIF) neurons

Intel Loihi

Primary use: geared toward practical applications

Notable features:

• digital
• based on leaky integrate-and-fire (LIF) neuron
• programmable synapse models for on-chip learning
• ships in different shapes and sizes: Kapoho, Nahuku, Pohoiki

BrainScaleS (Heidelberg University, HBP)

Primary use: understanding biological systems, specficially long-term learning

Notable features:

• analog
• physical models of neuronal processes (ionic circuits → electrical circuits) based on the adaptive exponential integrate-and-fire (AdExp) neuron model
• can run in accelerated mode, up to 10 000 times faster than real-time, hence the long-term learning objective

SpiNNaker (University of Manchester, HBP)

Primary use: modelling biological nervous systems in biological real-time

Notable features:

• digital (ARM968 processors) (nitpicking: not neuromorphic per se, but "neurocomputer" or "neural accelerator")
• massively parallel computing
• programmable models (allows researching different neural models and plasticity rules)
• design guidelines: scalability and energy-efficiency

The efficiency stems from the highly distributed architecture and the close proximity of processing and memory units (i.e., frequently used data is kept within 1-2 mm of the processor on each core).

The scalability is supported by local routing and the ingenious organization/stacking of boards.

March, 2016: \(18 \times 48 \times 24 \times 5 \times 5 = 518 400\) core machine

October, 2018: \(18 \times 48 \times 24 \times 5 \times 10 = 1 036 800\) core machine

2021-2022: SpiNNaker2

Further reading

• SpiNNaker 2 (early stage, 2019)
• Book about the SpiNNaker project: SpiNNaker: A Spiking Neural Network Architecture, Furber & Bogdan (2020)
• sPyNNaker: using PyNN on SpiNNaker (2018, version: 4.0.0, latest: 6.0.0)