Talos

Talos is a custom FPGA-based hardware accelerator built from the ground up to execute Convolutional Neural Networks with extreme efficiency. It isn't just a reimplementation of existing software logic in hardware; it is a rethinking of how deep learning inference should work at the circuit level.

Most deep learning frameworks are built for flexibility. They handle dynamic graphs, varying batch sizes, and a multitude of layer types. Talos takes the opposite approach. It strips away the runtime, the scheduler, and the operating system overhead to expose the raw compute capability of the FPGA. By implementing the entire inference pipeline in SystemVerilog, we achieve deterministic, cycle-accurate control over every calculation.

The Reality of Hardware

But don’t let the two-week timeline fool you. Those were two weeks full of 18-hour days, fueled by caffeine and sheer stubbornness. Building hardware is a completely different game from software.

In software, if something breaks, you check the logs, you add a print statement, and you recompile in seconds.

Hardware isn’t really “debuggable” in the way software is. You are bound by physics and the hard constraints of silicon, and the margin for error is brutally small. When a signal misses a timing window even by half a nanosecond, the entire system can collapse. You can’t just scale up or throw more compute at the problem — you’re working within a fixed number of logic elements, a fixed amount of on-chip memory, and a fixed clock budget that everything must obey.

There were long stretches where we sat staring at waveforms for hours, trying to catch a single bit flipping at the wrong moment. It’s a level of granularity most software engineers rarely have to touch. You’re not just debugging code anymore; you’re negotiating with the machine itself, trying to convince the FPGA to accept your logic within its physical fabric. It’s mentally exhausting, occasionally frustrating, and completely unforgiving — but that’s also what makes it real.

PyTorch isn’t slow because it was built badly. It’s slow because it was built for flexibility. Training needs dynamic graphs, autograd, and a huge surface area of features. For inference, you end up paying that overhead every forward pass. Talos takes the opposite approach: strip away anything that isn’t the math, and make the whole pipeline deterministic in hardware.

What's Behind the Inference Pipeline

Talos' first inference pipeline was built around one radical simplification: *do only the math that matters, nothing else*. No runtime, no scheduler, no abstraction. Every operation is grounded in fixed-point arithmetic, every cycle is deterministic, every path through the hardware is known.

Model Specs and Training

The architecture is reasonably straightforward. The model has a single convolutional layer which takes a 28×28 grayscale input and applies 4 kernels of size 3×3 with no padding, producing 4 feature maps of size 26×26. A ReLU activation layer follows with a 2×2 MaxPool layer with stride 2 which reduces the dimensions to 13×13. The feature maps are then flattened into a vector of 676 values and passed through a single fully connected layer that maps directly to 10 output classes for each digit.

Given below is a diagram that shows the architecture of the model:

The Q16.16 Backbone

Training a model in PyTorch is great, but how do you actually transfer the weights to hardware? Unlike software, FPGA does not understand floating-point numbers. There is no native decimal point at the circuit level as everything is bits, and the hardware has no idea how to interpret them unless you specify.

The answer is fixed-point arithmetic and quantization, where you basically fix the position of the binary point at design time and agree on it everywhere. We use Q16.16 method, where every number is a 32-bit signed integer where the upper 16 bits represent the integer part (including sign) and the lower 16 bits represent the fractional part.

Converting between floating-point and Q16.16 is straightforward: a float is mapped into Q16.16 by computing:

And the reverse is obtained by dividing by the same scaling factor:

The smallest representable step in this format is:

which provides fine grained precision for most computations. Arithmetic in Q16.16 follows simple integer rules. Addition and subtraction are just standard 32-bit integer operations. Multiplication is performed exactly in 64 bits, then rescaled back to Q16.16 with a right shift by 16:

And division is implemented by scaling the numerator before integer division:

Overall, the representable range is:

which is wide enough for typical neural network inference while keeping the math exact, deterministic, and efficient to execute on hardware.

Talos went through a series of architecture evolutions. Working with low-level digital design and FPGAs isn't just about getting the math right. It is about getting the math right within the hard physical constraints of the DE1-SoC. The FPGA has a fixed number of logic array bits, a fixed amount of memory, and a fixed routing fabric. You can't negotiate with it. Every architectural tweak was forced by those limits.

The First Attempt: Brute Force

Our first attempt was a not-so-genius brute force, running all four cnn and maxpool instances, one for each kernel, simultaneously in parallel. Logically, this is the fastest possible approach. In practice, however, it blew up the DE1-SoC, consuming nearly 4× the available LABs on the chip, making the design too big for the fitter to physically route it. We also initially had 10 instances of a neuron module with a massive port connecting directly to the maxpool outputs. The sheer width of that bus created severe routing congestion, and Quartus threw fitter errors before we even got to timing analysis. The design was simply too big to put on a chip as small as Cyclone V.

The beauty of constraints is that it forces you to think. Think about why something doesn't work and if the approach itself is wrong. In software, you can often brute force your way through and worry about optimizing later. In hardware, however, if it doesn't fit, it doesn't ship.

The Pivot: Time vs Memory

Hardware forces you to choose: it's either insanely fast or takes a whole lot of circuitry. The tradeoff between speed and area is worth noting. If it doesn't fit on the chip, no matter how fast it is, it's useless. Keeping overall memory footprint in mind while squeezing every cycle out at the module level, we decided to use a time-multiplexing architecture. Instead of four parallel instances, we used only one cnn module and one maxpool module, and ran them consecutively four times, one for each kernel. This is the architecture Talos ships with.

This is handled by a finite state machine in the inference module that cycles through the following states:

It starts by setting clear_accum to high in S_CLEAR to reset all 10 neuron accumulators. Then for each pass, the state first changes to S_CNN that sets cnn_en to high, which starts the cnn module and runs the convolution with the kernel selected by ker_sel. Once cnn_complete becomes high, indicating that all kernel operations have been completed, the state moves to S_POOL, where it sets mp_en high, and runs the maxpool module for that pass. After mp_complete goes high, it hits S_GAP, increments ker_sel to go to next kernel, resets the internal buses, and loops back to S_CNN for the next kernel. The neuron accumulators are never cleared between passes and thus, they keep accumulating across all four runs, which is exactly how the weighted sum across 676 inputs of the fully connected layer is supposed to work. After ker_sel hits 3, indicating all neurons have been completed, and the final pass completes, the state goes S_DONE and sets complete to high.

This approach alone allowed us to reduce the LAB (Logic Array Block) footprint to almost half of the initial design, showing that we were indeed moving on the right path.

Click to enlarge

Fusing Maxpool and the Fully Connected Layer

With the time-multiplexing architecture in place, we had one more key module to consider: how do the pooled results go between maxpool and the fully connected layer? The naive answer is to store them and feed the FC layer once all four passes are done. While we did do exactly this, the design was still too big to fit on the Cyclone V.

In the end, it all came down to the math. Logically, the fully connected layer is just multiply-accumulate and there's no reason to hold onto the activations at all. So we sort of cheated our way (it worked!) and fused the two modules together. The moment maxpool computes a pooled value, it immediately multiplies it against all 10 neuron weights and accumulates directly into the neuron registers. No bus, no extra device resource usage.

Cutting Resource Usage Further: Weights in ROM

Even with a single CNN and maxpool instance, the fully connected layer weights were still a problem. While the design was able to synthesize within the LAB limit of the DE1-SoC, the fitter was unable to perform optimized routing due to design congestion. Storing all 676 weights per neuron as port arrays meant Quartus had to synthesize them into distributed logic, creating massive routing overhead and pushing device utilization a bit too high.

The fix was relatively simple: move every neuron's weights into M10K ROM blocks using Altera's altsyncram IP, initialized from .mif files at synthesis time. Each of the 10 neurons gets its own ROM, all sharing a single address bus. This single change dropped overall on-device resource utilization to roughly a third of what it was, turning a design that couldn't fit into one that routed cleanly with timing to spare.

Priming the Pipeline

One of the subtle complexities of hardware design is latency management. When our control logic requests a weight from the on-chip ROM, the data doesn't appear instantly. There is a one-cycle delay.

If the arithmetic unit tries to use the data immediately, it will calculate garbage. To solve this, we implemented a "priming" mechanism. The state machine issues the read address, waits (primes) for one cycle to allow the ROM to access the data, and only then enables the multiply-accumulate unit. This ensures that the math is always performed on valid data.

We didn't build Talos just to make another accelerator. We built it to prove that we could. To see if we could take the abstract mathematics of deep learning and forge it directly into silicon, bypassing the layers of software bloat that have become the industry standard.

It was one hell of a ride. We went to the depths of hell and back to find the answer. There were days where nothing made sense, where the simulation was perfect but the hardware refused to cooperate. We stopped counting the hours spent staring at waveform traces, hunting for that one signal - that single misplaced clock cycle or uninitialized register - that was breaking the entire engine.

Hardware is unforgiving. It doesn't care about your intentions; it only cares about physics and timing. But that's also why it's rewarding. When you finally find that bug, when you fix that race condition and watch the pipeline flow perfectly for the first time, it's a feeling you can't get from high-level programming.

Talos stands as a testament to that struggle. It is the result of obsession, of refusing to compromise on performance, and of the sheer stubbornness required to build something from scratch. We hope seeing inside this engine inspires you to take on your own impossible problems.