With their low latency and flexibility, FPGAs are ideal for AI applications, particularly at the edge. Richard Warrilow attended a recent FPGA Front Runner event to discover why we need to be designing at a higher level of abstraction.
To date, AI has been dominated by GPUs. They are good at parallel processing tasks and handling large AI/ML models, and particularly good for model training. However, they are power hungry and the time it can take to launch a kernel makes them impractical for use in AI/ML edge applications requiring low latency.
FPGAs on the other hand are flexible and boast low latency, and in the battle of GPU vs. FPGA even NVIDIA concedes that to achieve a latency of less than 10microseconds, FPGA is the clear winner.
Giles Peckham, Head of Marketing at Myrtle.ai, gave an interesting example: “In the Conversational AI market there’s a lot of focus on hitting a speech-to-speech time of just a few 100ms. So, speech to text, then the use of a large language model to form a text reply, and finally text to speech.”
Trade-offs
There are many challenges associated with FPGA development at the best of times. AI/ML adds a significant layer of complexity. The model must be optimised against many priorities, including latency, throughput, energy consumption and cost. Optimisation requires the combined skills of hardware, software and AI/ML engineers, but few companies have all three fields of expertise in-house, and those that do cannot always find an optimal way of working.
For instance, the AI/ML team will use a framework like PyTorch or TensorFlow to build and train models on a 32-bit floating point (FP32) GPU. They will then want to evaluate its performance on an FPGA. This requires the hardware team doing their implementation, mapping and other tasks just to get back to the AI/ML team with real-world performance figures.
“However, the AI team will want to reiterate and re-evaluate the model several times without having to go back to the FPGA team each time,” commented Peckham, as a segway into introducing VOLLO, Myrtle.ai’s low-latency ML inference accelerator. It allows the AI team to create their models and immediately do cycle-level, bit-accurate simulations without the need for target hardware. VOLLO is fast too, achieving latencies as low as 5.08µs, with a throughput over 800k inferences/second, in a Strategic Technology Analysis Center (STAC) benchmark test.
FPGAs Help Develop New Processor
FPGAs are of course great for the development and prototyping of IP, and that is just what recent start-up Red Semiconductor is doing. The company is of the opinion that traditional processors do not have the most suitable or secure architecture for AI, particularly for edge processing (see figure 1).
Figure 1 – AI-enabled edge processing is the building of a situational awareness picture of what is happening and then making real-time decisions that affect that situation. (Source: Red Semiconductor)
James Lewis, CEO of Red Semiconductor, explained: “As an industry, we’re creating these complex infrastructures and ecosystems where large volumes of data must be processed in real time. Disseminating the valuable information computed from raw data is essential, but presents security, power and cost challenges.
Red Semiconductor has developed what it calls a Versatile Intrinsic Structured Computing (VISC) architecture, which allows AI developers to work at a higher level of algorithmic abstraction. Code for problems, such as a neural network, can of course be handcrafted to optimise performance, but VISC supports a highly abstracted level of mathematical or algorithmic expression and compiling to a few special instructions as opposed to hundreds or thousands of standard instructions.
Abstracted VISC Instructions (AVIs) are optimised for the solution that is being expressed and invoke a selection of hardware techniques (designed by Red Semiconductor) that involve very efficient vector processing and a process re-ordering architecture: turning a simple in-order processor into a context-modified, out-of-order system. VISC has a single pipeline for what the company calls single-issue / multiple execute (SiMex) tasks, which have already been put to good use.
Specifically, Red Semiconductor is working with a partner on an AI-enabled, medical diagnostics and therapeutics technology application. It is an analytical system for clinical sample analysis where complex pattern anomalies as low as one-in-a-billion must be identified and labelled in real time. Parallel processing on a massive scale for signal processing, error checking and correction (ECC), and pattern matching is essential, and VISC’s SiMex is a key enabler.
“Another beneficial feature of VISC’s architecture is that it slashes instruction memory accesses and uses a large register bank inside the chip to eliminate load/stores during cryptography computation,” continued Lewis. “This effectively removes a cyberattack surface, i.e. the memory bus, whereby hackers observe bus activity in an attempt crack ciphers.”
VISC will initially be available as IP as a frontend for the open-source RISC-V architecture, of which Red Semiconductor has made its own implementation. The company has taken the VISC architecture thorough its simulation, functional verification and prototyping (using FPGAs) phases.
In summary
Myrtle.ai and Red Semiconductor have developed different ways of providing a higher level of abstraction when developing AI/ML, and the latter’s new processor architecture is definitely one to watch. Both companies are getting amazing results too, and though Myrtle.ai targets FPGAs and Red Semiconductor uses them for product development, the suitability of FPGAs for AI is resoundingly clear.
Note: this article is largely based on material presented by three companies that presented at a FPGA Front Runner event hosted by Renishaw in February 2025. The next event is ‘Prototyping Systems Using FPGA’ and will be hosted by Rolls-Royce on 21st May 2025. It is free to attend, and you can register though LinkedIn: https://lnkd.in/eUXNd_WE.
Reproduced from the kind permission of New Electronics