Rethinking the AI Chip: How HyperAccel’s LPU Tackles the LLM Inference Bottleneck

GPUs were never designed for AI. They were built to render graphics for video games and 3D applications, and it just so happened that their massively parallel architecture was a great fit for training neural networks too.

Training and inference are different workloads, though. And as more companies look to run large language models themselves, the question of whether a GPU is really the best tool for inference is becoming harder to ignore.

On a recent visit to Seoul, ipXchange stopped by the offices of HyperAccel, an AI chip design company building processors specifically for LLM inference. We spoke with Joo-Young Kim about what’s holding GPUs back, how the company’s LLM Processing Unit (LPU) works, and where the technology is heading.

The real bottleneck is memory

LLM inference has a very particular workload. There is a pre-fill stage, where the model processes all of the input tokens in parallel, followed by a decode stage, where it generates the output one token at a time.

The catch is that every one of those tokens requires the chip to read the entire set of model parameters from memory again. That means the speed at which data can move from memory to the processor matters far more than raw compute.

GPUs use high-bandwidth memory, which is among the fastest DRAM available. But between that memory and the thousands of small cores doing the work sits a complex memory hierarchy, and Kim explained that this creates a system-level bottleneck. In practice, a GPU often only uses around 50 to 60% of the memory bandwidth available to it during inference.

It’s a problem we hear about regularly at ipXchange. Plenty of the companies we speak to point to the memory wall, rather than compute, as the thing slowing AI down.

A big-core approach

HyperAccel’s answer was to redesign the processor from the ground up around the transformer.

Rather than an array of thousands of small cores, the LPU uses a much larger core capable of accelerating every operation in transformer inference, without disrupting the flow of data from the memory interface. Each of these big cores can generate a token by itself.

At the chip level, the company has developed what it calls a Streamlined Dataflow Architecture, which simplifies the path between the memory channels and the cores. Because each core can complete the job on its own, data only needs to flow across the chip once instead of moving back and forth. According to HyperAccel, this allows the LPU to use up to 90% of the available memory bandwidth.

For anyone who wants to see the difference visually, HyperAccel’s website includes an animation comparing the GPU and LPU architectures.

Seeing it in action

The LPU is delivered on the Bertha PCIe card, which can slot into a desktop as well as a server. HyperAccel’s main focus is the server side, with up to four cards fitting into a 2U server and up to eight in a 4U server.

During our visit, Jongjin from the company’s performance modelling team showed us a single card running an 8-billion-parameter Qwen model, quantised to INT4. We asked it to explain why the sky is blue, and the answer came back very quickly.

Speed isn’t the main pitch, though. HyperAccel is aiming for better throughput per dollar and per watt. The card uses LPDDR memory, which is more power efficient and gives it a larger memory capacity. That larger capacity means longer context windows, and it can also mean fewer devices are needed to run the same model compared with a GPU setup.

Because the LPU accelerates transformer inference, it can run essentially any transformer-based model, which covers most of what you’d find on Hugging Face today, including vision language models and physical AI models. The software stack is also compatible with open-source inference engines such as vLLM and SGLang.

Built for enterprise AI

HyperAccel isn’t chasing the hyperscalers running frontier models in 100-megawatt data centres. Its focus is enterprise AI infrastructure.

These are companies that want to run their own LLMs on their own hardware, often for security, data privacy or cost reasons. Kim gave the example of a semiconductor company that wants to train an expert model on confidential information it can’t send to the cloud. Workloads like this typically involve medium-sized models of around 100 billion parameters, with fewer than 100 people using the server at once.

Running that kind of workload on GPUs can be expensive. HyperAccel says its device costs several times less than a Hopper-class GPU, which could significantly reduce the capital cost of building out in-house AI infrastructure.

How to evaluate it

For engineers who want to find out whether the LPU suits their own models, HyperAccel offers three routes in.

The first is a performance simulator, which lets you estimate how your models would run on the LPU architecture and explore different workloads and configurations before touching any hardware.

The second is the HyperAccel software development kit. Developers can port their own models using high-level, Python-based languages such as OpenAI Triton, and the SDK converts them automatically, so there’s no LPU-specific language to learn.

Finally, customers can access HyperAccel’s remote servers to run their own models on real LPU hardware and see the actual performance for themselves.

Tokens per dollar, and what comes next

Most conversations about AI performance focus on how many tokens a system can generate per second. Kim believes the more important metric is efficiency: how many tokens you can generate per watt and per dollar. That, ultimately, is what will drive the cost of AI services, and bringing it down is what could make AI more sustainable and accessible to far more people.

Looking ahead, HyperAccel expects the next big leap to come from memory rather than logic. Emerging technologies such as 3D-stacked memory promise even more bandwidth than current HBM, and the LPU architecture is designed to take full advantage of it.

The company is also bringing the same architecture to the edge. Working with LG Electronics, HyperAccel has shrunk the LPU to around a tenth of its size to fit within an edge power budget of around 15 W. The chip has already taped out and is expected early next year.

Watch our full visit to HyperAccel to see the LPU running live and hear Joo-Young Kim explain why the future of AI inference might not run on a GPU at all.

Learn more at https://hyperaccel.ai/