Tech

Meet the DPU: The Hidden Processor Behind Nvidia's AI Engines

Data systems, illustration
Data systems, illustration | Photo: Shutterstock

We are all familiar with the CPU and GPU, but the third and relatively unknown pillar is no less important — the DPU.

The picture of Mitzi the cat that you sent to Grandma on WhatsApp is not sent as a picture; there is actually no such thing as sending a picture over the internet. What you send is an enormous stream of very small data packets, each packaged in a virtual capsule that explains what it contains.

These packets reach the destination device, where the picture is reconstructed piece by piece using the network card and processor.

This works well when it comes to your phone, but when it comes to data centers that need to process petabytes of data in short periods of time, the processor burns tens of percentage points of its processing capacity just handling the assembly of the data, while the expensive, fast GPUs sit idle waiting to receive new data from the processor.

Data systems, illustration | Photo: Shutterstock

The DPU is a dedicated processor designed to solve this bottleneck. It contains ASIC processors whose software is hardwired at the silicon level, and they perform predefined operations over and over without the need for software.

The rigid structure of their algorithms allows them to handle enormous amounts of data continuously, quickly and with low power consumption. The latest generation of DPU processors from Nvidia also functions as a network card, and it can process data at an astounding rate of 800 gigabits per second. Data can also be transferred through it in server farms between different graphics cards, without passing through the processor at all.

The DPU also contains a large number of CPU cores running Linux, which handle security and the chip's overall management.

Amdahl's law states that the speed of a parallel processing system is limited by the part of the system that continues to process data serially and becomes a bottleneck. Data processing by the processor is a serial task, so using a DPU is what unlocks the full potential of the CPU and GPU.

Nvidia
Nvidia | Photo: Shutterstock

Although Nvidia is known primarily for its revenue from GPU sales, the DPU plays an important role in its business strategy. Customers that deploy DPU cards and build their cloud systems around them enjoy a significant improvement in output and revenue, but make any future switch from Nvidia to another competitor much more complex. 

Key takeaways

  • A DPU is presented as a third processing pillar alongside the CPU and GPU, designed to remove a data-handling bottleneck in data centers.
  • Instead of relying on the main processor to assemble incoming data, a DPU handles packet processing so GPUs do not sit idle waiting for data.
  • Nvidia’s latest DPU generation can also function as a network card and is described as processing data at 800 gigabits per second.
  • The article says DPU cards can move data between graphics cards in server farms without passing through the processor.
  • The piece argues that by shifting serial work away from the CPU, DPUs help unlock the full potential of CPU and GPU systems.

Questions and answers

What is a DPU and why does it matter?
A DPU is a dedicated processor built to handle data movement and related tasks that would otherwise consume CPU time. The article says this helps solve a bottleneck in data centers and lets GPUs work more efficiently.
How is Nvidia using the DPU?
The article says Nvidia’s latest DPU generation also works as a network card and can process data at 800 gigabits per second. It is also used to move data between graphics cards without passing through the processor.
Why does the article say a DPU improves system performance?
Because data processing by the processor is described as a serial task that creates a bottleneck in parallel systems. By moving that work to the DPU, the CPU and GPU can operate closer to their full potential.

Share this story

About the writer

Shlomo Strauss

Writes for TV10, Israel’s economic channel. This story was translated from the original Hebrew.

More stories by Shlomo Strauss

More for you