No, really. They are. Allow me to elucidate.
The CPU
Memory hierarchy is the arrangement of the physical memory on a computer’s motherboard.
We are mostly concerned with the upper 3 levels on that pyramid: RAM, Processor Caches, and Processor Registers. The CPU itself contains a Control Unit (CU), an Arithmetic Logic Unit (ALU), and those caches and registers. The CU orchestrates the flow of data into registers which the ALU operates on. The CU then directs that transformed data back into memory as needed. There’s more to everything the CU does, but this is a sufficient high-level view for this post.
In the CPU’s kitchen, the CU is the Chef de Cuisine and their Sous-chefs de Cuisine. It’s their job to make sure everyone is doing what they’re supposed to be doing at the appropriate tempo in order to keep the kitchen flowing. They’re the ones making sure the food is well-prepared and plated correctly before sending the runners to deliver the food to the customers. They’re the ones that order the food that comes in from external warehouses (SSD). When a ticket (CPU instruction) comes in from the front of the house (FoH) they’re the ones that read it out and make sure the rest of the brigade are all working on the same order.
If the CU is the head chef, then the ALU are the actual Cuisiniers or Chefs de Partie. Where the head and sous chefs are worried about managing the operation of the kitchen, the chefs de partie or line cooks do the actual cooking. The line cooks love to just cook and the ALU likewise wants to just operate on data as fast as it can. Their stations are where they do the actual food preparation such as on a flat top grill, a stove top or even a cutting board. In our CPU this is analogous to the Processor Registers: physical locations where the CPU operates on the data.
Every extra cook in the kitchen can be compared to having another ALU in your CPU. Modern CPUs can have 4-8 per core, so for the sake of this metaphor let’s assume we have just one core to work with. The CU and ALU comprise the very top of the pyramid in the Computer Memory Hierarchy chart.
Memory Caches
A chef can’t cook without food, and likewise a CPU can’t work without data. CPUs have various options for storing this data until it is needed. The first option is the L1 Cache, which is usually integrated directly into the CPU chip itself. Most modern L1 caches are about 64 KB in size. That might seem like a small amount of data, but actually it allows the CPU to work on that data at lightning speed. The L1 cache is physically located adjacent to the processor registers in the CPU. In our kitchen analogy this is equivalent to a countertop next to a work station. The chef can pull out whatever ingredients they need for cooking the current order that the kitchen is working on. The countertop can only hold so much food at a time, but this is actually a benefit as the chef doesn’t have to spend time hunting down the ingredients they need as they are all sitting right there in arm’s reach.
So where does the chef pull the ingredients out from when they need to cook? It would be a tremendous waste of time if the chef had to go to the walk-in fridge every time they needed to grab some onions. They instead store the food in a “lowboy”: a small reach-in fridge that sits under a counter or workstation. This is usually filled with prepped food that the chef uses when working. These lowboys are generally much smaller than a standing reach-in fridge or even the walk-in refrigerator downstairs, but that’s part of the secret to their usefulness. During a busy service, the chef can just reach down and grab what they need. The small size allows the chef to quickly search through the lowboy, locate the food for the order, and begin cooking it almost instantly. In a CPU this is the L2 cache. It is physically in the CPU, but is slightly farther away from the registers than the L1 cache is.
Modern CPUs also have an L3 cache. This cache provides data to several CPU cores and thus acts more like a large fridge that’s accessible via multiple kitchens. An example is that of a large hospital with multiple kitchens. If the kitchens are physically located next to each other they can share a large array of reach-in fridges that hold food that the kitchens need. Since we’re only focusing on one core (or kitchen), L3 won’t be important for this metaphor.
Random Access Memory
A kitchen can’t store everything it needs to operate in small lowboys under the work stations. Nor can they fill the line up with reach-in side-by-side fridges either, as the chefs would then be spending all their time searching for the next ingredient they need instead of staying at their station and just cooking. What’s the solution? We centralize all the ingredients the kitchen could use in one location: the walk-in fridge downstairs. Now when the trucks arrive from the warehouse (SSDs and other long-term, powered-off storage media), the workers can just unload all of the food into the walk-in until it’s needed by the chefs upstairs.
But unlike the lowboys and countertops on the line, the cooks can’t easily access the food. They have to either go down there themselves or send a kitchen assistant or commis chef to fetch it for them. Either way the cook isn’t cooking anymore. Furthermore, the walk-in, even if organized well, requires searching through a much larger volume of food to find exactly what the cook needs. On a CPU this travel and search time also exists. RAM is much larger than all the caches (measured in GB rather than KB and MB), but is further away from where the work is actually done.
The Kitchen is a CPU
The kitchen is a production pipeline that has to create a finished product just in time or else the entire operation grinds to a halt. This is bad for customers, and bad for the kitchen. Organizing the food across several locations to ensure that the cooks have exactly the ingredients they need at the time they need it is the rallying cry of data-oriented design. Keeping your data organized is just as important as efficiently operating on that data. If you treat your CPU like an organized kitchen in your program, you are guaranteed speedy performance when you need it. The next article will be expanding on this concept to show just how the kitchen…errr….CPU operates on data in both an organized and unorganized kitchen.
