I recently read through a performance report created by a firm for one of their clients. My first thought is that it is good to see more people using actual lab testing and data rather than theoretical arguments as to why a vendor’s architecture is well suited to a specific workload. My thinking has always been, “If the system runs a workload well, why not prove it with some real data points?”
My observations and comments are generalized. I have no intention of naming the analyst or vendor report, since critiquing a specific report is not my intention. Rather, I hope to shed a bit more light on how (and how not) to run workloads. By arming IT consumers with more knowledge, I hope that ultimately the recommendations get back to vendors so they stop asking for and using the type of pseudo workload generation that leads people astray.
In one particular test report I read, a firm had used the “Iometer” tool to generate different workloads. This is in fact fair and valid. However, they then went on to create several theoretical workloads that they equated with real-world workloads. This is where people go wrong when using tools like Iometer.
Now I talk about Iometer as a bit of a euphemism; that is, there is nothing particularly wrong with Iometer. Rather, it is representative of the poor testing processes I have seen in general and that can occur with any tool. It’s a bit like a loaded 9mm in a candy store. Anyone can pick it up and give it a try, but typically the results are not pretty.
How Not to Create Application Workloads
In this report, several pseudo application workloads were created. One workload was termed a “Virtual Machine Workload.” This was an Iometer profile, with 50% read and 50% writes, with 50% of the I/O’s being random and the other 50% being sequential, all at a 64 KB block size.
This workload can be a valid test. However, what is not fair, valid or even reasonable is to then call this test a measure of “VM performance.” The workload itself is far too static to come close to representing a single application, much less the diverse set of applications typically running together in a virtual environment. Not to mention duplication rates, compression, etc. etc.
The report continued to create workloads using Iometer profiles that were then equated with actual applications. The next workload was 100% random, with 50% reads and writes and also at 64 KB, was specified as a “VDI Boot Storm.”
First, I have measured both a complex set of virtual machine workloads and multiple VDI workloads. The real data is not even close to the parameters used here with a claim to be application workloads. Second, the dynamic nature of true workloads is not at all captured by a simplistic Iometer profile. That is, I/O rates and block sizes are not constant. Still another issue is the fact that Iometer “random” I/O operations may be too random, but may not be random enough.
By “too random” I mean that real workloads tend to come back to the same address ranges, but may do so 2, 5 or even 10 minutes later. To the life of an I/O this is an eternity and is deemed “random.” However, good storage designs are able to determine what data to cache, and have a good chance of having the correct address range in cache. Therefore, a well-architected storage system will have a higher than “random” probability that the data is in cache, and therefore random I/O’s are not realistic.
On the other side, you can simulate a “cache hit” by coming back to the same address space with your workload tool. However, Iometer and other tools typically use too small of an address region, thus over-stating the cache hit rate. So in both cases, the cache hit rate generated is incorrect; it can be either too high or too low.
The next question I get asked is typically this: “If Iometer isn’t the right tool to use, what should I use?”
OK, now we’re getting someplace. If you are to the point where you understand that Iometer results claiming to represent an actual workload are invalid, you are now ready to go to the next step.
My Recommendations
If you are an IT user and want to understand how a system handles an application workload, and you read a report that uses Iometer or another tool to generate an artificial workload, keep walking. Or better yet, call the vendor and ask them to show you some real results.
If you are a vendor and you have a company propose to use similar methods for a test report, I would encourage you to ask them to use better tools. There are more accurate ways to create highly accurate workload results. If you are a vendor and are asking for these types of workloads because you ran it in the lab and your box looked fast, please re-read this blog.
Using simplistic Iometer profiles is a fast and easy way to generate a lot of I/O traffic. However, this in no way represents true application I/O’s. I would encourage anyone to use Iometer as they like, but please do not expect the results to be representative of actual applications.
