Addendum – Answers to Questions Regarding Testing

By , Friday, February 7th 2014

Categories: Analyst Blogs


As an independent analyst firm, Evaluator Group was engaged by Brocade to conduct a comparison of two environments, a Cisco UCS with FCoE server connections and an HP BladeSystem with Fibre Channel server connections, both connected to Fibre Channel storage.  We did so with the caveat that it was an independent review.  We constructed the testing based upon discussions with our IT clients (typically Fortune 1000 firms).  The goal was to compare configurations that customers commonly evaluate from two leading blade server vendors to help them make more informed choices.

The objective of the test was to compare the impact of deploying the latest generation Flash storage on high performance enterprise applications in virtual server environments. A 16 Gb Fibre Channel flash storage array was selected as the storage target, and HP c7000 and Cisco UCS were selected for the blade servers given their market share leadership. Based on the most common deployment models with shared storage for each solution, UCS servers used FCoE to connect to a Fibre Channel SAN while end-to-end Fibre Channel connectivity was used on the HP c7000, with both connecting to the same solid state Fibre Channel storage array. The objective was to understand how the two different architectures would impact the application benchmarks using the same storage.

To conduct the test, we performed all of the testing in our lab, independent of vendors.  Support was provided via the vendors support lines, avoiding special assistance or inside information.  While our hypothesis was that end-to-end Fibre Channel would outperform FCoE to Fibre Channel, Brocade understood that outcomes are not known ahead of time.  In some cases, results do not show a vendor or products favorable.  That risk is borne by the vendor funding the work.

In order to ensure that the testing was fair and representative, we used application workloads representative of common and comparable customer environments.  We chose to use a Fibre Channel SAN storage solution, given that Fibre Channel is the leading interconnect for shared storage among enterprise customers.  The configurations were carefully chosen to be as fair as possible; ensuring storage network limitations were equal for both configurations.


1)     What type of FCoE initiators were used for Cisco UCS?  

The Cisco UCS VIC 1240 hardware CNA’s were utilized.  Referring to them as software initiators caused some confusion. The Cisco VIC is a hardware initiator and we configured them with virtual HBAs. Evaluator Group has no knowledge of the internal architecture of the VIC or its driver.  Our commentary of the possible cause for higher CPU utilization is our opinion and further analysis would be required to pinpoint the specific root cause.  While CPU utilization was not the main focus of the test, it is an important issue for customers, so this may be an area for further research in the future.

2)    This test does not seem to be the best way to compare only an FCoE connection to a Fibre Channel connection.

The goal of the test was to compare two typical configurations.  Customers are deploying high speed solid-state systems in virtual environments.  Currently, a majority of our enterprise customers prefer to use a Fibre Channel SAN for their high performance applications.  When comparing alternatives, customers often consider a blade server like the HP BladeSystem, with Fibre Channel interfaces.  Alternatively, they may also consider Cisco UCS blade systems with FCoE interfaces. In both cases, it is common to connect the blade servers to a SAN (in this case a Fibre Channel switch), as opposed to directly connecting to FCoE storage.

3)    What version of UCS software was used?

We utilized UCS GUI manager version 2.0(4b).  The UCS chassis and blades utilized version 2.0 software as well.  Although a more recent version was available, there were no known capabilities in the latest version that would affect the results and our conclusions.

4)    Why was a screenshot of a simulator used in the paper?

An example screenshot published in the report was chosen since the intention was to show a typical representative GUI screenshot of both environments.  All configuration information actually utilized in the analysis is provided in the report.  Versions of UCS software used in this analysis are noted above.

5)    Evaluator Group seems to penalize the UCS for being more difficult to setup, but maybe they just understand the HP system better.

Our objective was to report on our experience configuring both environments.  While Evaluator Group has many years of experience setting up enterprise servers; we did not have prior specific experience setting up either a HP c7000 BladeServer, or a Cisco UCS.  As explained, all testing and setup was performed within our lab environment, without direct assistance from any of the vendors.  When issues arose, we utilized typical support mechanisms, calling into the toll-free support lines of the vendor, without asking for special assistance.   Our opinion is based on the difficulty and amount of time needed to configure each system. Our experience was that it took us longer to configure the Cisco UCS than the HP system, and we reported that observation.

6)    Why not use multiple FCoE connections between the UCS and the 6248UP?

Each UCS blade had one dedicated 10 Gb FCoE link to the 6248UP.  This made for a total of two 8 Gb Fibre Channel links from the UCS into the Fibre Channel SAN.  The reason that more links were not utilized was to maintain fairness in the comparison.  The alternative HP blade configuration used one 16 Gb link that was shared between the two HP blades.  Thus, both configurations had equal amounts of bandwidth at the point of contention.

Forgot your password? Reset it here.