Is FCoE faster than Fibre Channel? Who knows? Just run your own tests

Cisco camp takes exception to Brocade-funded Evaluator Group study

The Cisco Fibre Channel over Ethernet (FCoE) supporters' club has taken grave exception to what it views as a deeply flawed Evaluator Group study, funded by Fibre Channel (FC) enthusiasts Brocade, which showed FC was faster than FCoE.

FCoE supporters say the study shows no such thing and lambasted both the Evaluator Group and the "churnalists," as they term them, who reported it without pointing out the flaws as they perceive them, in the study.

A Twitter storm followed the reporting of the report. One result was a blog by Cisco's J Metz, Strategic Product Manager, Storage and Unified Fabric at Cisco Systems, in which he tore the study to pieces, talking about "the ugly spectre of blatant dishonesty that exposed a disturbing trend of vendor-media collusion that I feel harms people."

Metz also laid into trade press for not evaluating the study properly: "[The] trade press is supposed to read and evaluate these studies for merit and summarise to a more general audience. They are also supposed to be more educated about the topic than their readers (after all, why bother writing about it otherwise?) and provide a sense of context that may offset the inherent bias which naturally accompanies the study."

Should this trade press person defend his reporting? (Why not? - Ed.) I doubt very much whether I'm more educated on the general topic of FC and FCoE than many of my readers. I just get to hear stuff sooner and report it, hopefully in a neutral enough way and, sometimes, with snarky hints about the provenance of things.

Anyway, back to Metz' criticism; "If you ever want a prime example of how not to do a research project, this is Exhibit A."

His main points are:

• The study uses the phrase “Gen 5, 16Gbps FC switch.” Gen 5 is Brocade’s marketing term for their latest platforms; it is not an industry standard term, despite its recent adoption by QLogic and Emulex
• Anti-Cisco bias in text: "Actual latencies would likely be higher for an all-Cisco deployment using FCoE… connected to Cisco MDS switch, than the tested environment.” But MDS' switch wasn't used in the testing so why mention it?
• Use of Brocade marketing brochure text
• Inexpert Evaluator Group testers: "The setup of the FCoE environment required approximately 8 hours of time. Several issues were encountered while configuring the UCS equipment. Additional assistance was requested from a VAR with certified UCS engineers.”
• Flawed test set up with FCoE using UCS servers and FC HP servers. Why vary the servers?
• Also FC was configured end-to-end, whereas two FCoE 10gigE came from the servers to a Cisco switch and then uplinked to a Brocade FC switch via two 8Gbit/s FC lines, that could not be aggregated, to make a logical 16Gbit/s FC link travelling to target arrays using FC. Why was this throttling done? Why not end-to-end FCoE?
• The test design had a 10gigE link get converted to an 8Gbit/s FC link which then went into a 16GBNit/s link. Metz says: "Whenever you put information on a wire it needs to be encoded, and all three versions have different encoding schemes, which means that from a throughput perspective, you have a massive bottleneck smack dab in the middle."

Metz considers that most of the reported under-performance by FCoE could be accounted for by this bottleneck. There's more; read the blog to see it.

Metz is seriously displeased about the study. "The real concern I have is that if something this bad, this egregious, isn’t called out by the so-called “watchdogs of technology,” what hope do the non-technical and the uninitiated have to find out the truth?"

He continues: "All in all, this study reeks of desperation on Brocade’s part, that they are willing to throw caution (and logic) to the wind, paying for – and publishing – some of the worst testing I’ve ever seen in my life, and got other media firms to promote it."

Russ Fellows of the Evaluator Group emailed The Register and said a Q and A about the study was available (here). It sets the scene for the testing.

"As an independent analyst firm,” says the Q&A, “Evaluator Group was engaged by Brocade to conduct a comparison of two environments, a Cisco UCS with FCoE server connections and an HP BladeSystem with Fibre Channel server connections, both connected to Fibre Channel storage."

Then there was this statement:

The objective of the test was to compare the impact of deploying the latest generation Flash storage on high performance enterprise applications in virtual server environments. A 16 Gb Fibre Channel flash storage array was selected as the storage target, and HP c7000 and Cisco UCS were selected for the blade servers given their market share leadership.

Based on the most common deployment models with shared storage for each solution, UCS servers used FCoE to connect to a Fibre Channel SAN while end-to-end Fibre Channel connectivity was used on the HP c7000, with both connecting to the same solid state Fibre Channel storage array. The objective was to understand how the two different architectures would impact the application benchmarks using the same storage.

The doc says: "The configurations were carefully chosen to be as fair as possible; ensuring storage network limitations were equal for both configurations."

Metz would, judging by his blog statements above, vehemently disagree with this statement.

It's a fairly short Q and A and, reading between the lines, you might think the Evaluator Group wished it had not gotten into a shitstorm over this study.

Which leaves us where?

If you are evaluating a choice of Fibre Channel or FCoE do not rely on vendor-funded studies. Run your own pilot test and draw your conclusions from that. ®