I’d also like to point out the the Dell R900 result you mention above was matched exactly by a later result from Sun using similar hardware and the production release of ESX Server 3.5. I hope such independent confirmation of results would put to rest any concerns that the use of pre-release builds shows any measureable performace variance and is in any way unfair.

Full disclosure for me: I work for VMware and am one of the original developers of VMmark. Prior to that, I worked in a benchmarking team at HP where I published a few results with future availability dates .

My HP account team was passed the link to this post, so if they want to reply here they’re welcome to.

It’s not clear to me what more is reasonable to expect. Since it appears to me that all version information is disclosed, I’ll assume, then, that it is availability dates that concern you. However, in the Dell submission you cited, under “Configuration,” both “Hardware Availability Date” and “Software Availability Date” are listed in Month-Year format. Other availability dates are also discosed, as required. This seems consistent with other industry-standard benchmarks.

I would concede that it might be an easier read for those specifically interested in release dates if all release dates for all components were lumped together in one place in the disclosure, but that would just move the burden of scrolling around the disclosure from the reader who cares most about the release dates to the reader who would prefer release dates for specific components be listed with that component description. So, for example, under such a new arrangment the person who is interested in the specifics of the “Virtualization Platform” would have to go scrolling to another area of the disclosure (the presumed new common release dates area) to find the release date for the virtualization platform used. The only way to avoid this would be duplication of fields in the disclosure, which has its own, obvious drawbacks.

> I believe I’ve identified a reason that HP
> doesn’t actively submit results.

Yes, but what I continue to find unclear is why HP submits results under TPC and SPEC when they are subject to very similar (if not identical) drawbacks as those you noted for VMmark submissions. In other words, I do not believe these industry-standard benchmarks do any better job of addressing HP’s concerns than does VMmark, and yet HP submits in these spaces. (I’m also petty sure they publish in these spaces using pre-release components from time-to-time.)

So if, the next time you speak with HP, they can clarify this seeming inconsistency, I would find that highly enlightening.

I wasn’t aware of the SPEC and TPC solution availability policies you specify, but they don’t alter my concern. While I completely agree that the disclosures are full, VMware could easily implement my second suggestion. That may not alter the state of things, but I think it would help customers.

As a customer, I’d like to see more vendors disclose VMmark results, even if the underlying platforms overlap. I believe I’ve identified a reason that HP doesn’t actively submit results. It may hold for the other vendors involved; HP’s the only one I’ve spoken with. I just wanted to get my thoughts and suggestions on the issue out to see if there might be others with opinions on the subject.