Share This:

In my last blog entry here and my guest blog over at Kiamesha, I went into what we had achieved with our DC consolidation and redesign, and I started to go into whats next.


At one level, we were happy the design lasted as long as it did, with as few in-flight changes as it all needed. But looked at another way there was a minor disappointment too, and that was that the design lasted so long.


Say what?


One would think, given Moore's Observation (It is not a law) that we should have seen some mid-life turns of the crank in the technology base. There were of course, but nothing that was financially compelling, and certainly nothing that would have saved us enough power to have made a huge difference.


Long Lasting Blades


I mentioned back in the Go Big to Get Small series that the blade we are using for X86/AMD64 virtualization is the m620. When viewed through the lens of capital investment, you have to love that over two years in, the m620 config never really changed. Nothing that came out in that products lifetime made us want to move to any config other than the one we started out with. All the capacity measurements just kept saying that, for our particular use case, that config was the best price / performer.


Recently Dell came out with the m630 blade, and it contains the Intel Intel® Xeon® processor E5-2600 v3 series of processors. The m620 used the Intel® Xeon® processor E5-2600 and E5-2600 v2 product lines. Is that change enough to warrant the move to as new config? We'll soon see. In the meantime though it got me to thinking about the evolution of CPU architecture and how it relates to the greening of the data center.


You Have to Measure


Most every time you see anyone making assertions about something being better than another thing, you see a caveat added. Like "Which smartphone is right for you? Well is depends on how you use it...".


What I am going to talk about here comes with that caution in bold face and italicized font. How a CPU is used really really depends.


Lets take the example of the well known VMware scheduler issue. If you are not a VMware shop or you are running newer versions that have this fixed, you won't have this problem. There was a time however that VMware dispatched guest vCPU's all at the same time. If the guest had 4 vCPU's then VMWare had to find 4 real CPU's on the server free, at the same time, in order to release the guest machine to run. The more vCPU's a guest had, the longer it tended to stay in CPU wait state. This in turn meant that the more real CPU's available to the ESX host OS the more quickly it was likely that the CPU's would become available.


Unless you were CPU bound. Or it was Tuesday. Without measuring, you just don't know.


Our workload, on the m620, tends to run out of virtual memory before it runs out of real CPU. We have seen it in capacity measurement after measurement, and for years on end, using BCO, and all the tools that came before it. It predated the m620 by at least two generations if not more. Our ratio is 16GB of RAM to one CPU (be that a core or a socket running one core). We have seen it so long we have started to wonder if it is not a law of virtual nature. Like Moore's, it is just an Observation.


If you are running KVM, or Xen or some other virtualization solution, the numbers will be different. If the workload you virtualized tended to run at 30% average utilization rather than 5%, your numbers won't be like mine. If you aren't virtualized, but instead run mathematical models all day long every day..


I usually dislike the mealy mouthed "which is better" comparisons that lead you to the answer "Depends on you", so I'm sorry I had to write one. On the other hand, if I had asserted this 16-to-1 RAM-to-Core ratio was a law of virtual nature, everyone would have had the right to climb through their Mac and pound my head, so at least I staved that off.


Green Perfect World


If you look at the CPU spec sheets I linked above, the watts / thermal performance (Intel calls it TDP) of the real CPU's of all three series ran in the range of 80 to 150 watts per socket. That makes sense. They have to be installed in the same systems with the same airflow already engineered. Put out a 200 watt part all the sudden and no one is going to be happy with you.


Two generations back for the e5-2600 it was 4, 6 or 8 cores though. V2 (1 generation ago) added 10 and 12 cores. V3 added 14, 16, and 18 cores. per socket, with other bits like the cache getting larger as well.


Clearly I am getting more and more cores for the same(ish) power. That's not counting anything that may have been done to increase per-core efficiency too, like increased cache, better / bigger / faster look aside buffers, better predictive pipelining, better out of order instruction execution, instructions per cycle, and all that (did I mention you have to measure?).


If I can more or less run "16 core per socket, double my RAM per blade" kind of config, and not have all that double or more the price of the unit, then I can use half the DC real estate to get the same amount of work done. I can hold down the number number of NEW chassis I need to add, and save that 'overhead' power too. Plus the costs of the internal-to-the-chassis switches on the back.


Same logic says that if doubling is too expensive, what about 50% increase? 12 cores and 386GB per blade? As long as that bump is less that 50% more expensive (and every other variable remains the same.. a big if, I know) then why wouldn't I do that?


Standards are good. New standards that save power are, to be very technical, "better". And in some ways: What took so long? Shouldn't I have been able to do this all more than 6 months ago?


No: Because price / performance ISN'T ramping down as fast any more.


That logic applies beyond AMD64 / X86 space too. Same for Sparc or Power. It will be true for ARM, when its data center day arrives. I'd say same for Itanium, but that ones getting ready to exit stage left.


Stay Tiny My Friends


If you have been doing Data Center Consolidation for a while, and particularly if you have been retiring older workloads into the virtual world, this is the next frontier. The servers are starting to get on up to their 3 year depreciation schedule sell-by dates (our oldest ones are 2 years old in this most recent project).


Re-hosting them to newer, more Core-dense servers is the next win, though lets face it: Its not like the last one. I was going after 10-to-1 reductions then. Now I am getting jazzed about 50% reduction possibilities.


Its not nothing though. When viewed on a watts per VM point of view, or from the perspective about how to stay in my new, tiny data center, its what has to come next. When viewed for the perspective about how we all reduce the amount of power our data centers will be using globally over the next decades, it something that remains front and center in the design of the Green DC solution.


Remember when a server had one core? Those were the days...