Powering and Cooling the Data Center

Question

FEATURE SECTION: POWERING AND COOLING THE DATA CENTER Data Center Efficiency in the Scalable Enterprise Measuring energy efficiency in the data center Building efficient data centers Enhancing data center efficiency through virtualization Implementing data center efficiency in the scalable enterprise For More Information: Inside the Dell Energy Efficiency Research Center Ongoing thermal and cooling research Energy Smart Dell PowerEdge Servers Related Categories: Computational fluid dynamics (CFD) Data center density Dell PowerEdge servers Power and cooling Visit www.dell.com/powersolutions for the complete category index. Data Center Efficiency in the Scalable Enterprise Energy efficiency is a top concern as IT managers struggle with rising power bills, cooling problems, usage limits imposed by local utilities, or requirements to deploy additional servers without expanding an existing data center or building a new one. To address energy efficiency effectively, data center managers must look at a variety of issues ranging in scope from the smallest piece of silicon to the entire data center. By John Pflueger, Ph.D., and Sharon Hanson Looked at holistically, the data center life cycle offers two overarching opportunities for optimizing energy efficiency. First, enterprises can right-size the data center at the planning and deployment stage. Second, during the operating stage, enterprises can begin to take advantage of emerging trends toward more tightly managed power usage in the data center—from infrastructure to racks of servers to individual servers, and even components within those servers. Analyzing how power is used in a typical data center can be useful in understanding these opportunities. Measuring energy efficiency in the data center A recent Dell study conducted during the summer of 2006 examined power consumption in a typical data center. This study looked at hour-by-hour power consumption at the data center level, then drilled down to the IT equipment and indi-vidual server levels. Figure 1 shows the results of the study. At the data center level, the study showed that for every kilowatt delivered to the data center, 41 percent, or approximately 400 W, was delivered to the IT equipment. The remaining 59 percent was split between power distribution losses (28 percent) and the energy required to run cooling equipment (31 percent). This 41 percent efficiency is not atypical. In fact, a substantial number of data centers run at 30 percent efficiency or less. Next, the study looked at how the 41 percent used by IT equipment was distributed among compute servers, storage servers, and other IT equipment. For this facility, roughly 63 percent of IT power was consumed by compute servers, 22 percent by storage servers, and 15 percent by the rest of the data center IT equipment. The study then looked at the breakdown of average power usage of a common class of servers used in the data center. The results of this study show that the CPU used approximately 31 percent of the power, and approximately 28 percent of the power was consumed in power supply and CPU voltage regulator (VR) losses. The rest was split between memory, chipset, hard drives, removable media storage devices, and miscellaneous components. This study shows that, although many IT organizations focus heavily on the power used by processors, addressing other infrastructure components provides at least as great an opportunity to save power and improve efficiency as focusing on processing power. For example, improving infrastructure efficiency from 41 percent up to 51 percent could allow the addition of 25 percent more servers to this data center. This is a useful insight for those IT organizations challenged to increase performance per square foot in a space-constrained data center—improving infrastructure efficiency can be key. Such organizations may be constrained by infra-structure limitations or by utility providers that have cut them off at a certain power level. They are typically focused on increasing rack density so that they can produce more work in their existing space. However, the gating factor usually is not available rack space, but the ability to cool and provide power to the data center. Building efficient data centers To enhance data center efficiency, IT organizations must tightly manage power consumption across the entire data center at the infrastructure level, rack level, individual server and storage system level, and even component level within these systems. For power consumption in the data center, two best practices should be heeded: • Buy what you need: In the planning and deployment stage of the data center life cycle, enterprises should right-size the data center. • Use what you need: In the operating stage of the data center life cycle, enterprises should take advantage of up-coming power management instrumentation to tightly manage power usages—from infrastructure to racks of servers to individual servers and components within those servers. Buy what you need The scalable enterprise is built upon a foundation of industry-standard data center building blocks, including storage sys-tems and x86 servers that scale out in incremental, cost-effective phases to keep pace with business requirements for increased capacity and capability. The industry-standard x86 servers—equipped with one, two, or four CPU sockets—are interconnected so that the workload can be divided across multiple servers. Additional capacity is achieved by adding enough servers and associated storage to meet immediate growth requirements—thereby avoiding the big up-front in-vestment in excess capacity that is often required when scaling up with symmetric multiprocessor (SMP) servers that have eight or more CPU sockets. Not only does the scale-out approach avoid the period of wasted capacity typical in the scale-up approach, it also avoids the extra power and cooling expense associated with running more compute capacity than is required. Excess capacity and overprovisioning are not just IT issues; they apply equally well to facility infrastructure. In fact, major infrastructure vendors have recently started introducing cooling and power products that embody this scale-out philosophy. For example, the Liebert Adaptive Architecture and the APC InfraStruXure architecture enable modular configurations for a variety of IT environments. Use what you need The pay-as-you-grow philosophy that underlies the scalable enterprise can be extended to using only what you need. For example, in a typical 24-hour period, servers may be operating at only 10–15 percent utilization more than 50 percent of the time. There is an opportunity for significant power savings if underutilized systems and even components within those systems can be transitioned to low-power states during nonpeak periods. A number of industry initiatives at the silicon, product, and infrastructure levels aim to reduce power consumption and to instrument data center IT equipment and infrastructure for more-granular power management than is currently available: • Silicon level: A key focus of the IT industry is gaining power savings at the silicon level, particularly in processor, graphics, chipset, and memory components. This effort is beginning to show results. For example, major processor vendors AMD and Intel have both introduced power-saving features in their latest processors—Intel® Demand-Based Switching with Enhanced Intel SpeedStep® technology and AMD PowerNow!™ technology—and memory and chipset designs are beginning to reflect more power-efficient architectures. • Product level: Dell product designs and specifications strike an aggressive balance between performance, fea-tures, and power consumption. In addition to supporting silicon vendors in their endeavors to create power-efficient architectures, Dell is requiring increasingly greater efficiencies in power supplies and voltage regulators, fans, and hard disk drives. • Infrastructure level: Recent products from Liebert and APC also enable more-granular control over operations, providing cooling capacity that can be matched to the IT load. These products include modular uninterruptible power supply (UPS) systems and computer room air-conditioning equipment with variable-speed fans. A clear trend is emerging: the data center ecosystem of IT, cooling, and power equipment is increasingly being instru-mented for effective power management so that in the future the data center can be more tightly power-managed than is possible today. Eventually, virtualization technology can build on this capability to yield additional power management gains. Enhancing data center efficiency through virtualization Advances in virtualization technology have been fueled by multi-core processing, hardware support for virtualization in processors, multi-resource I/O initiatives, and low implementation costs. These advances are enabling mainstream adop-tion of virtualization technology. In the scalable enterprise data center, virtualization can be used to consolidate workloads from multiple underutilized servers to multiple virtual machines (VMs) on one server. In addition, as systems management capabilities mature, virtualization can allow the data center workload—in the form of VMs—to be dynamically reallocated to meet changing business and demand requirements. This concept, sometimes referred to as workload mobility, can also be extended to power and thermal management. What if it were possible during nonpeak demand periods in the data center to consolidate the workload so that underutilized individual servers, racks of servers, or even entire segments of the data center could be transitioned to low-power states? And what if the associated power and cooling infrastructure were sufficiently manageable to also power down? Industry initiatives are underway that could lead to this level of tight power and thermal management in the data center. Figure 2 shows what could become feasible when IT, power, and cooling equipment are instrumented for effective sys-tems management. In this example, individual servers and racks provide static and dynamic system information such as power consumption levels, fan speed, airflow information, and temperature (represented by ?) to a console. An increase in the load on a power supply could generate information for the console, allowing data center administrators to make intelligent decisions concerning rack or room-level cooling modules. In this way, administrators could manage the level of cooling for a server or server rack. More-comprehensive information about the state of the data center also could enhance data center managers’ decision-making capabilities when planning and deploying new IT equipment. For example, IT equipment could be installed in locations known to have adequate cooling and power for the anticipated workload. This vision, as exemplified in Figure 2, and the heightened interest in data center power consumption has led to collaborative industry efforts such as the Green Grid—an association of top IT companies, including Dell, that are committed to increasing energy efficiency in the data center. Implementing data center efficiency in the scalable enterprise Improving the efficiency of the data center requires looking at the decisions made during every phase of the data center life cycle. For example: • Engineering and planning: Right-sizing the infrastructure for the expected IT load can provide an effective baseline for years of efficient IT operation. • Deployment: Making smart choices about server location helps use available resources to best advantage. • Day-to-day operations: Measuring and monitoring facility efficiency helps enforce discipline in the implementa-tion of industry best practices. • Equipment refresh: Replacing legacy systems with equipment that offers higher performance per watt than the systems being replaced enhances energy efficiency. Dell’s approach to energy efficiency addresses every aspect of the data center life cycle. Dell also remains focused on industry standards, leveraging interoperability standards where available and working to drive these standards where they are lacking. Problems such as energy consumption and density are only solved through the development and adoption of standards from industry groups such as the Standard Performance Evaluation Corporation (SPEC); American Society of Heating, Refrigerating, and Air-Conditioning Engineers (ASHRAE); and Distributed Management Task Force (DMTF). Going forward, scalability offers key opportunities to improve data center efficiency. First, administrators should avoid the inefficiencies involved in overprovisioning the data center. Second, administrators should leverage emerging technologies that can provide more-comprehensive power management capabilities. The focus on buying only what is required and using only what is needed in the data center is the fundamental thinking behind Dell’s scalable enterprise strategy to simplify operations, improve resource utilization, and scale out cost-effectively. John Pflueger, Ph.D., manages Dell’s technical strategy for system and facility thermals, focusing on methods and sys-tem improvements for helping customers improve the efficiency of their computer systems and data centers. He has 15 years of experience in product development, product marketing, and product management. John has a Ph.D. from the Massachusetts Institute of Technology. Sharon Hanson is a technical communications professional in the office of the Dell CTO. She has written and produced Dell white papers, articles, and other technical collateral on industry technology trends for the past 10 years. Sharon has a B.B.A. from the University of Texas at Austin. For More Information: Dell energy efficiency: www.dell.com/energy Metrics for Power Efficiency Today, power efficiency is measured as the ratio of power used by IT equipment to total data center power. Though useful in the past, when space and power typically were plentiful, the industry now needs more-meaningful ways to express efficiency that measure the amount of useful work produced. Efforts are underway in benchmarking organizations such as the Standard Performance Evaluation Corporation to develop such measures. Best Practices: Data Center Efficiency Data center planners and administrators should follow industry best practices to help improve data center efficiency. Fol-lowing are some examples: • Design for efficient cooling layout, such as implementing hot aisle/cold aisle configurations with suitably located air-conditioning equipment and installing blanking panels in unfilled server racks to preserve the hot aisle/cold aisle configuration. • Right-size physical infrastructure systems to the anticipated workload. • Put mechanisms in place to measure data center efficiency, and help ensure that both data center and facilities decision makers are accountable for data center efficiency. • Refresh servers on a regular schedule to take advantage of the performance/watt benefits that accrue with suc-ceeding generations. • Prepare the organization for virtualization and begin planning for deployment of virtualization technology. Dell Datacenter Capacity Planner Dell™ products, tools, and services today help organizations throughout the data center life cycle: planning, deployment, operation, and refresh. Tools such as the Dell Datacenter Capacity Planner help estimate data center power and cooling requirements. Using this capacity planner, data center managers can configure racks of Dell servers, storage, and peri-pherals based on estimated power, cooling, and airflow for each rack and piece of equipment within the rack. To help improve data center efficiency, a suite of Dell IT infrastructure services include server and storage consolidation, data center environmental assessments, and virtualization readiness assessments. By partnering with industry leaders such as Liebert, Rittal, and Sanmina-SCI, Dell is working to address end-to-end data center efficiency with standards-based technologies. For more information about the Dell Datacenter Capacity Planner, visit www.dell.com/energy. Estimating power and cooling requirements using the Dell Datacenter Capacity Planner Inside the Dell Energy Efficiency Research Center Thermal and cooling issues now rank among the top concerns at many enterprises. Over the past 10 years, server densi-ties and power requirements in data centers have risen sharply: data centers once designed for approximately 30 W per square foot are now commonly designed for 120 W per square foot. This power is largely released as heat; in fact, serv-ers release 100 percent of their input power as heat. As a result, the airflow required to cool increasingly dense server installations has begun to outstrip the capacity of typical raised-floor cooling systems. These systems generally handle 4–5 kW per rack, but a rack of blade servers today can require up to 25 kW. To fully populate dense blade server racks, enterprises must consider cooling alternatives such as those installed in the Energy Efficiency Research Center at the Dell Austin Design Center. This 875-square-foot lab is designed to simulate an enterprise data center environment in which Dell engineers test and demonstrate innovative cooling techniques using cur-rent and emerging technologies such as liquid cooling at the chip level and liquid-cooled racks. Heat and cooling simulation The lab is densely populated with racks of special-purpose dummy servers—consisting of a large fan, heating core, pro-cessor, and control board—that enable Dell engineers to simulate different heating loads by setting specific fan, temperature, and power levels. The lab can produce more than 1,000 W per square foot of heat, which is well beyond today’s average of 90–120 W per square foot. A smoke generator offers an effective mechanism for studying and demonstrating airflow: by showing recirculating air patterns, the lab team can test the effect of different cooling levels and techniques, and can visually demonstrate the thermal challenges organizations face as well as potential ways to mitigate those challenges. The lab is equipped with raised-floor cooling that is designed to provide up to three times the industry average in cooling capacity, overhead cooling that boosts capacity another 150 percent, and self-contained liquid-cooled racks. Liquid-cooled racks A liquid-to-liquid heat exchanger and pumping unit provides the interface between the building’s chilled water and four self-contained cooling cabinets—the liquid-cooled racks. Each rack includes 20–30 kW of cooling capacity. A liquid-cooled rack takes water from a closed secondary loop provided by the pumping unit. This water runs through coils in the left side of the front of the rack. The rack also generates its own airflow, so it does not rely on the raised floor for cooling: air is captured from the rear of the servers, pushed through these coils, and exhausted from the right side of the rack in front of the servers. Ongoing thermal and cooling research The Dell Energy Efficiency Research Center is dedicated to educating organizations and addressing cooling issues asso-ciated with increased server densities in enterprise data centers. It is also designed to scale to meet future requirements, including extra built-in cooling and electrical capacity that allow Dell engineers to model server loads significantly higher than those typically seen today. Such provisions equip the lab to continue testing and demonstrating innovative ways to optimize energy efficiency as power and cooling demands continue to rise. The Dell team uses powerful 3-D computational fluid dynamics software to help address airflow and heat transfer issues. Shown here is a sample temperature and air vector plot for a small data center equipped with six rows of rack-mounted serv-ers. Energy Smart Dell PowerEdge Servers The new Dell PowerEdge™ Energy Smart servers—the first industry-standard PowerEdge servers configured specifically to optimize energy efficiency—underscore Dell’s commitment to driving energy-efficient solutions from the data center to the desktop. Available for PowerEdge 1950 and PowerEdge 2950 servers, Energy Smart technology helps reduce energy consumption and associated overall operating costs without sacrificing the performance required to run enterprise applications. Last fall, Dell announced the Energy Smart line of Dell OptiPlex™ desktops. PowerEdge Energy Smart servers include several key features to help minimize power draw and cooling requirements: • Low-flow fan technology automatically adjusts fan speeds based on environmental characteristics to avoid un-necessary power use. • High-efficiency power supplies minimize wasted power and heat output to help reduce cooling needs. • Low-voltage Intel Xeon® 5148 processors are designed to use reduced voltages and half the wattage of pro-cessors such as the Intel Xeon 5160. By maximizing energy efficiency and minimizing cooling requirements, PowerEdge Energy Smart servers are designed to significantly increase performance/watt and reduce both data center operations costs and total cost of ownership. For more information, visit www.dell.com/energy/poweredge. Energy Smart Dell PowerEdge 1950 (top) and PowerEdge 2950 servers help lower total energy consumption and reduce overall operating costs Message Edited by DELL-Stephen on 02-28-2007 11:27 AM

semperreformed · Answer

This subject line gave me an interesting wrinkle in my bosses' thought process of late.   We have increased the number of servers in the data center (server room) from ~25 to approximately 150.   If one of the two air handlers should fail, the server room will certainly overheat. (It happened last summer, we had to shut stuff down). Long story short, I have been trying to get them to fund another and bigger unit.   My question is: is there somewhere in writing on the server warranty where excessive heat would void my dell servers' warranty? (I have read the specs where it says a 2850 is good up to 95 degrees F, but it can't be good for the server for many hours at that temp.   Many, many thanks in advance!!

DELL-Siobhan K · Answer

semperreformed,

I did some checking on your question....
This is kind of a tricky one to answer. If your cooling fails and causes damage to your systems (Dell, Cisco, etc.) then this would be something that you would report to your insurance provider.

All of our server products have thermal protection built in. If they get too hot the server will turn itself off.

The basic simple answer is this is not covered by your warranty, but by your insurance. We will in most cases help out anyways….

aljk · Answer

I have a similar question.  I've found what Dell says is the operational max/min temperatures for their servers.  My question is, what is Dell's recommended temperature for racked servers in data centers?  We are looking to redesign our cooling system and the HVAC guys want to know what the temps need to be in the server racks. Thanks for the help

PowerEdge Hardware General

Was this post helpful?