Fred Palma | Computer Scientist

BERKELEY, CA — A team of scientists from the U.S. Department of Energy’s Lawrence Berkeley National Laboratory (Berkeley Lab) has won a prestigious Gordon Bell Prize, sponsored by the Association for Computing Machinery (ACM), for special achievement in high performance computing for their research into the energy harnessing potential of nanostructures. Their method, which was used to predict the efficiency of a new solar cell material, achieved impressive performance and scalability.

The ACM Gordon Bell Prize annually recognizes the best performance of scientific applications on supercomputers. This year’s prize, presented in a special category for algorithm innovation, was announced Thursday, Nov. 20, at the awards session of the SC08 conference in Austin.
A test run of LS3DF, which took one hour on 17,000 processors of the Franklin supercomputer at NERSC), performed electronic structure calculations for a 3500-atom ZnTeO alloy. Isosurface plots (yellow) show the electron wavefunction squares for the bottom of the conduction band (left) and the top of the oxygen-induced band (right). The small grey dots are Zn atoms, the blue dots are Te atoms, and the red dots are oxygen atoms. (Image courtesy of Lin-Wang Wang)

The Berkeley Lab researchers used three of the most advanced scientific computing facilities of the Department of Energy (DOE) Office of Science for this award-winning work: the National Energy Research Scientific Computing Center (NERSC) at Berkeley Lab, the Argonne Leadership Computing Facilities (ALCF) at Argonne National Laboratory and the National Center of Computational Sciences (NCCS) at Oak Ridge National Laboratory. Their study was titled: “Linearly Scaling 3D Fragment Method for Large-Scale Electronic Structure Calculations.”

Nanostructures, tiny materials 100,000 times finer than a human hair, may hold the key to energy independence. Scientists believe that a fundamental understanding of nanostructure behaviors and properties could provide solutions for curbing our dependence on petroleum, coal and other fossil fuels.

To better understand and demonstrate the potential of nanostructures, the Berkeley Lab researchers simulated their behavior through development of the Linearly Scaling Three Dimensional Fragment (LS3DF) method. These computer algorithms use a novel “divide-and-conquer” technique to efficiently gain insights into how nanostructures function in systems with 10,000 or more atoms.

The LS3DF team consisted of Berkeley Lab’s Lin-Wang Wang, Byounghak Lee, Hongzhang Shan, Zhengji Zhao, Juan Meza, Erich Strohmaier and David Bailey, an agregate of materials scientists, mathematicians and computer scientists contributing their own special expertise to solve this problem.
Lin-Wang Wang, of Berkeley Lab’s Computational Research Division, led the development of the LS3DF algorithms, which used a novel “divide-and-conquer” technique to efficiently compute how nanostructures function in systems with 10,000 or more atoms.

The LS3DF application ultimately achieved a speed of 442 teraflop/s (442 trillion calculations per second) on a Cray XT5 system with 147,146 cores at the NCCS. The Berkeley Lab researchers were also able to run the code on the IBM BlueGene/P system at Argonne, reaching 224 teraflop/s on 163,840 cores, or 40.5 percent of the system’s peak performance capability.

The team first ran the LS3DF application on 36,864 cores of the Cray XT4 (Franklin) at NERSC, achieving 135 Tflop/s. These initial results at NERSC provided the key scientific insights from the application.

“By incorporating the correct chemical formulas into efficient computer programs, scientists can learn a lot about the structures and properties of molecules and solid,” said. Lin-Wang Wang, a computational material scientist who led the Berkeley Lab team. “I like to think of computers as chemistry’s third pillar. In most cases, computer simulations complement information obtained by chemical experiments, but in some cases they can also predict unobserved phenomena.”

A science run using LS3DF, which took one hour on 17,280 cores of the NERSC Franklin system, computed the electronic structure of a 3,500-atom ZnTeO alloy. This run verified that the code could be used to compute properties of the ZnTeO alloy that previously had been experimentally observed. The simulation led to a prediction for the efficiency of this alloy as a new solar cell material.

LS3DF offers a more efficient way for calculating energy potential because it is based on the observation that the total energy of a large nanostructure system can be broken down into small pieces, and each piece can be calculated separately. More traditional methods calculate the entire structure as a whole system and are much more time consuming and resource intensive. Because LS3DF scales almost perfectly with the number of compute cores, it is the first electronic structure code that runs efficiently on computer systems with tens to hundreds of thousands of cores.

“We are excited by the results we are seeing,” said LS3DF team member Meza, who heads Berkeley Lab’s High Performance Computing Research. “The efficiency of LS3DF on these large computer systems is impressive, but the real story is the power of algorithms. Using a linear scaling algorithm, we can now study systems that would otherwise take over 1,000 times longer on even the biggest machines today. Instead of hours, we would be talking about months of computer time for a single study.”

Getting codes to run with such high efficiencies on massively parallel machines is not a trivial task. Bailey, Shan and Strohmaier of the DOE Office of Science’s Scientific Discovery through Advanced Computing (SciDAC) Performance Engineering Research Institute (PERI) worked hand-in-hand with Wang and his colleagues to analyze the performance of LS3DF and to identify potential performance improvements. Responding to this analysis, Berkeley Lab researchers assisted with a major revision of the code, which led to the prize-winning submission.

“The computational power we have is staggering and it is important to make sure that each research project can effectively harness the power of Argonne’s Intrepid and optimize their calculations”, said Katherine Riley, the ALCF computational scientist who worked with the Berkeley Lab team. “Not only can we drastically reduce the time it takes to generate results, we can help scientists ask different questions and develop new insights in order to accelerate breakthroughs.”

Once the LS3DF code had been optimized it was a matter of days before it was running at each of the DOE supercomputing facilities. Oak Ridge National Laboratory invited Wang and other Gordon Bell finalists to carry out runs on ORNL’s leadership Cray supercomputer, Jaguar. In Wang’s case, the winning simulation was achieved after only two runs over a two-day period, demonstrating the ease of porting – and running – high-performance applications on the Cray XT architecture. The project had previously been awarded time on Jaguar under DOE’s Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program.

“We still don’t quite understand how the electron moves around in a nanostructure, and how such properties depend on the size, geometry, composition and surface passivations,” said Wang. “Understanding this dependence will allow us to design nanostructures for desired applications. Using our improved LS3DF method will help us to understand and predict these properties.”

The ALCF, NCCS and NERSC are funded by the Office of Advanced Scientific Computing research in the DOE Office of Science, providing some of the world’s most powerful computing resources and support to thousands of researchers around the country.

Berkeley Lab is a U.S. Department of Energy national laboratory located in Berkeley, California. It conducts unclassified scientific research and is managed by the University of California. Visit our Website at www.lbl.gov

Additional Information

For more information about NERSC, visit: www.nersc.gov

For more information about the ALCF, visit: www.alcf.anl.gov

For more information about the NCCS, visit: www.nccs.gov

From Berkeley Lab

Tags: Energy, Nanotechnology by Fred
No Comments »

From supercomputingonline.com

ESnet, Internet2 and USLHCNet Provide Critical Link For Petabits Per Day of Data to U.S. Scientists Participating in the Large Hadron Collider Research: Today marked the first-ever attempt to circulate a beam of subatomic particles around the Large Hadron Collider (LHC), a gigantic particle accelerator spanning the French-Swiss border. The event represents a major milestone along the path towards a new understanding of the fundamental nature and origins of the universe.

When the LHC officially begins its experiments, multiple terabytes of data per second will flow out of Europe via fiber optic cables to thousands of researchers spread across the globe, including over 1700 in the United States. This experiment will significantly increase the amount of data that the U.S. scientific community must transport and manage.

Fortunately, the U.S. Department of Energy’s (DOE) Energy Sciences Network (ESnet), Internet2, the country’s leading research and education network, and USLHCNet, which provides transatlantic network connectivity from the LHC facility to the United States have prepared for moving the massive amounts of data to U.S. sites where scientists can analyze the information.

These organizations have worked closely together to aggressively deploy the most advanced networks with enough bandwidth and capabilities to reliably transport multiple streams of 10 Gigabits of information per second – the equivalent of transmitting 500 hours of digital music per second for each 10 Gigabit line. The LHC will be the first experiment to fully utilize the advanced capabilities of these networks, which will connect DOE national laboratories and university researchers across the country to the LHC data.

“The science environment of today is very different from that of a few years ago. The advanced networks of ESnet, Internet2 and USLHCNet will provide the high-speed, extremely reliable connectivity between U.S. laboratories, universities and the international research institutions required to support the inherently collaborative, global nature of modern large-scale science,” said Steve Cotter, department head for ESnet.

Initially, the European Center for Nuclear Research (CERN), which manages the LHC, will store the experiments’ data. The information will then traverse the GÉANT2 network and migrate across the Atlantic Ocean via fiber optics, on a network called USLHCNet, which was developed and deployed by the researchers at the California Institute of Technology in Pasadena, California. The LHC will generate many petabytes of data during each year of operation, and will accumulate an exabyte of real and simulated data within the first decade of its estimated 20 years of operation. The data will be distributed for processing among 150 computing and data storage facilities around the world, and will be analyzed intensively, and repeatedly as physicists and students refine their analysis methods and respond to any emerging discoveries.

“As a physicist who has been preparing for the LHC for nearly fifteen years, I am extremely excited about the milestone we have reached today in circulating the first beams at the LHC,” said Harvey Newman, Professor of Physics at the California Institute of Technology. “The advanced networking and cyberinfrastructure resources created through partnerships among ESnet, Internet2 and USLHCNet make it possible for myself and my colleagues across the country to participate in the LHC experiments – which we believe will change scientific history.”

Like virtual Ellis Islands, two high-performance exchange points, MAN LAN in New York City and Starlight in Chicago, will be the U.S. entry points for LHC data. From there, ESnet will deliver data from the LHC’s ATLAS detector to The Brookhaven National Laboratory, in Upton, New York where it will be processed and stored. Meanwhile, data from the LHC’s CMS detector will go to the Fermi National Accelerator Laboratory in Batavia, Illinois, for processing and storage. From these laboratories, ESnet and Internet2 together with its regional network partners will distribute the data among 1700 U.S. scientists at 94 institutions throughout the country participating in this massive project, many of whom are supported by the DOE’s Office of Science and the National Science Foundation. Internet2 and ESnet officially launched a partnership in 2006 to develop and deploy the next-generation ESnet4 just in time for the LHC.

”Advanced networking is a critical part of the global infrastructure supporting the Large Hadron Collider, which represents the largest scientific experiment in history. Just as the World Wide Web was begun to promote information sharing among scientists, our advanced IP network and new networking technologies such as dynamic circuit networking that have been deployed by Internet2, ESnet and its partner networks ensures U.S. researchers have the most sophisticated resources to access the data from the most sophisticated scientific device in the world,” said Rob Vietzke, executive director of network services for Internet2.

The LHC has been nicknamed the “Big Bang Machine” because scientists will use it to recreate the cosmic conditions one trillionth of a second after the big bang, in hopes of finding insights into the origins of matter. It consists of a 27 kilometer tunnel and cathedral-sized caverns 100 meters underground. The accelerator magnets that guide the beams on their circular orbit are supercooled to a temperature just slightly above absolute zero, which is colder than outer space. It will accelerate matter to 99.999999% the speed of light, and recreate conditions a trillionth of a second after the big bang.

On October 15, 2008, Internet2 will provide a special peek behind the scenes at the LHC during its upcoming Fall 2008 Internet2 Member Meeting being held in New Orleans, LA—just a week before the expected first atomic collisions are anticipated at the LHC. The event will be netcast live for worldwide viewing. For more information, visit its Web site.

Tags: Development, Energy, research, science by Fred
No Comments »