This is only a preview of the July 2015 issue of Silicon Chip. You can view 35 of the 96 pages in the full issue, including the advertisments. For full access, purchase the issue for $10.00 or subscribe for access to the latest issues. Items relevant to "Build a Driveway Monitor, Pt.1":
Items relevant to "Install USB Charging Points In Your Car":
Items relevant to "Intelligent Charger for Nicad & NiMH Batteries":
Articles in this series:
Items relevant to "Ultra-LD Mk.4 200W RMS Power Amplifier: Preview":
Purchase a printed copy of this issue for $10.00. |
By Geoff Graham
The Pawsey
Supercomputing Centre
Just what is a supercomputer? How does it work? What do you use
it for? We take you inside the Pawsey Supercomputing Centre to
meet Magnus, the fastest and most powerful supercomputer in the
Southern Hemisphere.
F
ROM THE OUTSIDE, the Pawsey
Supercomputing Centre appears
rather modest, just a low building set
into the hillside. Located in Technology Park in the leafy suburb of Kensington in Perth, Western Australia it
houses two supercomputers, a host
of supporting computer systems and
a huge data-storage facility.
The statistics are impressive. The
smaller supercomputer called Galaxy
can perform at 200 teraflops – a teraflop
is a million million floating point calculations per second. So it’s no slouch.
However the star is Magnus, a Cray
20 Silicon Chip
XC40 supercomputer capable of 1.5
petaflops. A petaflop is a thousand million million floating point calculations
per second and this makes Magnus
the most powerful public research
computer in the southern hemisphere.
There may be a more powerful computer in existence (who knows what
ASIO have hidden behind their walls)
but this is definitely the fastest publicly acknowledged computer.
Everything here is big; the storage
systems can store 70 petabytes of
data with an expansion capacity to
100 petabytes. The power consump-
tion is around 900MW and water is
drawn from underground to keep the
supercomputers cool.
In the beginning
The Pawsey Supercomputing Centre
was named after pioneering Australian radio astronomer Dr Joe Pawsey. It
started life in 2009 with an $80 million
grant from the Federal Government,
in part to support Australia’s push
to be the southern hemisphere’s site
for the Square Kilometre Array radio
telescope (see SILICON CHIP, December
2011 & July 2012).
siliconchip.com.au
The Pawsey Supercomputing
Centre in Technology Park, Perth,
Western Australia houses the fastest
supercomputer in the Southern
Hemisphere. The building was
designed to merge with the landscape
and reflect the geosciences, a major
user of the supercomputers. Photo
credit: Pawsey Supercomputing
Centre.
The centre opened in 2013 and still
processes a lot of data from the radio
telescope but the rest of its capacity
(about 75%) is dedicated to the five
partners operating the centre (the
CSIRO and four WA universities) and
researchers in Australia in general.
In some respects the Pawsey Centre
is unique because they not only provide the computer facilities but they
also train and help researchers to get
the best results from the system. The
centre also has a number of systems
dedicated to visualising the data so
that researchers can watch the result
of a simulation and that makes understanding the data much more intuitive.
For example, a geologist would
normally take core samples in the field
and then analyse these to try to map
the ore deposit. By using the Pawsey
supercomputers, they can go much
further and calculate the distribution
of the sample results through the geology of the region. In addition, by using
3D glasses along with the visualisation
technology, they can stand inside the
ore deposit and look around to see how
it is distributed.
Inside Magnus
So just what is a supercomputer?
siliconchip.com.au
This is Magnus, the fastest computer in the Southern Hemisphere. The cabinet
artwork is by Margaret Whitehurst and pays homage to the centre’s close
connection to the north-west of Western Australia. It has been designed to reflect
“the ground below”, in reference to geoscience, one of the areas the super
computing centre supports most closely. Photo credit: Pawsey Supercomputing
Centre.
These days, it is basically just a massive set of processors that work on a
problem in parallel. In the case of Magnus, that is 35,712 processing cores.
The processors are standard Intel
Xeon E5-2690V3 Haswell units. Each
processor has 12 cores running at
2.6GHz and two processors together
with some local memory make a node,
which is the basic computing element.
Four of these nodes are physically
mounted on a blade which is a large
plug-in module and is the replaceable
part in the case of a component failure.
Magnus consists of eight cabinets with
each cabinet holding 48 blades for a
total of 1488 nodes (35,712 processor
cores).
You might think that you could
build a supercomputer like this using
a lot of standard desktop motherboards
but that wouldn’t work when scaled to
the numbers required by Magnus for
scientific workloads. Removing heat
is one issue but also getting the data
to each processor requires a special
network.
Interconnecting the nodes
In Magnus, the nodes are interconnected with 4km of high-speed optical
fibre and copper, making a network
July 2015 21
Galaxy is the smaller of the two supercomputers. Its primary purpose is to
process data coming from the radio telescope arrays (ASKAP and MWA) in
the Murchison in the north of Western Australia. When they are running,
these telescopes generate an amount of data equivalent to one DVD every two
seconds and this data needs to be processed in real time. Photo credit: Pawsey
Supercomputing Centre.
keep the specialised hardware and
operating software running smoothly.
Interestingly, the management
software for Magnus (called SLURM)
runs under Linux. Linux and SLURM
run on a specialised processor and are
responsible for distributing work to the
various compute nodes. So in practice,
the researchers and technicians running the supercomputer interact with
Linux – originally a hobby project by
a young lad in Norway.
Because a supercomputer is a scarce
resource (there are not many around)
getting time on it takes some effort.
The researcher must make a proposal
which is assessed by a committee who
consider the scientific value of the application and the processing time that
it would require.
Once over that hurdle, the program
must be prepared and queued for processing. It is rather like the old batch
systems of yesterday; you submit the
program and data and wait for a processing slot. However, it is worth it – in
just one hour, Magnus can do more
work than a conventional computer
could do in two years.
Using a supercomputer
The layout of the Pawsey Supercomputing Centre. The large white space at the
top is the supercomputer cell, below it is the I/O cell, and the lower white space
is the tape cell. Each cell has different temperature and humidity requirements,
differences between water cooling versus air cooling, and differences in
whether mains power or uninterruptible power is used. The supercomputer
cell is primarily water-cooled and on mains power. Photo credit: Pawsey
Supercomputing Centre.
capable of an aggregate bandwidth of
over 100,000 gigabits per second. The
network (called Aries) runs a special
protocol designed to keep latency low.
Local storage for Magnus is three
petabytes with a sustained read/
write performance of 70GB/s. This is
just used for temporary storage with
22 Silicon Chip
the end results going to a separate 70
petabyte storage system maintained
by the centre.
Managing Magnus
Magnus was built by Cray Inc in
the USA and two Cray engineers are
located permanently on site to help
Because a supercomputer is a massively parallel machine it tends to
work better at some jobs than others.
These include simulations of physical systems, image processing and
geophysical mapping.
A typical application that works
well is atmospheric modelling. In this,
the atmosphere is divided into cubes
of a few kilometres in each dimension.
Each processor in the supercomputer
is allocated the job of simulating the
changes in one cube and while it is
doing that, the other processors are
working in parallel on other cubes.
When one processor has finished,
its will be allocated a new cube to
process. Because there are many,
many cubes, all the processors in the
supercomputer will be busy for some
time. The results of each simulation
are then aggregated to gain an image
of the whole system.
Similar approaches are used to
model ocean currents, star formation,
the generation of tsunamis, investigate
the electromagnetic structure of matter
and more.
Some more novel uses of the Pawsey
supercomputers have been sequencing
the genome of the cane toad and investigating the porosity of bread. This last
siliconchip.com.au
Helping to put you in Control
Programmable Step Pulser
The KTA-301 provides signal
to control speed, acceleration/
decleration rate & direction to
a stepper via DIP switches. 2
potentiometer input for speed
& acceleration/deceleration
control. 8 to 30 VDC powered,
DIN rail mountable.
SKU: KTA-301
Price: $89 ea + GST
2-Switch Button Control Box
The Cray supercomputers use four kilometres of fibre optic cables (shown here)
and copper cables to distribute data to the processing nodes. The network,
called Aries, runs a special protocol designed to keep latency low. Photo credit:
Pawsey Supercomputing Centre.
Red, black, 2-switch push
button control station contains
1 x NC contact black pushbutton ad 1 x NO contact red
pushbutton. With addition of
a contactorrelay the user can
use this control station as a
direct on line (DOL) motor
starter.
SKU: HEE-020
Price: $27.50 ea + GST
TagTemp NFC Data Logger
TagTemp NFC temperature
data logger with 1 year
(typical) life. LogChart-NFC
an android smartphone app
allows configuration and data
download via the NFC link.
Measure range between -40 °C to +70 °C.
SKU: NOD-060
Price: $85 + GST
Teensy
The teensy is a breadboard free
development board with a 32
bit ARM Cortex microprocessor
and Arduino-like programming.
It features; 64K RAM, 34 I/O
pins, 12 PWM outputs, 3 UARTs, SPI/I²C/
Can Bus. 3.3 VDC powered.
SKU: SFC-012
Price:$29.50 +GST
Differential Pressure Transmitter
This photo shows a Cray X40 supercomputer blade which is the replaceable
module in case of component failure. Each blade holds four nodes and in
Magnus each node consists two standard Intel Xeon E5-2690V3 Haswell
processors with 12 cores running at 2.6GHz. The total number of processing
cores on a blade is 96. Photo credit: Cray Inc.
one might sound silly but it is quite
important to Australia as the international perception of Australia’s wheat is
that it is not suitable for bread making.
Researchers at the CSIRO’s Food
Futures National Research Flagship
used X-ray micro-tomography to examine the structure of bread and the
resources at the Pawsey Supercomputing Centre to visualise the structure.
With the knowledge gained, it is hoped
that future research will help improve
Australia’s standing in this important
market.
Another unusual application is
the Sydney-Kormoran Project which
is processing images from the WWII
shipwreck sites of HMAS Sydney and
siliconchip.com.au
HSK Kormoran. The aim is to provide
a moveable 3D image of the two ships
resting on the ocean floor for researchers and the public to examine.
Huge amounts of data
With all this processing, there is a
lot of data, especially from the radio
telescope arrays (ASKAP and MWA) in
the Murchison in the north of Western
Australia. When they are running,
these telescopes generate an amount
of data equivalent to one DVD every
two seconds and the data needs to be
processed and archived in real time.
One supercomputer (called Galaxy)
is dedicated to this task, with the data
saved onto a sophisticated storage sys-
IP54, DPS series differential
pressure transmitter has a 0 to
1 mBar or 100 Pa input pressure
range and loop powered 4 to 20
mA signal output. 10 to 30 VDC
loop voltage.
SKU: DBS-5501
Price: $199.95 ea + GST
Waterproof Temp. Sensor
DS18B20, digital thermometer in waterproof
6 × 30 mm probe with
15 metre cable. -55 to
125 °C range with ±0.5 °C accuracy from
-10 to 85 °C. 5 VDC powered.
SKU: GJS-002
Price: $19.50 ea + GST
PSU With Battery Charger
DIN rail, power supply with battery
charger (UPS function). Provides
AC fail and low battery alarms. 90
to 264 VAC input, produces 13.8
VDC <at> 4.5 A output.
SKU: PSM-1171
Price: $99 ea + GST
For OEM/Wholesale prices
Contact Ocean Controls
Ph: (03) 9782 5882
oceancontrols.com.au
Prices are subjected to change without notice.
July 2015 23
probably be still held on a spinning
disk, so it would be returned immediately. But if they requested some very
old data, the chances are that it would
have been archived and a robot tape
arm would then swing into action to
retrieve the right tape and place it in
a drive. In that case, there would be
a delay of some minutes before the
data is returned but other than that
it would be no different from reading
any other file.
Tapes may be regarded as low
technology and many would ask why
not just use more disk drives and do
without the complex tape system. The
reasons are capacity, power and heat.
The existing system has a capacity
of up to 100 petabytes which would
require an unimaginable number of
disk drives and even if they were used,
the power and cooling requirements
would be unsustainable.
Power
A view inside the robotic tape library. The large black column in the centre
is the robotic tape arm. This travels up and down on rails between the tape
cartridges on either side, retrieving cartridges and delivering them to tape
drives. The robot system is completely automated and looks like a very large
disk drive to the supercomputers. Credit: Pawsey Supercomputing Centre.
tem that is also used by Magnus and
other systems in the centre.
This storage system looks like a large
single disk drive to the rest of the facility but is in fact an array of spinning
disks which act as a cache to a large
tape library managed by robots. The
disks are high-reliability versions of the
standard disk drives that we all have
in our computers and on their own
add up to six petabytes. The operating software distributes the data over
the drives so that if one or more fail,
the missing data can be reconstructed.
The operating software is also responsible for automatically archiving
little-used data to the tape library. This
consists of robot arms which retrieve
tape cartridges from storage and place
them into tape drives so that data can
be written and read by the main system. The software keeps track of what
piece of data is written onto which
tape at what position. In total, the tape
system can hold up to 100 petabytes.
All this is transparent to the rest of
the system. A researcher could request
some data that is recent and it would
Gigaflops, Terabytes & More
The basic units used in the supercomputing world are gigaflops, teraflops and
petaflops for processing capacity and terabytes and petabytes for storage.
Giga means a thousand million or 109,while Tera means a million million or 1012
and Peta is a thousand million million or 1015.
A FLOP stands for floating point operations per second. Most scientific computing involves manipulating floating point numbers which is why this measure
is used. Note that benchmarking computers is a tricky business and the claimed
numbers can vary considerably depending on how someone runs the benchmark
or calculates the result.
By way of comparison, a typical dual core processor on a laptop or desktop
computer would have a theoretical maximum performance of about 20 gigaflops.
24 Silicon Chip
The overall supercomputing centre
draws about 900MW from the Western Australian grid. You might think
that a lot of this has to be backed by a
UPS and diesel generators to keep the
supercomputers running but that is not
so. Some systems are protected by a
UPS but the supercomputers are not.
This is because firstly they draw
such a huge amount of power that a
properly-sized UPS would be prohibitively expensive. The second factor is
that, by its nature, a supercomputer
does not need to be kept running during a power blackout. It does not store
much data and any interrupted jobs
can be simply restarted when the
power is restored.
The centre does have a system to protect the supercomputers from glitches
on the power line though. This consists of a large electric motor driving
a flywheel which is in turn connected
to a generator. If there is a glitch in the
power, the momentum of the flywheel
will keep the generator running and
insulate the supercomputers from any
ill effects.
Groundwater cooling
An intriguing feature of the Pawsey
Supercomputing Centre is the cooling
used for the supercomputers. Magnus
alone generates about 400kW of heat
and a cooling system for that heat
load would be expensive to provide
and operate.
In a world first, the CSIRO Geothersiliconchip.com.au
In a world first development by the CSIRO Geothermal Project fresh water is drawn from the Mullaloo Aquifer 100 metres
underground to cool the supercomputers. This water is at a constant temperature of 21.5°C and after doing its job is
returned to the aquifer at about 24.5°C. Given the size and depth of the aquifer the effect on it is minimal. Credit: Pawsey
Supercomputing Centre.
mal Project developed a system where
by cool water is drawn from an underground water body called the Mullaloo
Aquifer, about 100m below ground.
This water is at a constant temperature
of 21.5°C and after being used to cool
the supercomputers is returned to the
aquifer at about 24.5°C. On a particularly hot day, a second return system is
used but the effect on the underground
water system is minimal.
To further bolster the system’s green
credentials, the pumps used to move
the water are powered by solar panels
on the roof of the centre.
The power savings are significant
but the most important factor in water
starved Western Australia is the saving
of approximately 14.5 million litres of
water every year, compared to a conventional system using evaporative
cooling towers.
As part of the research involved in
this project, it was discovered that the
underground aquifer is slowly moving
and in a 100 years or more it will have
passed from under the supercomputer
centre. By then, computers and cooling
requirements will have changed so this
siliconchip.com.au
HMAS Sydney which was sunk with all lives in a battle with the German
auxiliary cruiser Kormoran which was also sunk during the engagement. The
Pawsey Supercomputing Centre is processing photographs of these ships lying on
the seabed to create a moveable 3D image for researchers and the public to view.
was not considered a concern.
This system of cooling involved
cutting edge research and has attracted
world-wide attention. Unfortunately,
funding for the group responsible
was terminated with the change of
government in Canberra and the expertise has dissipated. Such are the
ups and downs of a publicly funded
SC
organisation.
July 2015 25
|