David Porter, Sarah Anderson, and Paul Woodward of the University of Minnesota's Laboratory for Computational Science & Engineering (LCSE) have recently performed the first 3-D supercomputer simulation of a model red giant star.

This team has been studying the process of convection in the outer layers of stars like the sun for over a decade. Only in the last year, however, have both the supercomputers and the numerical methods allowed detailed simulations of the 3-D dynamics of entire model stars. The team's recent simulation of a red giant introduced special challenges, because these stars are convectively unstable throughout most of their volume. Red giants have hot, dense stellar cores which are surrounded by vast, diffuse stellar envelopes. A typical red giant envelope would extend from the sun's location as far out as the orbit of Jupiter. The heat generated by nuclear reactions in the core of such a star is transported to the surface by material motion, or convection. The diffusion of light through the gas, which carries this heat out through the dense core, is not efficient in the envelope. Instead, gas heated by contact with the hot core becomes bouyant and rises upward, carrying its heat to gas nearer the surface. This heat can be given up to gas nearer the surface by radiation diffusion, which is greatly increased in efficiency by mixing of the gases through small-scale turbulent motions. At the surface, this heat is ultimately emitted into the surrounding space in the form of light.

Simplifying the Model:

This process of heat generation, transport, and escape into space is tremendously complex - too complex to be treated in full detail in this first 3-D simulation of the phenomenon. To keep this computation down to manageable proportions, the LCSE team decided to simplify the stellar model significantly, while retaining as much as possible in this first trial of the essential interaction of the convective instability, the generation of turbulence, the conduction of heat by photon diffusion, and the escape of the heat at the stellar surface, or photosphere. Viscous effects, which act only on tiny scales unresolvable by the computational grid, were approximated by a carefully formulated numerical viscosity. This viscosity of the numerical scheme dissipates kinetic energy of fluid motion into heat, like the real viscosity of the gas, but on the much larger scales of the computational grid. This numerical viscosity was carefully designed to restrict its dissipative effects to the shortest length scales possible, consistent with accurate representation of the nearly inviscid flow on the longer length scales. The radiative heat conduction was modeled through a thermal diffusivity which depended only on the gas pressure, and which therefore had a relatively simple dependence upon depth in the envelope. The complex effects of the changing ionization state of the gas, which are thought to play an important role in the dynamics of red giant envelopes, were omitted from this first calculation, but are intended to be included in subsequent work of this team. Similarly, magnetic fields were not included, and the star was assumed not to rotate.

Aside from omitting the effects of varying gas ionization states, the largest falsifications in this red giant envelope model were made near the surface, where the transported heat escapes in the form of light radiated into space. The escape of the radiation at the stellar surface is a very complex phenomenon, and it is not clear that the details of this phenomenon have much effect upon the general dynamics of the gas layers beneath. The standard formula for the radiation flux of a black body, given its temperature, was used in the gas layers whose densities indicated that they were likely to reside about one optical depth or less below the stellar surface. Thus the total energy radiation rate of the star was kept consistent with its local surface temperatures and surface area, and a very costly and complicated treatment of the radiative transfer was avoided.

In this simulation, the focus of interest was on the convection of the envelope, not on the stellar core. As a result, only essential features of the core needed to drive the envelope convection were retained. Heat was introduced steadily into the core in order to drive the convection, but the detailed nuclear reactions causing this heat generation were not computed. Also, the team felt that only the small size of the core relative to the envelope was likely to be important for the convection dynamics, so they felt free to falsify the size of the core in order to permit its spherical shape to be resolved on the fine, uniform grid used to resolve the motions of the envelope. The core radius was therefore set to about 15% that of the entire star, which is a significant overestimate. Nevertheless, the core in the model does indeed, like a real red giant core, occupy only a small fraction of the stellar volume. Making the core artificially large allowed the dynamics of the gas motions near it to be accurately computed without requiring adaptive grid techniques to be introduced or prohibitively small time steps to be used.

Special power of the numerical approach:

After recounting all these simplifications and falsifications of the red giant model, one may wonder what was left to simulate correctly. The principal interest of this simulation for the LCSE team was the potentially global nature of the stellar convection, its potential coupling to global modes of radial pulsation of the stellar envelope, and the strong effects of compressibility of the gas which result from the nearly sonic gas motions induced, especially near the stellar surface. Computational power was lavished on these issues of scientific focus in this simulation. The numerical method employed by the team, the Piecewise-Parabolic Method (PPM) of Woodward and Colella, is especially well suited to handling fast compressible flows. It was enhanced by Porter, Woodward, and Anderson for this simulation by introducing heat conduction, a treatment for the free surface of the stellar envelope, and by a major restructuring for highly efficient parallel computation of the 128-processor SGI/Cray Origin-2000 of the National Center for Supercomputing Applications (NCSA).

This simulation of the model red giant star was carried out over about a week of dedicated computing on NCSA's 128-processor Origin-2000. Over a terabyte (1000 GB) of compressed data was saved from this computation on Ampex D-2 tapes and sent to the LCSE for post-processing, analysis, and visualization. In addition, the LCSE team was able to look in on this computation as it ran by bringing highly compressed data for visualization over the NSF's new high-speed research network, the vBNS, to Minnesota. Movies of LCSE visualizations made from this data using the vBNS were shown at the Supercomputing '97 exhibit in San Jose as the simulation was still running. Some of those images are shown in the figures below. These figures, as one might expect from a first 3-D simulation of a complex phenomenon like this, reveal a number of surprises.

Surprises from the initial results:

The first surprising result of this work was the development of convection on a global scale in the stellar envelope. Over the last 50 years of study of stellar structure and evolution, which has been restricted to 1-D models until very recently, researchers in this field have grown accustomed to thinking in one-dimensional terms. Convection is traditionally modeled as a small-scale phenomenon whose details may be averaged over a constant-pressure surface and over time to yield an effective description of convective heat transport in terms of the famous "mixing length," a model parameter comparable to a local pressure scale height. Over the last decade work of the LCSE team and of many others has extended our understanding of convection first to 2 and then to 3 dimensional detailed models. This work was aimed at understanding solar-type stars, which, unlike red giants, are convectively unstable only in their outer layers. Even though it revealed turbulent convective structures extending over several pressure scale heights in depth, this work ultimately confirmed the usefulness of the mixing length model for 1-D stellar evolution calculations. However, if this new 3-D simulation is taken as representative, the extremely deep convectively unstable envelopes of red giant stars behave very differently. At least visually, the convective flow in this simulation is dominated by a dipolar convection pattern encompassing the entire envelope. Although detailed analysis of the initial data from the red giant convection simulation is certainly needed, it is hard to see how a mixing length comparable to a single pressure scale height in this envelope could characterize this global convection in any useful way. This flow pattern is most easily appreciated by viewing the quicktime animation at the LCSE Web site:
http://www.lcse.umn.edu/research/RedGiant/OTIother.mov (97Mb quicktime)
http://www.lcse.umn.edu/research/RedGiant/OTI2330b.mov (947Mb quicktime)
These animation files are very large. They may therefore take some time to download, although both will play easily and well on a standard laptop computer. The smaller version sacrifices some spatial resolution.

Here we see the model red giant star near the extrema of its pulsation. At the top, the relatively diffuse envelope has contracted to about its minimum size, while at the bottom, the envelope is shown at about its maximum size. These images show volume renderings of the temperature perturbations relative to average values on each constant pressure surface. These temperature averages were computed for a single state near the middle of the pulsation. Relatively warm temperatures are red and yellow, while relatively cool temperatures are blue and aqua. The volume renderings at the right are relatively transparent, allowing us to see right through the envelope and to see the dense, stable stellar core within.

Here the model red giant star is shown near the middle of its pulsation. At the top, the temperature perturbations are rendered relatively transparently, so that we may see right through the envelope to the dense, stable, and hot stellar core within. Note that in this relatively transparent volume rendering, we have the "limb brightening" effect familiar to astronomers. This effect causes the warm regions near the surface of the star to seem more prominent and important than they actually are. Inside the convective envelope, the largest temperature excursions are in the cool gas, and therefore the relatively warm regions, which are not really that warm, are pretty much transparent in this rendering. There is a large component of a global convective mode encompassing the entire envelope. Relatively cool gas sinks from the surface at the upper right in the upper image, forming a rapidly moving funnel of gas which strikes the stellar core. This cool gas is strongly heated as it flows around the core. In the lower image we have moved inside the stellar envelope, viewing the funnel of cool gas flowing around the core from much closer up but from precisely the same angle as in the upper image. The turbulent nature of this flow past the core is evident, and heated gas can be seen leaving the downstream side of the core. Animations of such images reveal that this large-scale convection flow, in which one side of the stellar envelope is systematically cooler than the other, persists throughout the pulsational period of the envelope. An analysis of this preliminary data from the simulation, searching for quantitative correlations between the convective and pulsational motions, is underway.

The dipolar convection flow field is best seen in the more transparent images presented at the right in the first set of images above. These images show volume renderings of the temperature perturbations, relative to a roughly time-averaged state for each pressure within the star. Temperatures near the reference values cause the gas to be rendered as transparent, while higher temperatures make the gas visible as red or yellow (hottest) and lower temperatures are rendered as blue or aqua (coldest). Near the surface and near the stellar core both warm and cool gases are evident, while in the bulk of the volume the warm perturbations are weaker than the cool ones, so that only the cool gas can be seen there in these renderings. In each of the semi-transparent images, which allow us to see right in to the stellar core, a funnel of cool gas can be seen descending from the stellar surface at the upper right and aimed directly at the core. This is a persistent feature of the flow. In the second set of images, taken near the middle of the envelope's radial pulsation period, a close-up view from just outside the core shows this cool gas funnel surrounding the core, which heats it as it passes by, producing hot gas regions on the downstream side. This global convection pattern is so strong and so persistent that there are even a few snap shots within the sequence in the quicktime movie which clearly show that one side of the star is warmer than the other. In principle, this is a phenomenon that could be observed in real red giant stars.

The second surprise coming from this simulation is that the stellar envelope clearly pulsates in a regular fashion. Astronomers will note that this effect is to be expected of red giants. However, it is generally believed that such pulsations are caused by effects of the changing ionization state of the gas in the envelope near the stellar surface. Therefore we must be surprised that this simulation, which does not include any such effects, exhibits such regular and large pulsations. The amplitude of the pulsation, which is substantial, as it is known to be in real pulsating red giant stars, is shown by comparing the images below, which show representative smallest and largest states. Since this envelope exhibits such vigorous and large-scale convection, we should perhaps not find it surprising that this global motion couples to pressure oscillations of a more spherically symmetric but equally global nature. The simulation follows several pulsational periods, but of course it is not long enough to verify whether this pulsation is damped or not. Further work will be required to sort this out, both in the form of careful analysis of the 1 TB data set from this simulation and also in the form of new numerical experiments. It is important to understand this pulsation in detail, because pulsating red giant stars are used as "standard candles" to establish the distance scale by which we measure the size of our universe. Such a detailed understanding will require ionization effects to be included in the red giant model along with a more accurate treatment of the escape of radiation at the stellar surface.

How they did it:

This red giant simulation was performed on NCSA's 128-processor SGI/Cray Origin-2000 in November, 1997. This machine was the first machine of this size at a customer site to have a single system image, with all 128 processors transparently sharing each other's local memories while maintaining cache coherency. Local copies of shared variables stored temporarily for fast access in processor caches were all maintained consistent by the hardware with the globally accessible copies in the shared memory. This red giant star simulation with the LCSE's PPM gas dynamics code was one of the first applications to successfully exploit this unique, powerful new computing platform. The code was specially structured to take advantage of this distributed shared memory (DSM) computer architecture. It did so successfully by making only sparing and truly necessary use of the cache coherency mechanism. Each processor was assigned tasks involving the updating of shared array variables, but this data was first copied into private, local arrays in the processor's cache memory before performing the extensive arithmetical operations upon it which resulted ultimately in new values to be written back into the globally shared memory. In this fashion, the 128 processors were able to do their work from a common basis but without any need to frequently inform each other of the developing intermediate results of their independent tasks.

Those familiar with the MPPs of the last several years might think that such a computational approach would be easy to implement on machines which do not have globally shared memories with cache coherency. This is not the case. The red giant computation, because it was carried out on a regular Cartesian mesh, was highly irregular. First, each time step was decomposed into three 1-D sweeps, performed in sequence, each of which updated the fluid variables taking into account only derivatives in one of the 3 space dimensions. (This is a procedure called directional operator splitting, and it brings about a dramatic simplification of the numerical algorithm.) Each sweep was in turn decomposed into a series of subgrid updates, which could be performed independently and in parallel, but for which a particular order of task completion was preferred in order to allow the next 1-D sweep to begin before the present one was completed. First, the subgrids used for these single-processor tasks had long, thin shapes in order to optimize processing efficiency for the 1-D updating algorithm. They were grid pencils 8 grid cells on a side and as long as the entire mesh, 512 cells, in the direction of the sweep. As a result, the shared data for each of these tasks was distributed over the local memories of many processors. The DSM hardware of the Origin-2000 was exploited to collect this data rapidly and transparently into a processor's cache at the beginning of each task and to distribute the new data back again without any corresponding complexity in the Fortran code. Effectively, the DSM hardware was exploited to transpose the primary fluid state data upon each of the 1-D sweeps.

The DSM hardware was exploited by this red giant version of PPM in another, still more important way - it was used to achieve a dynamic balancing of the computational loads on the 128 processors in the system. Each of the grid pencils contained a different, dynamically changing number of grid cells in which the gas of this red giant star was present. Of course, no computational labor was performed in the cells holding the vacuum surrounding this star. Hence the amount of work varied from one grid pencil to another based upon the number of grid cells in each which were filled with stellar gas. Also, the grid cells, again of varying number, near the stellar surface required much more work to update than those in the interior. This resulted in still greater differences in the work required for each grid pencil. Consequently the computational labor required to update a grid pencil varied significantly from one grid pencil to another, and these variations were not static as the red giant envelope pulsated. Because the DSM hardware of the Origin-2000 and its high-bandwidth memory system allowed us to have each processor update whichever grid pencil was next on the global task list, regardless of the location of that pencil's data in the memory system, and because performing these grid pencil updates in a special order allowed the computation essentially never to stop and wait, this rather irregular computation was able to proceed at near optimum speed on this equipment. The red giant simulation therefore advanced not only the frontier of stellar astrophysics but also a frontier of parallel computation, exploiting the DSM hardware and the shared memory programming model to achieve nearly transparent computational load balancing over 128 powerful CPUs.

Click here to view more high resolution (2512x3300 pixel) Red Giant images.

Click here to view more Red Giant movies, as well as other LCSE movies.