On the Joys of Being a Computational Physicist

This is true:

It's even better, however, when your code takes days to run. No matter how little you're doing, as long as your code is running, you feel like you're doing something.

Keyboard Cleaning.

While cleaning my overly-gross keyboard, I realized the patterns in the gunk buildup tell me which keys I press most often.

Or which of my fingers are the dirtiest.

Or both.

Ew.

Computational Science and Reproducibility - Part 1

When developing numerical computer models with which to do science, an easy trap to fall into is creating an experimental environment that makes it very difficult to reproduce results from an earlier time. This problem is articulated very well by the people over at ArsTechnica, but given my every day experience with this challenge, I thought I would share my perspective here.

Depending on the model's complexity and the rate of development, it can become challenging to reproduce a simulation that was performed two years ago or one that you performed just two hours ago. This may sound surprising, but those who depend on such models understand how rapidly these situations arise. All of these things (and more) can change between one run and a subsequent, seemingly identical run:

1. The source code changes in a seemingly innocuous way. A change for efficiency or to improve the appearance of the source can lead to this. If changes in the input parameters of the code must be changed through the source, this can often lead to changes in the results that are difficult to diagnose in the future.
   
2. The input data changes. Many codes require a plethora of input data, and these files are often swapped around to see how different sources create different results It's easy to lose track of which files gave which results. Even when one is careful, small changes in the input can create large changes in output. Once, I updated some files that listed satellite positions so that they contained additional significant digits. Such a change sounds harmless, but the one one-hundreth change in the satellite's position produced drastically different results!
   
3. The settings have changed. Grid resolution, numerical scheme settings, input parameters, smoothing and blending factors, activation of additional capabilities, and more - all of these items can be changed from run to run. As a code matures, uncountable combinations of settings are possible. Trying to reconstruct the right settings to reproduce a set of results is not an enviable task.
   
4. The code was run on a different computer. Changing compilers, CPU architecture, or linking to different external libraries are all things that can change results- sometimes drastically. This problem frequently manifests itself when care is not taken to ensure the proper precision of floating point real numbers throughout a code. Various compilers handle this differently, and numbers that you thought were double precision can become single precision without warning.
   
5. Breaking a long run into several parts can also change results. When simulations take days to weeks (even on super computing systems), it is commonplace to develop a system that allows you to save progress and restart the simulation later. Care must be taken to ensure that stopping and restarting yields the same answer as an uninterrupted simulation.
   
6. The code gives different results when run in parallel versus serial mode. Large computer simulations can be sped up drastically through parallel computation- that is breaking the problem up into smaller parts and spreading the work over several computers. If not implemented correctly, there can be changes in the solution when switching from serial (one computer) to parallel mode. Things can get trickier if there are changes when the number of CPUs used increases (though the code was always used in parallel).
   

When you lose reproducibility, others can no longer independently verify your work- a key tenet of well-performed science. Furthermore, you can not return to an experiment for further analysis and open questions about your conclusions will persist. In the best case scenario, this leads to inconveniences in research and time lost when you are required to start from scratch. In the worst case scenarios, failure to reproduce results can strip you of your credibility.

So how do you avoid such problems? In my next post, I will outline steps that the space science modeling community frequently takes to overcome the pitfalls listed above.

So You Want to Model: Part 1

This post is part 1 of a two part response to a comment on a post concerning numerical modeling of the space environment. TelepathicTroy called for Open Source space codes in order to develop transparency as far as code capability and performance.

This post presents a non-open source solution to this issue: The Community Coordinated Modeling Center. The CCMC is a NASA-run organization that acts as a third party who runs and compares many different space weather models. Furthermore, they provide a resource to the community for running these models and visualizing their results. Although this is not opening up source code to everyone, it does give non-code developers the chance to run each model through a gauntlet of tests and evaluations.

This resource is open to everyone. Check out their runs on request section, or browse their library of simulation results and play with their plotting tools. There are results available from recent model comparisons, the conclusions of which were presented at the current GEM Workshop.

There are issues, however. A limited set of parameters and controls for each model are available to the end user, and this leads to model developers contesting CCMC results that may show their model in a bad light. It is constantly asserted by developers that the results of a certain run would be much more favorable if CCMC had used a certain set of paramters instead of their defaults (typically picked for robustness.) I must admit guilt to this very charge, but in some cases the developer is indeed correct. So while the CCMC is a powerful comparison and evaulation tool, it is still cannot do what an open source approach can: provide 100% transparency to what a code is doing and how well it does it.

It is interesting that such a tool could give rise to a hobby-based or grass roots space weather modeling crowd. Unlikely, but an intriguing prospect.

Stay tuned for Part 2: the SWMF.

Texas Bans Vista.

From this article:

"We are not in any way, shape or form trying to pick on Microsoft, but the problems with this particular [operating] system are known nationwide," Hinojosa said. "And the XP operating system is working very well."

Wow. Doesn't get more damning than that. Microsoft can't wait to get Windows 7 rolled out to end this Vista nightmare.
This post brought to you by Linux.