So You Want To Model: Part 2

DISCLAIMER: The author, being a Michigan grad and a former member of the CSEM, is probably pretty biased today.

This post is the second of a two-part discussion resulting from a comment calling for (rather lightheartedly) open-source space environment computer modeling. Open source numerical modeling would be a first (and big) step in solving a classic problem in theoretical studies using computer codes: such codes are often held tightly to the chests of those who wrote them, leaving others wondering what is really happening under-the-hood. At the now-finished GEM conference, attendees could see this effect first hand: several different models were used to simulate the exact same conditions using the exact same theory (ideally), but the results were fantastically different. Without a true understanding of what is going on under the hood, such differences are irreconcilable.

In the previous post, I presented one potential solution: taking advantage of the CCMC to evaluate several models in the community. Today, I would like to highlight the work of the University of Michigan's CSEM group, who has worked to be as open as possible with their models. The efforts of this group, in my humble opinion, provide the most transparent set of numerical modeling tools.

CSEM's flagship code is the Space Weather Modeling Framework, a program that executes, synchronizes and couples several separate space environment codes together. While not truly open source, the SWMF (or, simply, the framework) has features that bring it close to the mark.

• First, the source code is available for download after completing a license agreement. This download includes not only the SWMF, but most of the codes integrated into the framework already.
  
• The framework and included codes are fully documented. This provides the space science community with a full description of under-the-hood aspects of the code while simultaneously helping new, non-CSEM related modelers use the framework.
  
• The code is tested nightly on a variety of systems and compilers. This is not just for the benefit of the core CSEM users- it is an effort to proliferate the framework among interested parties. If it didn't work on your system with your favorite compilers, why would you use it?

While not truly open source, it's close. If you really wanted to see how the code is getting its results, you could easily do so. Although unlikley (for a variety of reasons), these are the types of steps that should be taken across the entire community.

So You Want to Model: Part 1

This post is part 1 of a two part response to a comment on a post concerning numerical modeling of the space environment. TelepathicTroy called for Open Source space codes in order to develop transparency as far as code capability and performance.

This post presents a non-open source solution to this issue: The Community Coordinated Modeling Center. The CCMC is a NASA-run organization that acts as a third party who runs and compares many different space weather models. Furthermore, they provide a resource to the community for running these models and visualizing their results. Although this is not opening up source code to everyone, it does give non-code developers the chance to run each model through a gauntlet of tests and evaluations.

This resource is open to everyone. Check out their runs on request section, or browse their library of simulation results and play with their plotting tools. There are results available from recent model comparisons, the conclusions of which were presented at the current GEM Workshop.

There are issues, however. A limited set of parameters and controls for each model are available to the end user, and this leads to model developers contesting CCMC results that may show their model in a bad light. It is constantly asserted by developers that the results of a certain run would be much more favorable if CCMC had used a certain set of paramters instead of their defaults (typically picked for robustness.) I must admit guilt to this very charge, but in some cases the developer is indeed correct. So while the CCMC is a powerful comparison and evaulation tool, it is still cannot do what an open source approach can: provide 100% transparency to what a code is doing and how well it does it.

It is interesting that such a tool could give rise to a hobby-based or grass roots space weather modeling crowd. Unlikely, but an intriguing prospect.

Stay tuned for Part 2: the SWMF.

GEM Update

The GEM workshop rolls on; here's an update from the world of space environment modeling.

GEM is very busy; there are several sessions, or "break-out sessions", occurring at one time. Because of the concentration of the material, you're always missing something that is both interesting and relevant to your research. The meeting is supposed to be in "workshop mode", which means talks are short and discussion is long. There is typically a 50% compliance rate with this.

Probably the most interesting information coming out of the workshop so far is how separate space weather models handle the same events. While there are some large-scale similarities, there are many important differences. These differences come up in the three major MHD models in the community - or, in plain-speak, three numerical models, relying on the same underlying physics, all give different results for the same solar driving conditions.

This is a somewhat scary revelation, though not at all surprising. In essence, it means that what we think we know, what the model is actually doing, and what we think the model is doing can all be conflated together and not consistently with each other. Much of this comes with the differences in the way the physics are implemented in the different codes and the different parameters you can change within a single code.

The solution is a lot of work: model parameter space mapping, data-model comparisons, and comparisons to other models. Within a single modeling group, this is a fine solution, but sharing and comparing such results is not always a welcome proposition. As funding can depend on model performance, teams are often reluctant to share these verification and validation results to the world even though it is a necessary step towards better science.

Such was the case this week: modelers could not settle on a set of community-wide verification/validation methods and metrics. This is, of course, a case of history repeating. Some groups do understand the importance of these comparisons and are making concessions and contributing, but others are clamming up as hard as they can.