This post is part 1 of a two part response to a comment on a post concerning numerical modeling of the space environment. TelepathicTroy called for Open Source space codes in order to develop transparency as far as code capability and performance.
This post presents a non-open source solution to this issue: The Community Coordinated Modeling Center. The CCMC is a NASA-run organization that acts as a third party who runs and compares many different space weather models. Furthermore, they provide a resource to the community for running these models and visualizing their results. Although this is not opening up source code to everyone, it does give non-code developers the chance to run each model through a gauntlet of tests and evaluations.
This resource is open to everyone. Check out their runs on request section, or browse their library of simulation results and play with their plotting tools. There are results available from recent model comparisons, the conclusions of which were presented at the current GEM Workshop.
There are issues, however. A limited set of parameters and controls for each model are available to the end user, and this leads to model developers contesting CCMC results that may show their model in a bad light. It is constantly asserted by developers that the results of a certain run would be much more favorable if CCMC had used a certain set of paramters instead of their defaults (typically picked for robustness.) I must admit guilt to this very charge, but in some cases the developer is indeed correct. So while the CCMC is a powerful comparison and evaulation tool, it is still cannot do what an open source approach can: provide 100% transparency to what a code is doing and how well it does it.
It is interesting that such a tool could give rise to a hobby-based or grass roots space weather modeling crowd. Unlikely, but an intriguing prospect.
Stay tuned for Part 2: the SWMF.
Showing posts with label gem. Show all posts
Showing posts with label gem. Show all posts
Wednesday, June 24, 2009
GEM Update
The GEM workshop rolls on; here's an update from the world of space environment modeling.
GEM is very busy; there are several sessions, or "break-out sessions", occurring at one time. Because of the concentration of the material, you're always missing something that is both interesting and relevant to your research. The meeting is supposed to be in "workshop mode", which means talks are short and discussion is long. There is typically a 50% compliance rate with this.
Probably the most interesting information coming out of the workshop so far is how separate space weather models handle the same events. While there are some large-scale similarities, there are many important differences. These differences come up in the three major MHD models in the community - or, in plain-speak, three numerical models, relying on the same underlying physics, all give different results for the same solar driving conditions.
This is a somewhat scary revelation, though not at all surprising. In essence, it means that what we think we know, what the model is actually doing, and what we think the model is doing can all be conflated together and not consistently with each other. Much of this comes with the differences in the way the physics are implemented in the different codes and the different parameters you can change within a single code.
The solution is a lot of work: model parameter space mapping, data-model comparisons, and comparisons to other models. Within a single modeling group, this is a fine solution, but sharing and comparing such results is not always a welcome proposition. As funding can depend on model performance, teams are often reluctant to share these verification and validation results to the world even though it is a necessary step towards better science.
Such was the case this week: modelers could not settle on a set of community-wide verification/validation methods and metrics. This is, of course, a case of history repeating. Some groups do understand the importance of these comparisons and are making concessions and contributing, but others are clamming up as hard as they can.
GEM is very busy; there are several sessions, or "break-out sessions", occurring at one time. Because of the concentration of the material, you're always missing something that is both interesting and relevant to your research. The meeting is supposed to be in "workshop mode", which means talks are short and discussion is long. There is typically a 50% compliance rate with this.
Probably the most interesting information coming out of the workshop so far is how separate space weather models handle the same events. While there are some large-scale similarities, there are many important differences. These differences come up in the three major MHD models in the community - or, in plain-speak, three numerical models, relying on the same underlying physics, all give different results for the same solar driving conditions.
This is a somewhat scary revelation, though not at all surprising. In essence, it means that what we think we know, what the model is actually doing, and what we think the model is doing can all be conflated together and not consistently with each other. Much of this comes with the differences in the way the physics are implemented in the different codes and the different parameters you can change within a single code.
The solution is a lot of work: model parameter space mapping, data-model comparisons, and comparisons to other models. Within a single modeling group, this is a fine solution, but sharing and comparing such results is not always a welcome proposition. As funding can depend on model performance, teams are often reluctant to share these verification and validation results to the world even though it is a necessary step towards better science.
Such was the case this week: modelers could not settle on a set of community-wide verification/validation methods and metrics. This is, of course, a case of history repeating. Some groups do understand the importance of these comparisons and are making concessions and contributing, but others are clamming up as hard as they can.
Sunday, June 21, 2009
Truly Outrageous.
On the final stop of my "Tour de Conference" for summer 2009, I will be going to the Geospace Environment Modeling (GEM) summer workshop. This is THE meeting for researchers such as myself; I'll be sure to keep the updates flowing throughout the week. I will be giving four seperate presentations (two talks, two posters), so my plate will be pretty full. We'll see if I can get in any arguments with other numeric modelers...
Subscribe to:
Posts (Atom)