Computational Science and Reproducibility - Part 1

When developing numerical computer models with which to do science, an easy trap to fall into is creating an experimental environment that makes it very difficult to reproduce results from an earlier time. This problem is articulated very well by the people over at ArsTechnica, but given my every day experience with this challenge, I thought I would share my perspective here.

Depending on the model's complexity and the rate of development, it can become challenging to reproduce a simulation that was performed two years ago or one that you performed just two hours ago. This may sound surprising, but those who depend on such models understand how rapidly these situations arise. All of these things (and more) can change between one run and a subsequent, seemingly identical run:

1. The source code changes in a seemingly innocuous way. A change for efficiency or to improve the appearance of the source can lead to this. If changes in the input parameters of the code must be changed through the source, this can often lead to changes in the results that are difficult to diagnose in the future.
   
2. The input data changes. Many codes require a plethora of input data, and these files are often swapped around to see how different sources create different results It's easy to lose track of which files gave which results. Even when one is careful, small changes in the input can create large changes in output. Once, I updated some files that listed satellite positions so that they contained additional significant digits. Such a change sounds harmless, but the one one-hundreth change in the satellite's position produced drastically different results!
   
3. The settings have changed. Grid resolution, numerical scheme settings, input parameters, smoothing and blending factors, activation of additional capabilities, and more - all of these items can be changed from run to run. As a code matures, uncountable combinations of settings are possible. Trying to reconstruct the right settings to reproduce a set of results is not an enviable task.
   
4. The code was run on a different computer. Changing compilers, CPU architecture, or linking to different external libraries are all things that can change results- sometimes drastically. This problem frequently manifests itself when care is not taken to ensure the proper precision of floating point real numbers throughout a code. Various compilers handle this differently, and numbers that you thought were double precision can become single precision without warning.
   
5. Breaking a long run into several parts can also change results. When simulations take days to weeks (even on super computing systems), it is commonplace to develop a system that allows you to save progress and restart the simulation later. Care must be taken to ensure that stopping and restarting yields the same answer as an uninterrupted simulation.
   
6. The code gives different results when run in parallel versus serial mode. Large computer simulations can be sped up drastically through parallel computation- that is breaking the problem up into smaller parts and spreading the work over several computers. If not implemented correctly, there can be changes in the solution when switching from serial (one computer) to parallel mode. Things can get trickier if there are changes when the number of CPUs used increases (though the code was always used in parallel).
   

When you lose reproducibility, others can no longer independently verify your work- a key tenet of well-performed science. Furthermore, you can not return to an experiment for further analysis and open questions about your conclusions will persist. In the best case scenario, this leads to inconveniences in research and time lost when you are required to start from scratch. In the worst case scenarios, failure to reproduce results can strip you of your credibility.

So how do you avoid such problems? In my next post, I will outline steps that the space science modeling community frequently takes to overcome the pitfalls listed above.

THERE IS A GOD AND HE LOVES US ALL.

HIS NAME IS VALVLE. OH HELL YES.

Valve Confirms Games, Steam for Macs in April

With the announcement of Portal 2, I was starting to feel a bit weepy about permanently ditching Windows several years ago. I loved HL2 and Portal, and spent more time than I care to admit playing CS:S. This announcement means that I won't have to kludge together a Linux solution if I ever feel the gamin' itch again. It is a good day.

DROID!

The commercials annoyed me, it's true. And I admit that at one point I declared that I would never buy another phone unless it was an iPhone. But I began eating those sentiments double-plus quick when I saw it in action. And, it's on Verizon, to whom I am very loyal thanks to their customer service that has bent over backwards for me on several occasions.

So now, I am the proud owner of a Motorola Droid.
More surprisingly, I've gone from being a Mac person to an Android person. Big time.

This weekend, I used the "My Tracks" application (free!) to follow my progress as I snowboarded down my first black diamond trail. I then uploaded the map to Google Maps while on the lifts. Be sure to click on the red way point marker to see my elevation, max speed, and other stats.

View First Black in a larger map

Android users can find the app here:

or learn more about it here.

I think I'll start posting more Android-esque items. This phone (and operating system) is pretty darn cool.

Whiskey Tango Friday: Lay Egg Is True!

Lay Egg is True!

Time Lapse Recession

Great animation of the recession as unemployment rate by county over the entire nation can be found here: http://cohort11.americanobserver.net/latoyaegwuekwe/multimediafinal.html

It seems to me that more and more of these types of demonstrations are beginning to surface. Has anyone else noticed that the U.S. is starting to feel more like a case study and less like a nation?

The 3rd Reviewer.

Waiting for me in my inbox this morning:

There's a good amount of truth to this satire (despite the rather tired "HITLER REACTS TO :blank: LOLZ" genre). The peer review process can be an absolute gauntlet- reasonably and unreasonably so. There are cases where a reviewer asks for ridiculous changes or additional work. Conversely, there are times when such requests are completely justified. On the other side of the coin, the authors have already put in an intense amount of work to produce their manuscript and want to be done with it once and for all! Even though the additional experiments may be warranted and not so burdensome, the scientists may be loathe to do so because, in their minds, this work is finished. I have been guilty of this myself. Finally, when only one of the three reviewers is requesting the additional effort, it is doubly frustrating. It feels like the odd man out is just trying to be difficult compared to the others.

What Does the U.S. Manufacture?

Computers, but apparently not for long:

"The trend for low-priced computers will last for the coming years," said Shih, according to the AFP's quote of a Commercial Times newspaper interview. "But U.S. computer makers just don't know how to put such products on the market... US computer brands may disappear over the next 20 years, just like what happened to U.S. television brands."

Excuse the anecdotal evidence, but a town that relied on jobs provided by a local television assembly plant was featured in a documentary concerning the impact Walmart has on U.S. economics. The cheaper, foreign made televisions acquired and sold by Walmart led to the collapse of the American plant. The freshly laid off workers in this small town found low paying jobs at their local -- wait for it -- Walmart.

A brief scan through the comments on this story as reported by Tom's Hardware suggests that the readers don't believe that this will happen. They cite higher quality products from companies such as HP versus their foreign counterparts and note, correctly, that the above prediction comes from foreign rival Acer. I am not so optimistic, however. When is the last time that Americans, en masse, valued quality over low prices?

Snow Shoes, Heras!

I haven't been on a good hike for a long while. I finally broke my dry spell this past weekend with the help of brand-spankin' new snow shoes! I took them up the local ski hill twice already; it was a fantastic hike. Click on the way points for pictures, pop the map open in a new window for a better view:

View Snow Shoeing on Pajarito - January 18th, 2010 in a larger map

I'm going to try to go very early this Friday to get some sunrise pictures; my next goal is to take my snowboard with me so that I can get down quickly.

Good Internet Monday BONUS - Delicious Science

Check out these tasty and peer-reviewed recipes over at Not So Humble Pie.

Good Internet Monday - First Person Tetris

I found this over at Tom's Hardware: First Person Tetris. It's a bit... disorientating.

Good Internet Monday - Google

Yeah, it's Wednesday. I know that. This one's a good one, though:

GOOGLE.
KICKS.
ASS.

In a nutshell: There was a recent attempt to hack gmail accounts of prominent human rights activists in China coming from China. Based on Google's investigations, the attack was most likely by the Chinese government. In retaliation, Google will no longer be filtering searches on google.cn, and may pull out of China all together.

We'll have to see if Google actually follows through with these threats, but if so, this is fantastic. The Chinese government has been horribly restrictive about what information circulates from the internet to citizens. By no longer supporting these actions, Google can deal a major blow to censorship in one of the most populous nations in the world.

Whiskey Tango Friday: Stand for Christmas!

Here's a post-Christmas WTF link: Stand for Christmas. It's a collection of people who are butthurt by stores not using enough Christmas decorations, using non-Christmas decorations, saying "happy holidays" instead of "Merry Christmas", do not have a manger scene on display, etc. ad nauseum. They call out the worst perpetrators of such heinous actions, rank the stores, and vow to boycott the top (or is it bottom?) of their anti-Christmas list. It's kind of like reverse Santa.

In a way, this is fine. Everyone has the right to be butthurt, and to try to take action either to relieve the butthurt or to prevent future butthurt. But it seems to me that these people have some things backwards, as evidenced by the testimonials. I focus on the Walmart section, because... do I need to explain that?

[...I] proceeded to ask her why was there no Christmas music playing. She stated that 'some' folks were offended by it & I said 'what if I am offended by it not being played?"

The answer is simple: don't shop there if the absence of "White Christmas" is so unbearable that you are compelled to protest.

The answer is also not so simple.
I like to think that it begins with the irony that is clearly lost on this person. If Christmas isn't about buying crap for everyone while hearing the same four songs sang by 200 different "artists" 24/7, then why would this bother our modest, God-fearing Christian? Apparently, defending Christmas involves the proliferation of the most shallow aspects of the holiday at large retailers (as opposed to, y'know, church.) If I wanted to do the most damage I could to this holiday in terms of its perceived meaning and the number of people who celebrate it in a religious manner, I would not employ a campaign of mass censorship. Rather, I would do that which has already happened and is strongly endorsed by "Stand for Christmas": ensure constant over-stimulation by means of watered-down imagery, sound, and commercialization. Jesus is now secondary (or tertiary or worse) in this nearly-secularized charade*.

Given this person's insistence that their particular beliefs and rituals must be respected by the establishment (under penalty of offending someone?), it is an easy extrapolation to guess their response if anyone else were to receive the same treatment. They would become greatly offended if any sort of Islamic, Jewish, Mormon, etc. etc. holiday were to be paid the same respect. How is it someone could expect such special treatment with blatant disregard for others?

We are turning into a secular European country which is exactly what our founders fought to escape. Wake up America before you lose everything you hold dear.

Oh, that's right... these people are batshit insane**.



*Not that I don't enjoy it immensely. It's a great time to see family, eat too much, and binge on buying and receiving presents.

**To be fair, there are a number of very reasonable posters on the site. For example, this person seems to have reasonable expectations given both the number of people who do celebrate, yet keeping in mind the diversity of the populous:

I have no problem with employees saying "Happy Holidays" or "Season's Greetings" since they don't have any idea which a customer celebrates. I do have problems if a store PROHIBITS the use of "Merry Christmas" or "Happy Hannukah" since we're spending money on gifts for THOSE specific holidays. Hello, if a customer is buying Christmas cards, Christmas tree, Christmas decorations, it's a safe bet they celebrate Christmas. Thanks to Wal-Mart for being inclusive to their customers rather than excluding customers who celebrate Christmas.

Hey, I know that guy!

National Geographic listed their top ten space discoveries of 2009. Number 7 goes to the Mars Phoenix lander, which discovered liquid water upon landing. Co-Investigator Nilton Renno is a University of Michigan AOSS-department scientist, and his (ex-)graduate student, Manish Mehta, performed the experiments that demonstrated that the lander would likely uncover ice by blowing away dust during touch down.

Manish was a fellow grad student during my time at Michigan; congratulations to him and the entire group.

Travelling Blues- Christmas Edition

My family traveled back to Michigan for the holidays, but it was not an easy trip. We were on 5 different flights, and something went wrong on every single one.

Our first leg from Albuquerque to Minneapolis took off on time and even landed- in Albuquerque. Mechanical problems killed the flight, and we were re-booked to go through Atlanta and then to Detroit the next morning. Give Delta credit: they gave us hotel and meal vouchers, then sent out apology letters and 10,000 bonus miles before we had a chance to complain.

The next morning, our plane had to de-ice. This takes a long time in ABQ. We landed in Atlanta 15 minutes before our next flight, which is when you are required to be on board so they can close the aircraft doors. We sprinted from terminal to terminal and then to our gate to find a large crowd. The flight had boarded, but onto the wrong type of plane! They had to clear the plane, find the correct one, and re-board. This gaffe allowed us to make our flight, which was a relief. Bonus surprise: our luggage made it, too.

The trip home had its share of fail, too: it took an hour to load the baggage after the doors had been shut. Another sprint found us on the next flight alright, but this time our baggage didn't make it. Delta told us it would arrive between 11 AM and 11 PM the next day. It showed up at midnight (note: the delivery company was a seperate entity contracted by Delta.)

It is both easy and tough to place all of the blame on Delta. Easy because the scapegoating feels right, but tough to pin them down completely. It was the worst time of year to fly, the flights were exceptionally crowded, and a bit of snowy weather in ABQ didn't help. We never had to argue to get flights changed or luggage delivered. In a way, these hangups were to be expected. On the other hand, when there are five consecutive screw ups, ranging from minor to major, you have to wonder if the experience is anomalous or systemic.

Anyone else have recent bad experiences? Opinions on the matter? I would love to hear them all.

Good Internet Monday - Scale of the Solar System

Christmas Vacation != Actual Vacation. Details forthcoming.

Anyway, here's a link that's been sitting in a Firefox tab for some time. I either stole it from Pharyngula or Bad Astronomy; I can't remember which (those are good blogs that you should be reading if you're not already.) It's the solar system correctly scaled. Be prepared to do some hardcore scrolling to find the planets.

Demonstrations like these are important. It's hard to grasp the vastness and emptiness of our solar system. In a presentation about the feasibility of traveling to Uranus, I learned that the largest barrier was simply the distance to reach the planet. It would take 21 years to arrive there. Think about that. If your child was born on the day the mission launched, you could legally (in the U.S.) share a champagne toast with him or her as they would be celebrating their 21st birthday.

Note: to zip directly to a particular planet, add #planetname to the end of the URL.