Wednesday, August 30, 2006
Hey, Nice Black Eye
Where should I begin? On Saturday morning at 6:30 our load balancer ceased to exist. Another outage to notch onto my cubicle wall. We have a persisting problem of TCP connections between machines just disappearing. We decided to go offline on mid-morning on Monday to compile new kernels from scratch for all our machines. TRACS was down for 5 hours and our network problems are not gone. The good news is that we haven't had a JVM crash on the new kernel. Let's hope it sticks.
While all hell was breaking loose I thought it would be a good time to seriously think about how we maintain our repository. I worked out a little shell script that can update our repository with the latest code from the 2.2.x maintenance branch. I think this is going to make a big difference to how closely we can track Sakai development and how quickly we can get bug fixes. I also think it will make me more likely to contribute to Jira, since I will have at least a shred of confidence that I'm not finding old issues.
I had a Breeze meeting on Monday with some of the folks who are helping out with archive import. We're aiming to have something ready for Sakai 2.3. Honestly, I would rather be working on that full-time right now, but the fires must be put out.
Right now I'm waiting for sessions to drain off of two of our instances so I can update them with a new CourseManagementProvider and a couple of bug fixes for archive import. I really wish there were a way to move session state between nodes in the app server cluster. Then updates could be done in a snap. I imagine it must be technically possible. You just have to find all the state and instantiate it on another node fast enough that the users don't notice anything. Easier said than done, right?
I've got Sakai 2.2.1 running on our staging server, and it mostly looks good. At first I got ClassDefNotFound errors on startup for some Base64 utility class in xerces. When I switched to jre 1.4.2 the error went away. Maybe this is a dependency that doesn't come with Java 5. I'll have to sort that out before I can move this version into production.