Sakai Times

Here's the shell script I wrote that updates the vendor drop in our repository with the latest code from the Sakai 2.2.x maintenance branch. All the magic is in svn_load_dirs.pl which comes with Subversion. One piece that may look odd is those two lines of perl in the middle. I am erasing all those $Url and $Id comment lines from all the code, because those will actually show up as conflicts with my own code when I do the svn merge. #!/bin/sh rev="`svn info -r HEAD https://source.sakaiproject.org/svn | grep Revision: | sed -e 's/Revision: /r/g'`" echo exporting Sakai at $rev svn export -q -$rev https://source.sakaiproject.org/svn/sakai/branches/sakai_2-2-x $rev echo getting rid of those darn Url and Id comments for file in `grep -lr '\$URL' ./$rev/*` do perl -pi -e 's/\$URL.*\$//i' $file perl -pi -e 's/\$Id.*\$//i' $file done echo synching repository with Sakai $rev svn_load_dirs.pl -no_user_input -t $rev svn://projects.its.txstate.edu/tracs/vendor/sakai current ./$rev

Where should I begin? On Saturday morning at 6:30 our load balancer ceased to exist. Another outage to notch onto my cubicle wall. We have a persisting problem of TCP connections between machines just disappearing. We decided to go offline on mid-morning on Monday to compile new kernels from scratch for all our machines. TRACS was down for 5 hours and our network problems are not gone. The good news is that we haven't had a JVM crash on the new kernel. Let's hope it sticks. While all hell was breaking loose I thought it would be a good time to seriously think about how we maintain our repository. I worked out a little shell script that can update our repository with the latest code from the 2.2.x maintenance branch. I think this is going to make a big difference to how closely we can track Sakai development and how quickly we can get bug fixes. I also think it will make me more likely to contribute to Jira, since I will have at least a shred of confidence that I'm not finding old issues. I had a Breeze meeting on Monday with some of the folks who are helping out with archive import. We're aiming to have something ready for Sakai 2.3. Honestly, I would rather be working on that full-time right now, but the fires must be put out. Right now I'm waiting for sessions to drain off of two of our instances so I can update them with a new CourseManagementProvider and a couple of bug fixes for archive import. I really wish there were a way to move session state between nodes in the app server cluster. Then updates could be done in a snap. I imagine it must be technically possible. You just have to find all the state and instantiate it on another node fast enough that the users don't notice anything. Easier said than done, right? I've got Sakai 2.2.1 running on our staging server, and it mostly looks good. At first I got ClassDefNotFound errors on startup for some Base64 utility class in xerces. When I switched to jre 1.4.2 the error went away. Maybe this is a dependency that doesn't come with Java 5. I'll have to sort that out before I can move this version into production.

We still have a couple of JVMs crashing a day. A few of my nerd friends have seen my crash logs, and I’ve got some ideas to try, but nothing definitive. I think we’ll be ok for the weekend, since the crashes seem to be directly related to high demand. I finished updating our CourseManagementProvider. There was still at least one method in SiteAction that assumes a particular format of your course ID, so I cleaned that up too. We have a very simplistic mechanism set up for single signon from our Blackboard system. There is a servlet in TRACS that takes an XML-RPC request from Blackboard and returns a list of TRACS course sites for a given user. So users in Blackboard have a handy list of their TRACS courses with links that can take them straight over there. Each link url has an encrypted token affixed to it consisting of the user’s ID and a timestamp. The link takes them to another servlet that decrypts the token, determines if it is new enough, and then redirects the request to the correct site. It has never worked 100% of the time, and I finally figured it out. Each server in our cluster generates its own encryption key, so the single signon would only work when the load balancer happened to send you back to the same server that the token had been generated on. I changed it so that it will take the encryption key from sakai.properties. This should fix it, but I still need to test it out. Only three weeks before the code freeze for Sakai 2.3. I’m trying to get my import/export coalition to help me produce something release-worthy in that timeframe. We have a Breeze meeting on Monday. Onward an upward, y’all.

We had a very productive meeting this morning with the University of Houston Clear Lake. They wanted to pick our brains about how we run our Sakai implementation. Then the JVMs started crashing. We actually had about a 10 minute outage this morning because I had disabled two of the app servers to update them, and the remaining two JVMs seg faulted nearly simultaneously, leaving nothing but a friendly apology. And another one crashed this afternoon. I felt certain that when we switched over to real hardware, our JVMs would all be fat and happy. They've got plenty of RAM, I've been deliberate about their JAVA_OPTS etc. etc. This is the kind of thing that causes my phone to ring first thing in the morning, and I just hate that. In better news, I fixed some exceptions that our LDAP user provider was throwing because it turns out that some people in our directory don't have an email address. My old friend the NullPointerException drops by for a visit. I'm supposed to be spearheading a campaign to get some fancy import and export into Sakai 2.3, but I'm still holding these servers up with my bare hands. Our database server is taking a beating. I need to get the MySQL optimizations installed as soon as I can. My dear wife just asked me if I wanted anything from the store. “Beer, please.”

Hours and hours of debugging later, the problem with my archive import code turned out to be happening in a method called getEntityAuthzGroups from BaseContentService. Each file resource you want to add is a Sakai Entity, and this method is supposed to return all the groups this Entity belongs to, for authorization purposes. I had been relying on a simplifying assumption that used to work and no longer does: that you could add a resource multiple levels deep into your site hierarchy, and ContentHosting would automatically create any directories upstream from the file you're trying to add. Now when you try to add a file with super directories that don't exist, getEntityAuthzGroups tries to use the file's super directory to get permissions to assign to the file, comes up will null, and the resulting NullPointerException is subsequently swallowed by one of these: catch (Throwable e) { } The fix was to have my code check for the existence of a file's parent directory and create it if necessary. This is good housekeeping anyway. It will take some time to get the new code into production, because we have to take each of our four instances of Tomcat down one by one to minimize the disruption. Everything will probably be up-to-date by midday tomorrow (today!).

We’ve got 345 concurrent sessions right now. I’m not sure we ever broke a hundred before.

Our brand of Sakai is called TRACS, for Teaching, Research, And Collaboration System. I know, you're not supposed to use “and” in your acronyms, but naming a software system is hard. Ever tried it?

Today is the first day of class, and all the basics are working properly. But not everything is right. Off the top of my head, here is the short list of things that are not working in TRACS:

You can’t request a roster in the site setup tool. I haven't finished updating the CourseManagementProvider yet with the new methods that define the format of your institution’s roster IDs. The methods are simple, but since we have tons of sites already in the database with the old Sakai-preferred format, I have to have a way of migrating those, and I haven't gotten to it yet.

The text areas seem to have a cap on what you can paste into them, and this is bugging our faculty who want to paste their multi-page syllabi into the syllabus tool. We saw this problem a year ago, and before I could do something about it, it appeared to go away on its own. I still need to isolate the conditions that cause the text to truncate.

My code that lets faculty import their Blackboard course archives is generating a permissions error. Even though the faculty have the content.new permission on their sites, the BasicContentService says they don’t. The way that permissions are computed on a piece of content is really complex, probably owing to the whole groups thing. It looks like every directory in the Resources tool gets its own AuthzGroup, but for some reason the site maintainer doesn't have any grants to it. Isn’t it enough that they have the maintainer role?

The import code is supposed to be able to do Blackboard question pools and assessments, and it doesn’t yet. This is nobody’s fault but my own. I just need to get on this.

We use MySQL, so I need to get around to applying the query optimizations from Unisa. Maybe these have already been folded into Sakai 2.2.1, but I need to find out. We’re breaking our record for usage today, and the database is going to be the bottleneck.

There are a few places where the database needs cleaning up. There are quite a lot of roles granted to users who no longer exist, and this is causing problems such as when we try to add guests to a site, and it tells us “so and so has already been used.” Really? Thanks. Having accounts in the SAKAI_USER_ID_MAP table also prevents them from getting guest access.

That’s all for now. On top of this, I need to figure out the fastest/smoothest way to move us to Sakai 2.2.1.

We've got 170 active sessions as of this writing. That’s a lot for us.

My inaugural post is all about what I'm doing here. I am a solitary computer programmer at Texas State University in what we call Instructional Technologies Support (I have always been a little miffed about those two S's put together like that). I am part of the well-established schism in higher education between technology for administrative purposes, and technology for instruction. Administrative technologists have the luxury of knowing exactly what they're there for. I came onboard 32 months ago to see about migrating the campus from Blackboard 5.5 to Blackboard 6. Our man Jeff had written some customizations that needed to be ported, and it was beginning to look like the port would be just as much work as the original code. We began casting about for alternatives. In my previous life at the University of Texas at Austin I had seen Indiana University give a presentation about their home-grown course management system called OnCourse. They really seemed to know what they were doing, and it had been a few years, so I decided to drop by their website to see whether I could steal their ideas. That was May of 2004, and it turned out I didn't have to steal anything because they were just weeks away from releasing Sakai 1.0. Now that's what I call service! I will summarize the next two years like this: Sakai taught me how to roll up my sleeves. Our pilot is over; We are entering open migration. Classes start tomorrow, and I bet it's going to be interesting. This weblog is about one nerd's day-to-day life surfing a tidal wave of open source software for higher ed.