The good news is that people around the world are trying out SchoolTool right now. The bad news is that we know that from all the bug reports that came in this weekend after Ignas updated some packages on Friday afternoon.

The first lesson here for me is not to give Ignas the goal of putting out a non-trivial update "by the end of the week," which tends to result in late Friday releases followed by weekend bug reports. We'll return to our previous practice of doing releases on Tuesday, which gives us ample time to discuss the details in our Monday IRC meeting and time to put out fires without ruining anyone's weekend.

Note that post-1.0 (April) we'll tighten this up further, but at this point we want to push changes and fixes out quickly and let people have a crack at them.

Finally, I'd just like to stress that if you're having a problem with SchoolTool, don't bang your head against it for hours. If you are having a problem, it is a bug, either a bug in programming, or user interace design, or documentation. We need you to tell us what is broken. You aren't inconveniencing us!

An RFP went out yesterday from the Commonwealth of Virginia for funded pilots of CanDo, which is a component of SchoolTool that tracks student competencies:

Local school divisions, regional technical centers and academies are invited to participate in a pilot project that will test the functionality and capabilities of the “CanDo” electronic student competency recordkeeping system for career and technical education courses. This project will be funded and administered by the Office of Career and Technical Education, Virginia Department of Education (VDOE).

There will be one pilot site per superintendent’s region. The Office of Career and Technical Education will award eight grants of $5,000.

This is a big step for both projects.

Last weekend the Arlington Career Center hosted the post-PloneConf Plone sprint. SchoolTool took the opportunity to piggy-back a little sprint on our gradebook.

We started by going through a specification document that the Arlington Public Schools created to guide their upcoming selection of a new gradebook. Jeff Elkner and David Welsh helped us prioritize these, sorting requirement-laden index cards into piles.

Based on that process, Douglas Cerna, visiting from El Salvador, and Alan Elkner worked on adding category weighting to the grade book, e.g., tests are 30% of the grade, homework 20%, etc. They completed the basic implementation, with just a few known edge cases remaining to be cleaned up. Filip Sufitchi added Javascript hooks to the gradebook view and implemented fill-down to enter the same grade in a given column, while still having enough time to attend the Thomas Jefferson High School homecoming dance.

Beyond coding, Jeff, David, Filip and did a phone interview with education reporter Suzie Boss, which will hopefully result in some well-deserved CanDo buzz. Jeff and I also met with the technology director of the Arlington Education and Employment Program to discuss uses for SchoolTool in adult ed programs. This is potentially a good case for us if we could figure out how to get the funders of these programs to see that putting open source administrative software in the hands of their grantees would make their program evaluation much more efficient. I also got to introduce Jeff to fellow travelers Brian Jepson (Make) and Joan Peckham (URI/NSF) over dinner Saturday. Overall, it was a very productive weekend.

Mondays are meeting days for SchoolTool, and this week's news and updates show things starting to come together as we approach the new school year and a new beta release.

In Vilnius, Ignas Mikalajūnas, with some help from Marius Gedminas, is through the hardest part of implementing a proper structure for school years and terms and will be checking that code in soon. This is the last large, potentially troublesome bunch of functionality that we need to add before the beta in October. Really, it is one of the first things that should have been done, but that mistake was made a long, long time ago. It feels like jacking up a house to add a new first floor. Seeing the end of this part of the project is a big relief.

In Arlington, Virginia, SchoolTool developer Alan Elkner, Programmers of Vilnius (POV) developer Justas Sadzevičius, Arlington student developers including Filip Sufitchi and Chris Carey, and assorted local teachers, administrators and techs got together for a CanDo development sprint, in anticipation of deploying this SchoolTool-based competency tracking application for 120 Virginia teachers later this month. By all accounts things went well, and the meeting illustrates the fairly complicated way our two projects are intertwined. In terms of money, the Commonwealth of Virginia paid for the developers this time around, Alan is an independent contractor who currently works full time on SchoolTool and CanDo, Justas is a co-worker of our lead developer, Ignas. This was Justas's first significant work on either project, but through POV were able to hook CanDo up with an skilled developer to fill in when they needed one on short notice, and we get another developer with SchoolTool experience, and Justas seems to have enjoyed it and impressed everyone in Virginia, so that ended up being a big win all around. Filip and Chris were among the student developers Jeff Elkner trained last year, in part with funding from Mark Shuttleworth and SchoolTool. They're doing high quality work now, making a good buck, and both have at least another year of school. Working closely with our professional developers over the past year has been a big part of their development. It is just about time to really document how we've done all this.

Figuring out how to make the relationship with POV work has been key. Having one full time programmer acting as lead developer, while still working within a small shop (rather than on his own as an individual) seems to be the right combination of flexibility (adding a developer when the schedule fits, having senior colleagues to go to for advice) and stability (commitment to POV as well as SchoolTool, possibility of organized succession if Ignas leaves the project, having local management to write the invoices, pay the rent and other non-programming tasks).

Meanwhile, this week Alan is back in Philadelphia, working to hit his deadline to get some new features running at SLA. This is another kind of relationship we're experimenting with, where SchoolTool is essentially paying a developer to work for a local school. We meet, plan out some goals that make sense for the individual school and the project as a whole, then put Alan to work on them, with a good chunk of his pay dependent on meeting the schedule. The whole dynamic is a little weird, and you lose unity of command, but it is useful in exceptional cases, like when you happen to know an extremely tech-savvy principal and staff and an available developer living in the same city. It has helped us bootstrap things this year, but I don't know that it will go down as a best practice. We'll see how the rest of the year goes.

The main topic of conversation at Monday's IRC meeting was how to update our translation processes (on Launchpad, natch) to get ready for regular production releases. That is, we need to switch from just translating "SchoolTool" to translating specific versions of SchoolTool, how we transfer the old translations into new versions, etc. If SchoolTool is successful, we'll be able to point to this kind of infrastructure work as one reason why; if it isn't successful, doing this kind of infrastructure work before shipping a working application will be one reason why. But really, we can't avoid it. Specifically, Jeff Elkner found us an enthusiastic Zope 3 developer in El Salvador who was at yesterday's IRC meeting and is willing and able to run SchoolTool down there, so we really do need to get the translation story straightened out ASAP.

And that's the way SchoolTool is, as of August 11, 2008.

Ignas, Alan and I held a three day sprint in Vilnius, Lithuania in conjunction with the EuroPython conference last week. We used the face to face time to plan out the remaining significant changes to be implemented before our beta release in October.

Ignas presented his plans for re-organizing the data structures around school years and semesters/trimesters. The basic design sounded very solid, so we spent most of the next two days sketching out the UI implementation and various details.

Most of the remaining time was spent designing and beginning to implement an event log that can be used for basic integration with external server applications. Basically, a log of changes to persons and sections will be created which an external process can periodically scan and propogate changes to a Moodle, Drupal, etc. database. It is not the ultimate in data integration, but a simple, practical step in that direction.

This will be the last sprint we'll hold at a Python convention for the foreseeable future. It has been a useful strategy the past couple years, given the rapid pace of change in Python web development and Zope. We've had to assimilate eggs, WSGI, paste, buildout, and other new (and advantageous!) technologies over the past couple years, and attending talks on the subjects and discussing them face to face was helpful.

With a beta and 1.0 release on the horizon, it will make sense to switch to holding sprints at schools and educational technology conferences, as we switch our focus from technical issues to user feedback, fit and finish.

In other news, Jeff Elkner is in El Salvador, and he managed to stumble upon Douglas Cerna, who may be the only Zope 3 developer in the country, if not all of central America. Jeff is also working with a school down there, and with an interested local developer, we migh be able to get a pilot going.

Also, CanDo has seen some major work on its data model to clean up some nasty deep bugs. Last year's crop of interns are doing very productive work this summer.

This last week saw great advances for the CanDo package of SchoolTool.  It started with a meeting in Richmond, Virginia, with the state administration regarding the up-coming deployment of CanDo in eight new pilot sites across the state.  All parties agreed that we were to go ahead with the deployment this Fall.  It was decided that support would be handled by the state office with coordination back with the Arlington Career Center when needed.  The priority for reliability and ease of use will greatly drive the upcoming summer development effort.  Lastly, we had the opportunity to discuss some of the technical issues regarding competencies which, following the meeting,  led us to make some very important design decisions.

After the meeting with the state, we gathered back at the Arlington Career Center to plan the development sprint that Alan Elkner and Eldar Omuraliev undertook this last week.  Together with David Welsh and Jason Straw, Alan and Eldar discussed a new design for course and section competencies.  The first issue addressed was removing the need for competency inheritance, a source of customer confusion and occasional data-integrity problems.  The second issue was how best to address year to year changes in the global competencies that the state publishes and the complexity of reimporting those changes into an existing CanDo deployment.

The good news was that one solution will be able to address both issues.  Since the state will always supply us with global ids for the competencies they publish, we can reliably use them as keys for the course and section competencies.  Secondly, the fact that these global ids will never change regardless of how they may be organized into the competency groups, we will not need to worry if the structure of those competency groups changes from year to year.  By keeping a list of global competency ids attached to courses and sections, and not keeping any references to groups, we have the ability to deliver those competencies to the grade book with complete reliability.  Bottom line, no need for inheritance and no difficulties reimporting.

The rest of the week involved the actual coding sprint.  The first task was to integrate CanDo into the new SchoolTool tabbed interface and, in the process, remove the need for specifying the CanDo skin in the url or CanDo requests.  The integration took only a couple of days, and now CanDo looks even cleaner than before.  The seem-less integration with SchoolTool will allow users to switch between CanDo and the rest of SchoolTool merely by clicking on the tabs at the top.

With the remaining two days of the sprint we got started with the overhaul of the course and section competencies.  The use of global ids is now unit-tested, and the presence of inheritance has been completely removed form the user interface.  Eldar's job for the next month will be to create views for assigning global ids to courses and sections and for creating the leaner and cleaner grade book that uses them.  The new grade book will not only take advantage of the more reliable competency design, but it will also reduce the time to load by only loading those competencies of the most recently graded competency group.

In all, it was an exciting and highly productive week, and Eldar has a clear understanding of what he needs to do for the next month.  When we gather for the next development sprint in June, we will be bringing last year's interns aboard as junior developers, and we will be able to focus intensely on writing comprehensive functional tests for the entire CanDo application.  This will allow us to address the high priority issue of application reliability favored by the folks at the State of Virginia as well as allow us to move forward with any new features with complete confidence.  In short, we will be ready for the Fall deployment.

What was most remarkable about the SchoolTool sprint at PyCon was how routine it was, especially considering we had a team of three adults and four high school students.  The students, Andrew, Chris, Filip, and Willi are all veterans of sprints on the CanDo project and got right down to work the first morning of the sprint.  Filip gets bonus points for taking a cab to a nearby Radio Shack to buy a soldering iron to fix his power cord prior to the beginning of the sprint. Also, Ignas's buildout configuration made it much easier for everyone to hit the ground running with a working set of packages.

The main goal for the sprint was to finally break out of SchoolTool's traditional navigation paradigm based around traversing the ZODB's object hierarchy and establish a more task based system.  For example, if a teacher wants to do their grades, they should be able to just click on a prominent "grades" tab or link and get to work.

The hard-won lesson from previous sprints is that the navigational changes need to be done in a backward compatible way.  If every single view in the system has to be touched and/or redesigned to complete the transition, it is impossible to complete the job at the sprint, and
difficult to maintain the momentum to complete the job in the face of other pressing local needs.  So while the new navigation system is not "done," it is functional, and will be used in our test schools and upcoming releases.  It can and will be prettier in the future, and there are numerous refinements and finishes necessary, but I don't anticipate another full scale reorganization for a long, long time.

A couple observations from after we made the initial changes.  Jeff Elkner looked at the main calendar page, which he's presumably seen dozens of times over the past couple years (at least) and said "Hey, you can print out PDF's and export to iCal."  Of course, you've been able to do those things all along, they're just more noticeable now.  So that looks like an improvement.

Also, Ignas observed that not having breadcrumbs and and  navigation menu hanging around will take away some crutches that let us get away with not thinking about the user experience.  It was too easy to let the user find his or her way out of traps with the breadcrumbs, but every time that happened, it was really a failure of usability.

Ignas also prepared a queue of nagging, mostly calendar-related, bugs to keep everyone busy throughout the three day sprint.  Ignas did an excellent job of organizing and leading development at the sprint and getting the best out of our interns.

Removed breadcrumbs (Ignas).
Moved navigation menu to top (Ignas).
Made actions menu row of buttons (Ignas).
Yearly calendar view displays term information (Filip).
Made SchoolTool more beautiful in many small ways (Andrew).
Creation of a manage tab, centralizing management screens (Ignas).
Fixes to timetabling management UI (William).
Fixed score system bug in add activities view (Paul).
Added a home view for persons (William).
Removed use of deprecated zapi (Alan).
Refactored calendar portlet to be viewlet based (Filip).
Significantly improved weekly calendar view with timetable integration (Chris).
Improved SchoolTool resource booking view (Andrew).

Also, we'd like to thank Harper Reed of skinnyCorp for showing us around Threadless HQ.  It was great for the interns to see the inside of a wacky web startup.

Ignas Mikalajūnas has overhauled the SchoolTool build system to automate the process of fetching and building dependencies.  The updated instructions assume you're on Ubuntu or another .deb based Linux distribution.

Overview

The basic changes in process we made this year were effective. Development has moved forward much more steadily and consistently than in the past and more closely aligned with the immediate needs of schools. Local funding of active CanDo development has boosted our base of developers and testers, addressing our most pressing constraints. Our school/developer relationship in Brussels did not work out, at more of a cost of time than money.

Code

We've now got running code being tested or used in production for our major components:

• Demographics: Ignas wrote a simpler demographics package to meet the needs of the Vilnius Lyceum and CanDo. It is being used at both sites. There is still a lot of functionality to be built on this base--primarily reports--but the foundation is solid.
• Gradebook: Alan Elkner picked up development of the gradebook module Stephan Richter started last year. It is currently used by Jeff Elkner. Ian Benson will be using it to gather data from maths teachers in eight schools in the UK using his Tizard curriculum. The CanDo competency-based gradebook is also being used in production in the Arlington Career Academy.
• Attendance: Ignas has implemented a "journal" attendance and grading application for the Vilnius Lyceum, which allows teachers to enter a grade for each meeting of the class or mark and absence. The Computer Science department at the Lyceum is currently testing this. Going forward, this will become the standard SchoolTool attendance system.
• Calendaring: The calendaring code has been cleaned up and improved with input from CanDo and testing at Vilnius Lyceum. It needs more testing and tightening in real use.

Other ongoing code issues:

• Interoperability. We funded initial LDAP integration via the Tizard project; Ignas is doing further work on it for use at Vilnius Lyceum. We are planning on supporting CAS for single-sign-on with popular PHP apps used in schools like Moodle. We've got a simple CAS implementation already using existing open source code.
• Releases. As you know, the Zope 3 world was in turmoil this year, especially regarding packaging, and particularly for us the (non-)release of Zope 3.4. Once it is released we will stick with Zope 3.4 as long as possible, because a lot of the burden of packaging Zope 3 and related components for Ubuntu and Debian has fallen upon us.  We have worked under Brian Sutherland's direction on a set of Ubuntu packages for the past six months. We had a set of packages ready for Gutsy but they could not be included in the distribution due to the non-release of Zope 3.4. We subsequently created a new snapshot and developed some scripts to generate packages using the Launchpad PPA, which makes the packaging process much easier going forward. Unfortunately, we also discovered an arcane bug in the PPA, which is now being addressed, but in the meantime, we haven't done a full public release. Since our installation story is now just "apt-get install schooltool", there is no sense in making a release until the packages work.

Big pieces missing from the code:

• Dealing with the passage of time, terms, history. For example, when a student transfers between classes during a term, they can't simply be deleted from the old section, we need to retain the records from that section and keep it integrated in reports for the term. These are issues that touch all parts of the application and will be Ignas's main development priority in 2008.
• Reports. A general release of SchoolTool will at minimum have to have a large set of standard report templates. In 2008 we'll be writing a lot of custom reports for partner schools and deriving standard ones for general distribution.
• More ways to get data in and out from other applications.

Developers

Ignas Mikalajūnas: Switching Ignas to working full time on SchoolTool and acting as "lead developer" and maintainer has been immensely helpful in getting the project back on solid footing. Put simply, SchoolTool now functions like an organized open source development project, not a loose aggregation of contractors managed by a teacher. Ignas has been available and responsive to me and other participants in the project and demonstrated strong leadership when needed.

Jean-Francois Roche: When we started working with Jean-Francois, we were at a point where we needed to find both a developer and a local school. I think we did ok with Jean-Francois, but the school he (and Nicolas Pettiaux) found didn't pan out. They just weren't sufficiently engaged in the project, and Jean-Francois ended up with an unmotivated and uncooperative "customer." This caused repeated delays, required increasing nagging to get information from the school, and with plenty of other (higher-paying) work he could do, Jean-Francois just ended up spending less time on SchoolTool than we would have liked him to. The work he did do was fine, and his billing was in line with the work completed.

After our meeting at the beginning of November, Jean-Francois and I decided to dissolve the partnership with the school, I gave him some work to do on the SchoolTool calendar for the rest of the year, and we left open the possibility of part-time work on SchoolTool next year.

Alan Elkner: An experienced Java developer, Alan started working on learning Zope 3 and SchoolTool, and open source development processes, about a year ago. During the summer he did substantial paid work on CanDo to close up bugs and add features before and after the beginning of school. After CanDo's budget was exhausted, I started paying him with unused development funds to continue CanDo bugfixes (some of which are SchoolTool bugs regardless) as necessary and work on the SchoolTool gradebook.

Once he climbed the Zope 3 learning curve, Alan became a very useful and productive member of the team. He has the right disposition for following through with this work. He's older, not interested in developing new web framework features, just used to writing functional, unsexy software for users. He takes Ignas's advice and direction well. His steady reliability is exactly what we need now.

CanDo Interns: We funded $10,000 worth of work on CanDo by student interns. Jeff Elkner ran a very active program through the spring to train high school students to do SchoolTool and CanDo development. About three students are genuinely productive Zope 3 developers (which, as you know, is hardly easy). Given the scarcity of Zope 3 developers and the effort already spent training them, we are trying to keep them engaged.

Schools

Vilnius Lyceum

Ignas has primarily worked with the three teachers in the computer science department at the school, which is probably sufficient right now. When I met with them in Vilnius, it was clear that having local users had given Ignas a much more focused sense of the requirements and how reality conflicted with his "beautiful abstractions." I pushed to have Ignas visit the school weekly to get feedback in person, and that seems to be helpful as well. Given that Ignas will also be responsible for the remaining chunk of core development and organizing releases in 2008, keeping a small but active group of testers at Lyceum is more important than scaling up to the whole school.

Lycée Emile Jacqmain

This is the school in Brussels. We have stopped working with them due to a lack of interest on their part.

Science Leadership Academy

Science Leadership Academy (SLA) in Philadelphia was one of the schools we hoped to partner with in 2006, when we found that partnership didn't work without a local developer. It is essentially the perfect school for us to work with in terms of capacity. It is a small, innovative school with a 1-to-1 laptop program, a sys admin on site and good professional development support. Most importantly, the principal, Chris Lehmann, developed open source administrative software (in PHP) as a teacher at his former school. As is usually the case, it was too specific to that school to be portable to his new school, but he not only knows exactly what he needs, he can often show you an example.

The good news is that Alan Elkner moved to Philadelphia this fall, so we now have our local developer. I flew down last month (another advantage, I can easily fly there and back in a day) and Alan, Chris and I met. They are both excited about working together. I know Chris will push hard to get SchoolTool to do what he needs and deploy it school-wide in the fall. In particular, he is interested in student and parent access to data on SchoolTool and basic integration, including single sign on, with Moodle and Drupal.

Tizard Schools

Ian Benson approached us via The Shuttleworth Foundation, interested in using SchoolTool to help gather data from research he's conducting using his maths curriculum. In the spring we funded his team's work to add LDAP integration to SchoolTool, allowing him to sync data back to his LDAP server from the schools. We are currently funding their packaging of SchoolTool for distribution and deployment to 8 test schools in the UK. I don't like operating at an additional level of remove from the actual schools, but since they have some development capacity, and I had funds available, we've tried this route.

The deployment is currently behind schedule. Hopefully it will be completed before the end of the calendar year so payments won't be rolled over to the next year.

2008 Roadmap

Bug tracking, feature requests, translations and FAQ for SchoolTool are now managed on Launchpad.  The SchoolTool page on Launchpad is here.  The CanDo page is here.