State Software is [not] dead!

 

tl;dr:

For some of you, this blog post will be in the "TL;DR" category (Too Long; Didn't Read). For you, here's a summary of the key points:

  • State Software is Not Dead
  • The Current software is still a good value and serving districts well
  • The Redesign projects:
    • Are taking longer than hoped
    • Are making signficant progress
    • Will have some remarkable capabilities

Keep reading if you want the details.

No Where Near Dead

I've been hearing lately, from various sources, that there are individuals or a few organizations using the phrase "State Software is Dead" as part of their presentations or marketing.  I don't know if this is rumor mongering, wishful thinking or marketing FUD.

Regardless, if you've heard this and have any doubts, the SSDT is funded and State Software is actively maintained, well supported and in use by the vast majority of school districts in Ohio.   We've had no indication that ODE is lessening it's support of State Software. 

I don't mind competition from other vendors.  School districts should have choices for fiscal software.  However, if your business plan depends on the imminent death of State Software, you need a better plan.

The current (Classic) software is, despite it's limitations, a good value with significant features (see my earlier blog posts).  However, it has also been in "maintenance" mode for several years while we have concentrated on the Redesign projects.  This is the only sense in which the software can be considered "dead".  The Classic software is only changed for defects and legislative mandates.  From an end-user's point of view, the Classic software looks static and "dead".  

Nearly all of the SSDT's efforts are being placed into the Redesign.  If you're not tracking our Redesign Milestone releases, you're not seeing the real activity in State Software.

I'll use the rest of this post to describe the USxS Redesign status and some of the latest innovations that will be in the new systems.

The Bad News

WIth any large software project, there's always bad news so I'll deal with it first.

Accurately estimating large software projects is one of the human endeavors that's in the "nearly impossible" category.   If you're a company that builds cars, the car you built last year is very similar to the car you're building this year.  After building a few thousand cars, you can reliably estimate how long the next million will take.  When you do the same or similar thing over and over, it's reasonably easy to create estimates.

Software development is very different.  We never write the same piece of software twice.  Everything we do is something that has never been done before.  This makes estimating software, even on small projects, ridiculously hard.  But users keep insisting on estimates and I keep providing my best estimates (even though I know I'm likely to be wrong).  

When I estimated the USxS Redesign projects, I made long lists of "things" that I knew were going to be needed to replace USAS and USPS.   After creating the list, I made guesses about how long each thing might take.  Most of the individual guesses were SWAGs (because they were unlike anything we had done in the past).  When you add up a long list of SWAGs, small errors pile up and you end up with a pure WAG.  It's a useful number for planning and making go/no-go decisions.  But the number is almost certainly wrong and the margin of error increases with the size of the project. 

Anyway, I estimated the Redesign for USAS at between 10 and 13 person-years.  With the staffing available that put the project at around four years. One way I can express this early estimate is "I hoped it would take three years, I thought it would take four, I was afraid it would take five". 

If you haven't guessed, the bad news is that my hopes and thoughts are gone and we're into my fears… The projects are 12 to 18 months from producing "Release Candidates" that districts could consider using in production.  So we will be entering the fifth year.

The "errors" in the estimate come from several sources.  First, the original list (which we call the Backlog) was too short.  During the design and implementation, we've had to add more work to the Backlog.  There was simply more work then I realized or contemplated.  Second, we have not achieved the "velocity" that my estimates assumed.  The tools we use are allowing us to develop faster than ever, but not fast enough to satisfy my early estimate.  Third, I did not add enough slack for "re-work".  Software developers always make mistakes in design and implementation.  Re-working the errors as soon as we discover them costs time over the short term but saves time in the long run (and makes the system healthier for the future).

This, of course, frustrating for everyone.  We'd all like these projects to be finished and done.  I realize that some districts may give up waiting and buy other software.  Before you run off and buy something, please read:

The Good News

For the rest of this post, I'll concentrate on USAS, but nearly everything below also applies to USPS as well.  The USAS Redesign is about to release it's 7th Milestone.  Over the milestones of the past year, the SSDT has made significant progress on the new UI and functionality.  Due in part to the scheduling pressure, we have had to be more creative and innovative to prevent the projects from falling further behind.  In the old style of programming, we would manually write each field and screen by hand.  In the new systems, we have developed techniques for "dynamically generating" much of the stock functionality.  This saves us time in creating the software. But it also makes the software much more adaptable and malleable for future enhancement.  

We have also leveraged several Open Source technologies. Some of them are architectural/technical (hard to describe) so I'll restrict myself to several that everyone can appreciate.

Query/Filters

In Classic USAS, you could only query for fields which the SSDT placed on the query form.  The SSDT had to write custom code for each query field.  In the new systems, query forms are generated dynamically from the data model, so you will be able to query for any field (including Custom Fields defined by the district).   

For example, in Classic USAS you can only query on fields on the Requisition "header" (number, vendor, etc), you could not query for values on an "item".  But in the Redesign you'll be able to find Requisitions which have a certain word in the item description or a unit price in a specific range.

Dynamic Reporting

In the Classic software, the SSDT provided a substantial number of "canned" reports.   They were somewhat flexible, in terms of  sorting and sub-totaling, but the formats were static.  If a report didn't meet your needs, you had to request a new one from the SSDT (and wait), ask your ITC to write it, or use ODBC to create your own report.

In the Redesign, all reports are dynamic using a "Report Wizard".  You will be able to construct exactly the report you like (pick the fields, sort order, control breaks and subtotals).  Every field in the system will be reportable.  You'll never have to ask the SSDT for some field to be added to a report. Custom Fields you've added will also be reportable.  After you design a report, you'll be able to save it, recall it and share it with other users.  

The SSDT will be providing some "canned" reports which are more complex (like ACCSTS and PAYRPT) but the majority of reporting will be available the the "Report Wizard".

Responsive (AJAX) UI

In traditional web applications, you click a link or button and the server responds by sending an entirely new page.  The browser must receive and re-draw the entire page.  Usually this is quite fast, but there is almost always a noticeable delay in the refresh.

Modern, more sophisticated applications, use a technique called AJAX to refresh only part of the page. This makes the feel more responsive and more like a desktop application. 

For the USAS-R Milestone 7 release, the SSDT is using a set of tools to implement AJAX for query and entry screens.  The new interface style is much smoother and responsive and has allowed us to implement the oft requested "Previous" and "Next" buttons for scrolling through records.  On the Milestone release, there will be three interfaces for the Vendor object ("Legacy Vendor", "Vendor" and "Vendor AJAX Experimental") so you can compare them. Based on feedback, we expect the "AJAX" version to become the standard for future query pages.

Rules Engine

This is perhaps the hardest to explain, but may be the most significant innovation in the history of State Software.  

Executive Summary:  The Redesign USAS and USPS will allow the internal business rules to be customized.  Each district will be able to develop custom rules to determine how the systems behave.

If you're intrigued, keep reading and I'll try to explain what this means.  First, a little background is in order.  Early in the development of the Redesigns, the SSDT was studying various enterprise software architectures including "Event Driven" architectures. We had never done such a thing but it was interesting as a way of "de-coupling" modules.  So we ran an experiment implementing Events in USAS-R. It worked nicely and was useful, but we didn't realize its full potential until later.  Now, the Event model is the key design aspect of the Redesigns and became the basis for other innovations (including the Rules Engine).

An "Event" is a piece of information that represents a "change of state" in the application.  Nearly all business rules in any application are tied to "when something changes" then "something else should happen".  The "something else" could be a validation, a security check, notify someone via email or recording an audit log.

For example, when a PurchaseOrder is created, the Purchase Order module creates a "Purchase Order Created Event" and "publishes" the event to the system.  Other modules in the system can "listen" for events and perform additional processing.  In this example, the Encumbrance Module would be listening for "Purchase Order Created" Events and update the related Budget's encumbered balance.  It would also do "balance checks" and issue a warning, or could check for "inactive accounts" and cancel the transaction.  The Purchase Order Module is entirely unaware that the Encumbrance module exists (thus the modules are "de-coupled" from each other).  This makes the system very flexible. You can change the behavior of the application just by adding a new "event listener".

Now, enter the "Rules Engine".  The Rules Engine is an "event listener" that listens for all events.  It looks for rules that apply to the event and executes them.  The rules can come from various sources.  The SSDT will be developing the "core" rules.   We will also provide "optional" rules that you can turn on or off based on your needs, or use as examples   Most importantly, districts will be able to write their own rules (perhaps with ITC or SSDT assistance).

This perhaps can be made more clear with a few examples.

Suppose, in your school district, you want to insist that all Vendor's have an Email Address. This is not a validation the SSDT could implement for all districts, so it calls for a custom rule.  You would go into the Rules Admin interface, and create a rule like:

rule "Vendors are required to have an Email Address"
      when
           RepositoryModificationEvent()
           Vendor( email.address == null )
      then
           error(kcontext)
  end

 

The rules are always in "when/then" form. When a certain set of conditions exist, then perform some action. 

In the above example, the "RepositoryModificationEvent()" means "an object is being created or updated".  The next line says "a Vendor is being modified and the email is null" (Null is nerd-speak for "empty").  If this rule matches, the "then" part raises an "error".  The error will cause the transaction to be canceled and the name of the rule will be presented to the user as an error message.

Rules can do virtually anything the application can do including use a "Notification Service" for sending emails.   Suppose the Treasurer wants an email when a Purchase Order over some threshold is created. Here is a possible rule:

rule "Notify Treasurer when large PO is posted"
      when
           CreatedObjectEvent()
           po :  PurchaseOrder( amount > 50000 )
           user: User()
       then
           notify("treasurer@example.org", "Alert: Large Purchase Order Posted",
                      "Purchase order #" + po.number 
                      + " has been posted by " + user.username
                      + " for $" + po.amount ) 
           info(kcontext, "Big Brother is watching")
    enr

This rule looks for a new PurchaseOrder with a total amount greater than $50,000.  The Treasurer is notified by email with the PO number, user and amount.  Notice that we can use information from the event to create the email.  The user who posted the PO also receives an "informational" message: "Big Brother is watching".

As a final example, suppose you want to prevent users in the "SECRETARY" role from posting transactions outside of business hours (for security or FLSA concerns):

rule "Transactions may only be posted during business hours"
      when
           RepositoryModificationEvent()
           User ( hasRole("SECRETARY") )
           EventTimestamp( hour > 16 || hour < 8 )
      then
           error(kcontext)
   end

This rule checks the "timestamp" of the event and raises an error if the "hour" is greater than 16 (4pm) or less than 8 (8am) for any user who has the "SECRETARY" role.

The above examples give you a flavor for what the Rules Engine could do, but only scratches the surface. There will be many possible uses for rules, especially when combined with Custom Fields, as well as being the basis for future "work flow" capabilities.

If you're an ITC, you might being going pale when you realize the inherent "risk" of exposing this type of capability to users.  It will be possible for a user (with admin privileges) to write rules that entirely disable the system (even preventing login). The SSDT will mitigate this by creating a "safe mode" that only loads core rules.  Still, there are risks and support/training burdens because each District's system will behave a little (or a lot) differently.

Conclusion

State Software is alive and well.  The Redesigns, while taking longer than hoped, are making significant progress and will have some amazing capabilities when finished.