At Global Vision Technologies, Inc.,(GVT), we spend a lot of time on Research & Development. This R&D falls under several different categories: new features, bug fixes, and performance improvements.

When we started pushing quarterly Vision Server upgrades 5 years ago. We pushed all R&D changes so that everyone would have all the latest features, bug fixes, and performance improvements.

As we did upgrade after upgrade, it became clear that bug fixes and performance enhancements are critical to a system being reliable, secure, and fast. However, new features are in a difference category that is less critical.

The key lessons learned over the past 5 years:

• Installing lots of changes is far more error prone then installing a few. So, upgrade regularly with small upgrades and avoid large upgrades that have many changes
• Developers should detail the key install steps right after they finishing making code. Originally, we compiled all the changes at the end of the quarter (and sometimes key steps were forgotten). Now, we have tools in place that let developers easily markup what code needs to be included and exactly how to install it.
• Bug fixes and performance improvements are more critical than new features. New features can be rolled out when people need them, but bug fixes and performance improvements should be rolled out quickly.
• If you miss or choose to be excluded from an upgrade push, your risks go up more and more with each upgrade that isn’t applied.

Here is an example:

We had a customer a couple years ago that opted out of the quarterly auto update. They did this for 4 quarters and then needed one of our new features. In order to get the new feature, they had to first get up to speed on the missed upgrades. Normally upgrades auto install once per quarter with no involvement from the customer.

During the period where they didn’t get updates, the code in the client's system started to diverge from our base code because of customizations that were being done. So, when the upgrade was applied, we couldn’t be sure it would apply without error. Instead of the upgrades taking 0 hours to install, we spent 20+ hours testing and fixing customizations. They were also vulnerable to security bugs for 12 months. The pain of skipping the upgrades was enough to convince the customer that auto upgrades are the way to go.

That being said, I don’t blame the customer for asking us not to apply upgrade. When we started pushing quarterly updates 5 years ago, it was a rocky process that often led to problems that needed to be patched the next day. Our process has drastically improved. Our methods for marking code to be included in upgrades and for testing upgrades is substantially better now. In the past 12 months, it has become rare that an upgrade causes a problem for any customer.