TITLE

Well, is it almost year end and I realize I haven't done a post in 2024 yet. I have been busy, but I'll post announcements on my blog in January 2025 regarding that. Here is one issue that I've been meaning to write this year. It is on managing change in a system, the types of changes that can be made, and the risk associated with each type.

There are 3 basic types of change that can be made to a system:

1. Additive
2. Deprecating
3. Breaking

These changes can be managed either with our without versioning.

These types of change apply to any system. It could be public changes to a DLL or package (NuGet, PyPi, NPM), an API data contract (REST JSON format, XML DTD, WSDL, gRPC .proto), or a database schema. These rules could even be applied to user interfaces or non-computerized processes.

Additive changes are where a new property or method is added to an existing class (e.g. a new column on a database table) or a new class (new database table).

For example, let's say you had this JSON structure to represent a person:

{
  "FirstName": "Timothy",
  "LastName": "Klenke"
}

To this we add a Gender property

{
  "FirstName": "Timothy",
  "LastName": "Klenke",
  "Gender": "Male"
}

This type of change is the lowest risk of the three. Most consumers of the system should be implemented to ignore new data they aren't programmed to utilize.

To minimize risk when adding a new property, make sure that applications treat the property with a default value if it is missing, or allow them to be null. For example, if adding a soft delete "IsActive" property to the object, make sure applications treat objects as active if the property is missing. This allows you programs to offer backward compatibility with consumers sending the old format.

The second type of change is a deprecation. Here a property, method, class, column, or table is removed from the system. It is the reverse of the additive change. For an example, it would be removing the Gender proper from the Person object shown above.

Deprecating changes do have some risk, but it is relatively easy to manage. It is a safe change to make if it is known that all consumers are not using nor depending on this property.

If you control all the consumers of the system, then you can examine all the source code of those consumers to verify that the references have been removed. Although that's not quite enough. It isn't just that the current version of source code have references the deprecated property removed; all production installations of those programs must also be upgrade.

If you don't control all the consumers (e.g. it is a public API), then it is critical to engage with those customers to make sure they are removing those references. An announcement should be made for upcoming deprecating change and when that will happen. Customers have until then to make the change.

Adding analytics to your applications can also be a good way to manage change. You can log and show on a report or dashboard the number of messages or clients that have adopted the new format. You can send alerts to clients that have failed to migrate.

There are a lot of ways to break a system. I'll go through few examples.

A deprecating change with zero or insufficient notice would generally be considered a breaking change.

Renaming a property would be a breaking change. Consumers are referencing the old property name. You can't swap it out with a new property name without warning. If you break it down a rename operation, it is really adding a new property with the new name, then deprecating the old property name.

Making a property be required where it wasn't required in the past can be a breaking change. This risk is managed very similarly to deprecating changes, where clients need to be monitored to ensure they are all upgraded.

Another example of a breaking change is changing the data type of a field. For example, if an ID property was changed from a 16-bit integer to a 64-bit integer, or to a GUID. Or if a String was changed from a string of ASCII characters to UTF-8 or UTF-16. Adding support for new enumerated values should also be treated as a change to the data type. Under certain circumstances these changes might be transparent, but once the values hit a boundary of using values outside the original data type range, this can break the system.

Behaviour changes to existing methods may or may not be considered a breaking change. For example, changing the Notes field so that when it is saved it is trimmed of leading and trailing white space or automatically reformatted so that there are line breaks every 80 characters, or automatically spell checked. By Hyrum's Law with enough users on a system, every change no matter how small may be considered a breaking change by someone.

The rule to follow when maintaining a system is to never make breaking changes. Avoid requiring that servers and clients must all be upgraded at the same instant in time to successfully transition over a breaking change. Even if the number of servers and clients is small (even just one production server and one production client), it is generally still a good idea to avoid breaking changes.

Decompose breaking changes into additive and deprecating changes, which are manageable. All breaking changes can be decomposed into a series of additive and deprecating changes. This makes the changes smaller and allows less coupling between the systems.

There are two ways to manage change in the data contracts: with and without versioning. Let's first look at how to make changes without versioning. Let's decompose the breaking change of renaming a property.

To safely rename a property this change would have to be broken down into smaller changes. First do an additive change to add the property with the new name. The programs need to be made future compatible, so they support looking at the new property name. They also need to be backward compatible and look at the old property name to see which ever one is populated. Then a deprecating change is announced that the old property will be removed. Consumers migrate their code to start using the new property name. Once all the consumers have been updated and installed into production, then the backward compatibility for the old property name can be removed. The old property name can finally be deprecated and the rename is then complete. It can take months or years to rename a property safely, depending on how much time is given for client to make the migration.

The order of these changes depends on if your API is used for reading or writing. It is usually better to separate reading APIs from writing APIs to allow for changes like this to be more easily managed.

In this example without versioning, there would be several supported formats at various points in time. The original format is:

{
  "FirstName": "Timothy",
  "LastName": "Klenke"
}

Then to rename the "LastName" property to "Surname", the server would have to support the following format so that it would support both backward and forward compatibility:

{
  "FirstName": "Timothy",
  "LastName": "Klenke",
  "Surname": "Klenke"
}

Either the "Surname" or "LastName" properties would be optional. For "write" interfaces, it would be unlikely that clients would send both properties. They would send "LastName" or "Surname" depending on whether they have completed the migration. The behaviour of the server would be it would support both. For "read" interfaces the server would return both properties populated. The clients would read whichever property they support.

The final format would be:

{
  "FirstName": "Timothy",
  "Surname": "Klenke"
}

Here the "LastName" property is deprecated and support for it removed.

With versioning, the supported payloads can be explicitly marked as version 1 or 2. There is no need for the hybrid format that simultaneously supports both the "LastName" and "Surname" properties.

Client applications would explicitly request that all their communication with the server use either version 1 or 2 formats. There are several ways to tell the server which version should be supported. The most common way is to have the version number built into the URL of any web API request. The version number would appear in the path or query string of the URL. Alternatively, different servers could be offered to support the different formats. The version number would be implicitly implied by the host name.

Yet another approach is to include the version number in the payload itself. This can be done either as an additional property, or by adding a parent envelope object.

{
  "Version": 2,
  "FirstName": "Timothy",
  "Surname": "Klenke"
}

Or

{
  "Version": 2,
  Person: {
    "FirstName": "Timothy",
    "Surname": "Klenke"
  }
}

The examples above are for JSON payloads sent to a Web API. However, the same principles can be applied to database changes as well. Relational databases do tend to prefer the "without versioning" approach. However with using views, stored procedures, schemas, and/or triggers it is possible to make changes to relational databases using refactoring techniques.

The key to successfully maintaining a system is being able to manage change effectively. For every change that it made to the system, know what type of change is being made. This will give clear guidance on how the change needs to be communicated and managed.