Maintain backward compatibility
Introduction
When it comes to the reusability of modules created and used by other teams, it is important to pay enough attention to how you introduce your changes to the existing module.
A crucial rule to follow is that all your changes must be backward-compatible.
It's not always evident what counts as a breaking (incompatible) change. The guidance here should be treated as indicative rather than a comprehensive list of every possible change. The rules listed here are only concerned with client compatibility. It is expected that API producers are aware of their own requirements concerning deployment, including changes in implementation details.
The general aim is that clients should not be broken by a service updating to a new minor version or patch. The breakage types under consideration are as follows:
Source compatibility: the code written against v1.0 fails to compile against v1.1.
Binary compatibility: the code compiled against v1.0 fails to link or run against a v1.1 client library. Exact details depend on the client platform.
Wire compatibility: an application built against v1.0 fails to communicate with a v1.1 server.
Semantic compatibility: everything runs but yields unintended or unexpected results.
To put it another way, old clients should work against newer servers within the same major version number. When a client wants to update to a new minor version, for example, to take advantage of a new feature, they should be able to do so easily.
In the documentation, v1.1 and 1.0 refer to logical version numbers that are never made concrete. They are purely for the sake of making it easier to describe changes.
In addition to theoretical protocol-based considerations, there are practical ones related to client libraries involving generated and hand-written codes. Wherever possible, you should always test the changes you consider by generating new versions of client libraries and making sure their tests still pass.
The discussion below divides proto messages into three categories:
- Request messages are sent from the client to the server only, for example,
GetBookRequest. - Response messages are sent from the server to the client only, for example,
ListBooksResponse. - Resource messages are sent both ways, for example,
Bookand any messages used within other resource messages. In particular, consider the resources you can update in terms of a read-modify-write cycle.
Backward-compatible non-breaking changes
Adding API interface to API service definition
From the protocol perspective, adding the API interface to an API service definition is always safe. The only caveat is that client libraries may have already used your new API interface name within the hand-written code. If your new interface is entirely orthogonal to the existing ones, this is unlikely. If it's a simplified version of the existing interface, this can cause a conflict.
Adding method to API interface
This should be fine unless you add a method that conflicts with a method already generated in client libraries.
For example, if you have the GetFoo method, the C# code generator already creates the GetFoo and GetFooAsync methods. Adding the GetFooAsync method to your API interface is a breaking change from the client library perspective.
Adding HTTP binding to method
Assuming the binding doesn't introduce any ambiguities, making the server respond to a URL it previously would have rejected is safe. You can do it when applying the existing operation to a new resource name pattern.
Adding field to request message
Adding request fields can be non-breaking as long as clients that don't specify the field are treated in a new version the same way as in the old one.
The most obvious example of incorrect behavior is pagination. If API v1.0 does not include pagination for a collection, it cannot be added in v1.1 unless the default page_size is treated as infinite, which is generally a bad idea. Otherwise, v1.0 clients who expect to get complete results from a single request can receive truncated results, unaware that the collection contains more resources.
Adding field to response message
A response message that is not a resource, for example, ListBooksResponse, can be expanded without breaking clients as long as it does not change the behavior of other response fields. In a response, any field previously populated should retain the same semantics, even if this introduces redundancy.
For example, in v1.0, a query response can have a boolean field of contained_duplicates to indicate that some results are omitted due to duplication. In v1.1, more detailed information can be provided in the duplicate_count field. Even though it's redundant from the v1.1 perspective, the contained_duplicates field must still be populated.
Adding value to enum
enum used in a request message can freely be expanded to include new elements. For example, a new view can be added in a new minor version using the Resource View pattern. Clients never need to receive this enum, so they don't have to be aware of values that they don't care about.
The default assumption is that clients should handle enum values they're unaware of for resource and response messages. However, API producers should know that writing applications to handle new enum elements correctly can be difficult. API owners should document the expected behavior when a client encounters an unknown enum value.
Proto3 allows clients to receive a value they are unaware of and reserialize the message maintaining the same value, so this does not break the read-modify-write cycle. The JSON format allows a numeric value to be sent where the "name" for the value is unknown. Still, the server typically doesn't know whether or not the client is actually aware of a particular value. Therefore, JSON clients can realize that they received a previously unknown value, but they only see either the name or the number, not both. Returning the same value to the server in a read-modify-write cycle is not supposed to modify that field as the server should understand both forms.
Adding output-only resource field
In a resource entity, it is possible to add fields that are only supplied by the server. The server can validate that any client-supplied value in a request is valid, but it must not fail if the value is omitted.
Backward-incompatible breaking changes
Removing or renaming service, field, method, or enum value
Fundamentally, if the client code can refer to something, removing or renaming it is a breaking change and must result in a major version increase. For some languages like Java, the code referring to the old name causes failures at the compile-time, whereas in other languages, it can cause execution-time failures or data loss. The wire format compatibility is irrelevant here.
Changing HTTP binding
In this context, "to change" is "to delete and add." For example, if you decide that you really want to support PATCH, but your published version supports PUT, or you used a wrong custom verb name, you can add a new binding. However, for the same reasons, you must not remove the old binding as removing a service method is a breaking change.
Changing field type
Even when the new type is wire-format compatible, this can change the generated code for client libraries, resulting in a major version increase. For compiled, statically-typed languages, this can easily introduce compile-time errors.
Changing resource name format
A resource must not change its name, meaning collection names cannot be changed.
Unlike most breaking changes, this affects major versions as well. If a client can expect to use v2.0 to access a resource created in v1.0 or vice versa, the same resource name should be used in both versions.
More subtly, the set of valid resource names should not change for the following reasons:
If it becomes more restrictive, a previously successful request fails.
If it becomes less restrictive than previously documented, clients making assumptions based on the previous documentation can be broken. Clients are very likely to store resource names elsewhere in ways that can be sensitive to the set of permitted characters and the name length. Alternatively, clients can perform their own resource name validation to follow the documentation.
Such a change can only be visible in the documentation of a proto. Therefore, when reviewing a CL for breakage, it is not sufficient to review only non-comment changes.
Changing visible behavior of existing requests
Clients often depend on the API behavior and semantics even when such behavior is not explicitly supported or documented. Therefore, in most cases, consumers see the changing API data's behavior or semantics as breaking. If the behavior is not cryptographically hidden, you should assume that users discovered it and depend on it.
For the reason, it is also a good idea to encrypt pagination tokens even when the data is of no interest. This prevents users from creating their own tokens and potentially being broken when the token behavior changes.
Changing URL format in HTTP definition
In addition to the resource name changes listed above, there are two kinds of change to consider:
- Custom method names: Although a custom method name is not part of the resource name, it is part of the URL posted by REST clients. Changing it shouldn't break gRPC clients, but public APIs must assume they have REST clients.
- Resource parameter names: A change from
v1/shelves/{shelf}/books/{book}tov1/shelves/{shelf_id}/books/{book_id}does not affect the substituted resource name but can affect code generation.
Adding read-write field to resource message
Clients often perform read-modify-write operations. Most clients do not supply values for fields they are unaware of, and proto3, in particular, does not support this. You can specify that any missing message type fields (rather than primitive ones) mean an update is not applied to them, which makes it harder to remove such field values from an entity explicitly. Primitive types, including string and bytes, cannot be handled this way as, in proto3, there is no difference between explicitly setting int32 to 0 and not specifying it at all.
If all updates are performed using a field mask, this isn't a problem as the client won't implicitly overwrite fields it isn't aware of. However, that would be an unusual API decision: most APIs allow "entire resource" updates.
Checking changes
To ensure that your changes do not break anything, use the analysis above for every change you are about to introduce. But in parallel, it is recommended to unit-test your module with 80%+ code coverage.
Following the "all previous tests must pass" rule backs you up, helping to avoid issues missed due to lack of attention to detail. It can also give you the confidence to introduce changes safely. And don't forget to add tests for your new code or change.