The Expand and Contract pattern
I disagree with the popular notion that software products resemble buildings. I believe that software products resemble humans more. They are living organisms that communicate with other software with people and most importantly it is subject to changes from time to time. Of course, since software is changing, the same thing happens for the database as well.
Making changes to the databases is sometimes scary because if the due diligence has not been taken care of, a major mistake might occur, which could even result in lost data. So it is very important to be able to make changes to the databases in a safe way.
The solution to this problem is called the Expand and Contract pattern. It provides us with a predefined set of discrete steps that allows us to commit changes to a productive database without downtime, and with rollback options.
What exactly is the Expand and Contract pattern?
It is a process that is used to transition data from an old data structure to a new one, without affecting the up-time of the product, and at the same time provides adequate testing and rollback options. It consists of a series of steps to introduce the new data structure in the background, prepare the data for live usage, and then switch over to the new structure harmonically.
It allows for easy rollback of changes, for most steps of the process if something does not go as planned, or if the business requirements change along the process.
How can the Expand and Contract pattern be used?
Step 1: Create the new table
The first step of the pattern is the design of the new table that satisfies the business needs. Ideally, this new table will only include small-scale changes to the current structure. If the needed changes are more of a large scale, then the process should be broken down into more steps, which will be implemented serially, only after a successful run of the previous step.
Since the current structure is used by clients, the changes cannot be made directly on the columns of the original table. During the deployment of the new table, we must make sure that there are no constraints with other tables, that can cause conflicts to the system. At the end of the current step, the client should have access to the original table as before.
Step 2: Start writing to the new table (Expand)
After creating the new table, now is the time to configure the client in a way that it will now write to both tables, but it will keep reading exclusively from the original table. In this step, we add data to the new structure in real-time, and thus we have the opportunity to make sure that the data are stored in the new structure in the way that they were designed.
Step 3: Transfer data from the original table to the new one.
In this step, we transfer the data from the original table to a new one. So both tables are now ready to be used. In some cases, a simple copy might suffice, but in other cases, changes to the type of data may be implemented.
This step requires a lot of attention to make sure that after the implementation of this step, the original table and the new table are theoretically and practically the same.
Step 4: Testing the new table
In this step the testing of the new table takes place. It is advised to run queries and the client in parallel while targeting the new table, to make sure that the results returned are the expected ones in both cases.
Step 5: Start reading exclusively from the new table
In this step, the configuration of the client takes place to start reading from the new structure, but keep writing to both tables. Now we can make a sufficient end-to-end test to confirm the expected functionality of the system. Since the writing still takes place in both tables in case an error occurs, we can safely roll back to the old structure until the problem is taken care of.
Step 6: Stop writing to the original structure
Since in the last step, we ascertained the correct functionality of the new structure we can, from now on, stop writing to the original table and use the exclusively the new table.
Step 7: Removal of the original structure from the system (Contract)
The last step of the process is to remove the old table from the structure. At this point, meticulous tests have been made, and the correct functionality of the new structure has been confirmed, consequently, there is no reason to keep the old structure.
Keeping the system running while we try to implement large-scale changes to the database is quite a challenge. The “expand and contract” pattern, provides a quite safe approach to this problem.
References:
[1] Danilo Sato: https://martinfowler.com/bliki/ParallelChange.html