Commenting Changes
Thoughts on marking significant changes in computer systems with a somewhat formalized change notice format.
Change notices become increasingly relevant when multiple groups (Developers, Quality Assurance, Operations) manage an application. All information about the current state of systems and how they differ from the norm cannot be communicated, as the meetings would be onerous, and humans will forget things, especially at 04:20 following an hour of bad sleep. Detailed change information next to an important change, where the on call should eventually look, will clue them in, and save time wasted hunting down who made the change, or worse, enabling of jobs that should not run.
Significant changes must be commented, so another person can revert the changes to their normal state, or know why the value differs from the norm. Uncommented, the changes could be undone by someone attempting to solve another problem. This could include an application setting, such as a timeout, or whether or not special jobs run from crontab(5).
If the change will be soon reverted, include the previous values—increase timeout to support large files (was 30)—and sufficient instructions on how and when to revert the change. Periodic changes due to load, for example during the holiday season rush, may be grouped into known configuration blocks and switched between depending on the time. These blocks should use a common keyword easily searched for, or perhaps could be hosted on an wiki.
Technorati Tags: coding
Change Notices
Recommendations on change notice metadata and formatting: quick comments that tie a change to a user, date, and other relevant background information. Enough breadcrumbs allow those unfamilier with the change—a new on call, for example—track down the who, what, why, and when details efficiently. An example entry that illustrates the format discussed in detail below:
# jmates 2006-10-29 [SIAL #123] libfoo 1.3.6 buggy, # upgrade to 1.3.9 to avoid "failure to init errors" …
This formal specification must not be overused. Limit to trouble ticket related changes, or where significant changes require special attention.
Formalize the change notice format.
Formalizing change notice structure will help search for and reference changes. In addition to date format standardization, also mandate order of the comment (username, date, optional trouble ticket number, notes), and the format by which trouble ticket numbers are referenced. Without standardization, humans will refer to the same trouble ticket number in a variety of formats that may be difficult to search. Mandating a format then allows tools to be written. The following perl command would print any lines matching a username timestamp ticket number change notice:
$ perl -ne 'print if m/\w+\s+\d{4}-\d\d-\d\d\s+\[SIAL/' *
And from there changes made by particular people, in particular timeframes, or by ticket number. An extended format might use a common comment prefix, or additional keywords to assist searches.
Include a username or e-mail address.
Avoid initials: these may conflict between different users, and most sites will not have an initial to person dictionary available. Instead, use your username, or if usernames are not unique, e-mail address. Anyone interested in the change should be able to find the changer without wasting time querying entire groups or digging through version control logs for the culprit.
Timestamp the change.
Include the date of a change, especially if no trouble ticket number or reference to other dated material exists. Several weeks may pass before problems from a bad change become apparent, and a date helps search for changes made around when historical logs show the first signs of trouble. Standarize the date format to simplify searching. I favor ISO 8601 style YYYY-MM-DD time stamps. Do not include HH:MM:SS data: too much detail, and exact time stamps of the change can be derived from other sources, such as a trouble ticket or version control log.
Reference related trouble tickets or incident numbers.
Trouble ticket numbers should be included as they lead to background information. Organizations should make trouble tickets available for permanent reference, indexed or searchable by trouble ticket number.
Include URL with caution.
Links to permanent blog entries are okay; avoid using WikiWords, as these may change without notice. Only use URL shortening services if the shortened links are permanent enough. URL may change, so should be used with caution. Another option: include terms searchable for on a wiki, not an exact reference. In the event of no hits, as a post mortem task update the wiki to include appropriate terms.
Wiki pages for projects can be referenced, as these are unlikely to change, and will serve up plenty of background information, such as group contact information, relevant external partners, and change schedules. On the other hand, project pages need not be referenced if a group maintains a summary Wiki page detailing all ongoing work.
Include as much relevant metadata as possible.
If possible, also include version numbers, as these will assist future searches. The change comment libfoo 1.3.6 buggy, upgrade to 1.3.9 to avoid init errors will match searching on the 1.3.6 version number, if another code tree has a similar problem that another fixed months ago. Sample error messages or log snippets on subseqent lines will also help.
