IT Archaeology of existing systems

There are lots of architects around but I do think that we are in dire need of IT archaeologists. Based on my work as a coach to IT teams working with production business systems, I have to saythat the latest and greatest in technology, frameworks, patterns, and what not is not at all the success factor. Too many times, appetite for technology is hiding the real issues. People do prefer to dive in tech and burn time than to address the core business needs and achieve ROI. The characteristics of the systems at hand are that they do often exist for ages, most of the knowledge does lie in the heads of people, there is a firehose producing an endless stream of changes to implement, and existing documentation material lies in tons of solution-centric documents. Typically, early contributors have moved to other jobs without transferring knowledge. Most of the time, when confronted with change requests, the jump is automatic from high level requirements to technical implementation since people in the know are so coupled with the system and change-centric, tactical documentation is produced. Real issues are associated with the current situation: the mobility of personnel is very limited (I will dare to say frozen), the truck number very low, and it is hard to implement visionary changes to system. Also, implementing normal changes takes eons. When it comes to raising the bar in terms of innovation, the defensiveness of personnel is hindering the success of the initiatives. Mostly, this is not related to a competency problem, nor a lack of goodwill. As far as I am concerned, I do think that the personnel is quite stressed out when required to maintaining the current system alive and any improvement initiative has to consider that factor. When approaching the challenge of documenting existing systems and applications, one must ask what documenting means in the first place. More precisely, documenting an existing system must have clear benefits. If nobody is able to articulate then the effort is just adding more work to the backlog and should be avoided. So, always document with a clear purpose. One very important purpose is to be able to navigate the system code from a higher level of abstraction. Navigation usually requires a kind of map. The map is so that a contributor can understand the key concepts, constructs, and mechanisms being in place and perform changes on the system in an efficient way. This means being able to see the system from a logical perspective, devoid of technological jargon (no files, Oracle databases, CICS, JSPs, ...). Technological jargon is obscuring things more often than not. What is important is to know the responsibilities that a piece does provide, not which technologies are used to implement them. Producing such maps is the key output that an IT archaeologist can provide. But producing useful maps requires global vision. It also means that you have to accept to do research (sometimes exhausting research) and not always be sure to find a decent map. You need to be tolerant to ambiguity to be a good mapper. Assembling bits of knowledge is good but will never be providing a good map. Studying them and figuring out the patterns behind the tons of sand requires exposure to lots of systems of various types. You also need to have a good theory to back the thinking. Notations such as the UML can definitely help. Also, awareness of process frameworks such as the Unified Process is good for separating the various disciplines at play. But all the theory in the world is not worth a single sheet of a useful map. Theory can back action but must stay at its place and not in the foreground. The point is that the map must help to clarify the situation and not further confuse it. The process is looking like: Study existing Documentation ? Remix, and Structure ? Produce Useful Documentation For documenting the system, here is our way to perform some archaeology: As a system is a solution to a given problem, we must be clear about the purpose: are we making a map of the problem or are we making a map of the solution? In our case, we do have a system and do want a map of it. The solution is made out of building blocks, let's call them logical components for now (another name would be subsystems). These logical components are often « virtually present » but are not documented as such (most of the time, you'll find folders grouping files). A characteristic of subsystems is that they do have strong cohesive features : the logging subsystem, the calculation subsystem, the dispatching subsystem, the presentation subsystem, the external gateway subsystem, the security subsystem, the audit subsystem etc. Some items are not easily amenable to the grouping into a subsystem but would rather be « aspects ». For example, even if the security subsystem would concentrate the responsibility of implementing the core of security, enabling security across other subsystems would not be possible to concentrate in that subsystem. But we can introduce a security « aspect », that would be spread to whatever subsystem or subsystem interaction that is subject to realizing a security requirement. Finding out these logical components and aspects and creating the base map out of them would already be quite an achievement as an archaeologist. Another output of the archaeologist is a number of typical interactions between these logical components. And in order to make sense of all of the domain-specific words and concepts used in the project, conceptual documents explaining the business that the system automates is part of the job of the archaeologist. IT Archaeologist may not be the sexiest job on the planet at first sight but it adds tons of value and does indeed have the potential to save millions that would have been spent burning cash spinning wheels.