Walter Almeida's Blog

Golden Rule #2: Don’t be afraid of refactoring!

Here I am, back after more than a month publishing the first golden rule. I said to myself today that I should keep up to speed with this subject in order to finally end-up with a good number of rules as I find the exercise interesting and opening great paths for thinking and reflecting on limited but compelling subjects.

So the Golden Rule #2 is about refactoring code and software architectures. One may wonder: "Why are you talking about continuous refactoring as a Golden rule??!! After all if you follow development best practices and apply all golden rules, aren't you then expecting that the design and code produced are of good quality, therefore not requiring refactoring?!"

Introduction - what is all the fuzz around refactoring?

Refactoring is often associated to "repairing" legacy code or poorly designed applications. Even though it is sometimes what refactoring is used for, refactoring is also first class citizen of a full agile process of developing high quality applications. I would even go one step further: if a reasonable sized project is achieved with no refactoring phase during its implementation, then there is probably something wrong and a flawed underlying design that will bring you trouble on the path ahead.

Software development is a complex discipline, and the vast number of patterns, tools, frameworks and best practices leads to numerous ways for implementing one scenario. It is all about pros and cons, compromises, and identifying the potential future points of evolutions of the business need in order to be ready for it. 

Even though it is always recommended to do a design first and coding later, I don't believe one could or should do a fully detailed design before writing the first line of code. The key winning idea is to be pragmatic, agile and iterative. Any design should be validated by proper implementation and validation, to then be adapted, enhanced and implemented again. There is a limit to paper diagrams and even experienced designers cannot fully predict the behaviour of the future implementation of there designs in terms of performance, security, usability and fitting in the target infrastructure, nor can predict that the frameworks and techniques that they theoretically plan on using will actually be relevant perform/provide a service as excepted; one exception is if they already designed and developed a very similar application, following very similar constraints and non-functional requirements, and targeted to a very similar platform. Added to these technical only aspects, business needs can also evolve, during the lifecycle of the project, get re-scoped and adapted and therefore require changes in the application technical design. The consequence of this is that refactoring phases will be mandatory to iteratively build the software solution.

 

Another point is that a successful software architecture and design is the result of a collaborative work between several people, architects, analysts, and developers. So, all participants should be involved in the process as soon as possible thus requiring starting the development phase early for efficient and productive feedback from the development side ASAP. As a consequence, refactoring phases will have to be planed and performed until reaching an acceptable quality level.

So why do you need refactoring?

Including refactoring phases in your development process and lifecycle answers to the following needs:

• You need to make sure that your software design and architecture properly address evolving business needs: Refactoring phases can be necessary if a major business evolution requires rethinking of part of the design of the application. The application should always be designed with flexibility in mind and requires that aspects of potential evolutions are properly identified to know where to design and build flexibility.

• In an agile process, refactoring is an essential technique to iteratively build the most appropriate software architecture and design, while gaining in maturity iteration by iteration. This is a key for agile development where functional slices of the application are frequently delivered and most of the times require partial refactoring to move from one iteration to the other and increase quality and relevance of the design.

• Late discovery of functional or technical constraints can lead to the need for a partial refactoring of the application to address this unexpected event. By the way, an agile, highly iterative process can help reducing the risk by early addressing all the aspects of the application, and especially the aspects that are considered high risk.

• Results of test campaigns can reveals areas that need refactoring, for either technical reasons (poor performance, low scalability, etc.) or functional reasons (inappropriate functional behaviour revealed by the UAT that has an impact on technical design). Again: and agile iterative process will help address early these issues.

Rigorously including refactoring phases in your development process will therefore help you increase the quality and relevance of the overall application design, making the application more maintainable, robust and flexible. Moreover, being able to regularly and efficiently perform refactoring sessions without breaking the whole application and entering uncontrolled regression phases will be the undisputable proof that your software design is effectively of high standard.

 

Can I include refactoring in my development process seamlessly and for free?

Ok, refactoring is not a simple thing and it is definitively not a painless, safe action to perform if you are not prepared for it.

First condition to include regular refactoring phases as part of your development process is to actually have a well designed and architected application. And refactoring becomes a way to iteratively increase the quality of this design. But you can't apply proper refactoring as part of the development process for commando like or any other kind of quick and dirty type of programming.

Second condition is that you should have a proper set of unit tests covering your full code base. Refactoring is about rewriting and restructuring code thus meaning creation of new bugs. And you don't want to break what was working before. Therefore a set of unit test with high code coverage will help you (yet not always suffice) verify the quality of the refactored code and make corrections if required. Important refactoring will also require the re-writing of the unit tests, which is ok: just make sure that your new set of unit tests still covers the same functional use cases.

Refactoring can be painful if you don't have the proper tooling to assist you and automate repetitive operations. And the pain can be even worth if you need to refactor across several projects. Visual Studio includes basic refactoring tools that are really useful and a big step forward since version 2005. But you can go even a step further and use commercial refactoring tools. The one I use (and love!) is ReSharper, and you can read my post about it for more details.

Another thing you need is a refactoring compliant source code control system that allows you to seamlessly move and rename files while keeping the integrity of source code repository and version history. And I haven't found such a tool yet. To be honest I haven't tried too many tools as I am quite happy with the Microsoft Team Foundation Server package for building a full continuous integration environment. Unfortunately, TFS does not quite like renaming and moving files and can act weird when you try to do so. For large refactoring, the solution is to start on a new, clean TFS project that is built on the new file structure implied by your refactoring. You still keep the full version history though the previous TFS project but lose the connection with your new project. Another advice to limit the issues you can have with your source control while refactoring is the following: try to avoid to a too deep hierarchy of projects, trying to mirror the namespace tree. Just save your projects on a flat, or as flat as possible, file structure. You still can have a hierarchical view in Visual Studio by creating solution folders. This way, it will dramatically make moving or reorganising namespaces much easier! Don't forget that reorganisation namespace is one of the most common thing you will do while refactoring...

Last point to mention is that it will be useful to have an automated process for updating documentation. Refactoring implies changes in the documentation and documentation is not a good candidate for manual update because it will then never be updated. Solution for the documentation is the use of tools that automatically reflect on the code comments to generate the detailed documentation. So rule is: comment properly your code and run a tool on your code base. The tool I use is Sandcastle. And for more high level class diagrams, you can choose a UML tool that can reverse code-engineer UML diagrams from you code. Visual Studio embeds one very basic tool. I personally use Enterprise Architect that I will describe in a future post. Key element is that you should be able to automatically update the technical documentation if you want to have technical documentation at all!

 

So am I ready to go?

You must be aware that refactoring cycles are time consuming, require courage and patience for the extra effort induced. It therefore requires discipline and a well accepted and applied process that includes these "technical" refactoring phases, apparently not bringing any additional features or business value, but really mandatory in the medium to long term for creating a robust and flexible applications. Therefore the sole idea of technical refactoring being part of the software application development lifecycle is something that needs to be strongly supported by the project management board, and more importantly: really applied during the project lifetime. And this is probably the most difficult part as management teams can have difficulties understanding the need of spending a lot of time, effort and money for something that apparently (but apparently only) does not bring any added value to their business application. And this difficulty can be caused by a lack of understanding of both parties and a low trust given to others.

This leads us to another Golden rule on which I will write later. This Golden rule is not really a development golden rule but a more high level, project level, strategic golden rule: go for collaborative and responsibility driven team organisation rather than for a hierarchic centric organisation. This rule says the following: to make a project successful (be it a software project or any kind of project BTW), you should build a team with well defined responsibilities and skills and trust everyone in the team for his/her competencies and believe and take into consideration the well-founded of their advice. This is the best and more human way IMHO to have a real success on your projects and long term involvement and efficiency of your overall project team. But more on this later, so keep connected to the Golden rules series!!

Golden Rule #1 - Don't write code!

The software development golden rule number 1 is: don't write code. Well… seems quite inappropriate to say that about a software development project! The title is a bit exaggerated and the idea behind this golden rule is that you should code only what's mandatory and linked to your business and limit the lines of code to the minimum.

The less code you write, the less time you will spend in writing code (obvious), the fewer bugs you will write, the less code you will have to document, maintain and make evolve. And the less code you will have to refactor sooner or later. This refactoring aspect is quite important and will be the object of another rule stating that any good quality code or software architecture has to go through many refactoring cycles during its lifetime.    

So here are some advices to drive you to limit the code you will have to write:

1. Develop simple solutions. The simpler, the better (another golden rule BTW) and the less code is required. Obvious isn't it? However creating simple solutions IS a challenge. How many times have you been in the situation where you're asking yourself: hey that was obvious, how didn't I think about it in the first place? Thing is that as developers we have complex minds and are looking for the most amazing solutions, and it can become over killing and drives us to developing over-engineering solutions. Good practice is to always try to think twice on a solution and wonder if it could be made simpler. Do peer to peer code reviews with simplicity as one of the goal, presenting your work to others and having your solution challenged is a very good way to enhance and simplify your design

        

2. Linked to previous point is: always develop minimal solutions, answering a direct functional need. As passionate developers we often tend to do too much for the sake of building great applications. Yes you need to design and develop with evolutivity in mind, but wait for the functional requirements before make your design evolve and implementing more that necessary. In the mean time you can concentrate on building simpler, bug free applications and spend the remaining time to develop the unit testing code. And don't worry: new requirements will come and you will have the opportunity to express your creativity on something really required and useful, so be patient!

        

3. Reuse existing components (or frameworks) whenever possible and whenever they are not a company asset or directly linked to the business you are working for. There reusable components can open source components, third party out of the shelf components or your own components. One of the most obvious examples is the use of a GUI framework. You don't want to write a new grid component unless you have a very specific need or unless your business is to build GUI components. What you will have to care of though, is to make sure you choose stable, good quality and well supported components. So spend some time in selecting them and then reuse them in all projects where their use makes sense. Other obvious components to use are those which are related to technical, transverse concerns such as logging, dependency injection, data access and much more...

        

4. One layer of abstraction above components is the use of existing applications or server applications. For instance (almost) no-one will develop his own operating system, or relational database system. These are obvious examples, however an apparently not so obvious one is reporting and you can see companies developing their own reporting layer where one could leverage an existing reporting solution. In the Microsoft world this is Reporting Services, part of the SQL Server services. Another example of an out of the shelf application server is Microsoft SharePoint Server, full electronic document management, collaboration, publishing (and much more) application, fully integrated with the full Microsoft application suite, especially Microsoft Office. Of course there can be a licensing price to pay, but usually the benefit and return on investment is far beyond the licenses you will have to pay, providing that… you choose the right tool for your need! So same story than for components: you should carefully take time to evaluate and choose the right set of applications for your need! But when that's done the gain is then huge.

     

So as a developer you should focus more on evaluating and choosing components, frameworks and applications (unless you're working in a big team and have an architecture group doing it for you, but then try to understand the choices that were done) and compose and integrate them that coding everything from the ground up. You obviously still have to write code but at another level and directly linked to your business. And it will be for you a first step toward a larger vision of software system and toward the software architect role. This also help keeping a good separation of concern in your application design (another golden rule, hey! we are starting to see a common link between all this and it probably confirm the more I will be writing on the subject!)   

So that's enough for now and hope to have caught you attention and you will join me for later post on other golden rules!!

The Golden rules of Software Development series

Last night I thought about an idea for a "post series". I often have coming to my mind quick thoughts about short tips and best practices related to software development. Nothing big, just some general thoughts that could be used as the "tip of day" kind of scenario. I have the tendency to think a lot and the best way for me to release this energy and free my mind is to write down my thoughts and share them. 

 

Good thing about writing down your thoughts and ideas and sharing them is that it forces you to formalize them which often is a more difficult task than first thought but prove to be a very creative activity. I indeed fell and realize that writing down and sharing my own thoughts uses and converts simple, volatile brain energy to fully constructed and usable ideas, and concepts and therefore leverage their power.

 

So I decided upon starting today a series of post called "The golden rules of software development". It will be a series of short posts, about general software development best practices, in random order. First post will for instance be titled "Software Development Golden Rule #1 : Don't write code!"

 

Hope you enjoy the reading and please feel free to comment and react on them!

 

Click here for access to all Software Development Golden Rules available.